VmRSS 过去仅占 25% 左右,但 oom-killer 却来袭

VmRSS 过去仅占 25% 左右,但 oom-killer 却来袭

我有一台配备 128 GB RAM 的专用 MySQL 服务器。MySQL 最近被 oom-killer 杀死了,尽管 MySQL 配置为在最坏情况下使用 95 GB。在我的研究中,我遇到了这种情况:

# cat /proc/11895/status
Name:   mysqld
State:  S (sleeping)
Tgid:   11895
Pid:    11895
PPid:   24530
TracerPid:      0
Uid:    27      27      27      27
Gid:    27      27      27      27
Utrace: 0
FDSize: 1024
Groups: 27
VmPeak: 72188044 kB
VmSize: 72122508 kB
VmLck:         0 kB
VmHWM:  33294036 kB
VmRSS:  32829668 kB
VmData: 72076496 kB
VmStk:        88 kB
VmExe:     11800 kB
VmLib:      3608 kB
VmPTE:     73388 kB
VmSwap:  4139376 kB
Threads:        59

我想知道,为什么 VmHWM 和 VmRSS 仅在 33 GB 左右,而在另一台服务器上(也是同一主服务器的从属服务器,配置几乎相同(缓冲池除外),只是它有 256 GB RAM),输出如下:

# cat /proc/51298/status
Name:   mysqld
State:  S (sleeping)
Tgid:   51298
Pid:    51298
PPid:   50443
TracerPid:      0
Uid:    27      27      27      27
Gid:    27      27      27      27
Utrace: 0
FDSize: 2048
Groups: 27
VmPeak: 243701128 kB
VmSize: 239628932 kB
VmLck:         0 kB
VmHWM:  209331200 kB
VmRSS:  205515868 kB
VmData: 239582156 kB
VmStk:        88 kB
VmExe:     11800 kB
VmLib:      3608 kB
VmPTE:    409600 kB
VmSwap:        0 kB
Threads:        281

这里内存使用率约为 80%,而在发生 oom-killed 的服务器上,内存使用率仅为 25% 左右(请注意,这些值是在 oom-killer 再次发生之前不久观察到的)。原因可能是什么?没有竞争进程。我该怎么办?

编辑:这是 dmesg 告诉我的:

mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
mysqld cpuset=/ mems_allowed=0-1
Pid: 11902, comm: mysqld Not tainted 2.6.32-573.7.1.el6.x86_64 #1
Call Trace:
 [<ffffffff810d6d71>] ? cpuset_print_task_mems_allowed+0x91/0xb0
 [<ffffffff8112a570>] ? dump_header+0x90/0x1b0
 [<ffffffff81232c5c>] ? security_real_capable_noaudit+0x3c/0x70
 [<ffffffff8112a9f2>] ? oom_kill_process+0x82/0x2a0
 [<ffffffff8112a931>] ? select_bad_process+0xe1/0x120
 [<ffffffff8112ae30>] ? out_of_memory+0x220/0x3c0
 [<ffffffff8113780c>] ? __alloc_pages_nodemask+0x93c/0x950
 [<ffffffffa00bd560>] ? ext4_get_block+0x0/0x120 [ext4]
 [<ffffffff8117021a>] ? alloc_pages_current+0xaa/0x110
 [<ffffffff81127967>] ? __page_cache_alloc+0x87/0x90
 [<ffffffff8112734e>] ? find_get_page+0x1e/0xa0
 [<ffffffff81128907>] ? filemap_fault+0x1a7/0x500
 [<ffffffff81151ec4>] ? __do_fault+0x54/0x530
 [<ffffffff81152497>] ? handle_pte_fault+0xf7/0xb20
 [<ffffffff810672e0>] ? wake_up_state+0x10/0x20
 [<ffffffff810b4e7c>] ? wake_futex+0x3c/0x60
 [<ffffffff81153159>] ? handle_mm_fault+0x299/0x3d0
 [<ffffffff81066053>] ? perf_event_task_sched_out+0x33/0x70
 [<ffffffff8104f156>] ? __do_page_fault+0x146/0x500
 [<ffffffff810672b0>] ? default_wake_function+0x0/0x20
 [<ffffffff811dfc50>] ? timeout_func+0x0/0x20
 [<ffffffff8153eb4e>] ? do_page_fault+0x3e/0xa0
 [<ffffffff8153bef5>] ? page_fault+0x25/0x30
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
CPU    4: hi:    0, btch:   1 usd:   0
CPU    5: hi:    0, btch:   1 usd:   0
CPU    6: hi:    0, btch:   1 usd:   0
CPU    7: hi:    0, btch:   1 usd:   0
CPU    8: hi:    0, btch:   1 usd:   0
CPU    9: hi:    0, btch:   1 usd:   0
CPU   10: hi:    0, btch:   1 usd:   0
CPU   11: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd:   0
CPU    1: hi:  186, btch:  31 usd:   0
CPU    2: hi:  186, btch:  31 usd:   0
CPU    3: hi:  186, btch:  31 usd:   0
CPU    4: hi:  186, btch:  31 usd:   0
CPU    5: hi:  186, btch:  31 usd:   0
CPU    6: hi:  186, btch:  31 usd:   0
CPU    7: hi:  186, btch:  31 usd:   0
CPU    8: hi:  186, btch:  31 usd:   0
CPU    9: hi:  186, btch:  31 usd:   0
CPU   10: hi:  186, btch:  31 usd:   0
CPU   11: hi:  186, btch:  31 usd:   0
Node 0 Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd:   0
CPU    1: hi:  186, btch:  31 usd:   0
CPU    2: hi:  186, btch:  31 usd:   0
CPU    3: hi:  186, btch:  31 usd:   0
CPU    4: hi:  186, btch:  31 usd:   0
CPU    5: hi:  186, btch:  31 usd:   0
CPU    6: hi:  186, btch:  31 usd:   0
CPU    7: hi:  186, btch:  31 usd:   0
CPU    8: hi:  186, btch:  31 usd:   0
CPU    9: hi:  186, btch:  31 usd:   0
CPU   10: hi:  186, btch:  31 usd:   0
CPU   11: hi:  186, btch:  31 usd:   0
Node 1 Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd:  29
CPU    1: hi:  186, btch:  31 usd:   0
CPU    2: hi:  186, btch:  31 usd:  30
CPU    3: hi:  186, btch:  31 usd:   0
CPU    4: hi:  186, btch:  31 usd:   0
CPU    5: hi:  186, btch:  31 usd:   0
CPU    6: hi:  186, btch:  31 usd:  15
CPU    7: hi:  186, btch:  31 usd:   0
CPU    8: hi:  186, btch:  31 usd:   0
CPU    9: hi:  186, btch:  31 usd:   0
CPU   10: hi:  186, btch:  31 usd:   0
CPU   11: hi:  186, btch:  31 usd:   0
active_anon:8256706 inactive_anon:760868 isolated_anon:0
 active_file:280 inactive_file:51 isolated_file:0
 unevictable:0 dirty:166 writeback:0 unstable:0
 free:87964 slab_reclaimable:6774 slab_unreclaimable:11217
 mapped:138 shmem:2 pagetables:21060 bounce:0
Node 0 DMA free:15736kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15340kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 2955 64565 64565
Node 0 DMA32 free:248508kB min:2060kB low:2572kB high:3088kB active_anon:1292624kB inactive_anon:388708kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3026080kB mlocked:0kB dirty:16kB writeback:0kB mapped:76kB shmem:0kB slab_reclaimable:5776kB slab_unreclaimable:556kB kernel_stack:0kB pagetables:1824kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 0 61610 61610
Node 0 Normal free:42672kB min:42960kB low:53700kB high:64440kB active_anon:14828712kB inactive_anon:1241992kB active_file:244kB inactive_file:476kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:63088640kB mlocked:0kB dirty:20kB writeback:0kB mapped:8kB shmem:8kB slab_reclaimable:11740kB slab_unreclaimable:27576kB kernel_stack:5568kB pagetables:42220kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1136 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:44940kB min:45076kB low:56344kB high:67612kB active_anon:16905488kB inactive_anon:1412772kB active_file:880kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:66191360kB mlocked:0kB dirty:628kB writeback:0kB mapped:468kB shmem:0kB slab_reclaimable:9580kB slab_unreclaimable:16736kB kernel_stack:1472kB pagetables:40196kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 2*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15736kB
Node 0 DMA32: 1410*4kB 1324*8kB 1136*16kB 941*32kB 659*64kB 332*128kB 162*256kB 51*512kB 25*1024kB 1*2048kB 1*4096kB = 248520kB
Node 0 Normal: 1169*4kB 619*8kB 477*16kB 304*32kB 132*64kB 48*128kB 8*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 43628kB
Node 1 Normal: 939*4kB 637*8kB 446*16kB 296*32kB 165*64kB 53*128kB 4*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 45364kB
60924 total pagecache pages
60506 pages in swap cache
Swap cache stats: add 16616696, delete 16556190, find 18336133/18806499
Free swap  = 0kB
Total swap = 4194300kB
33554431 pages RAM
527380 pages reserved
497 pages shared
9240689 pages non-shared
[ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
[ 1010]     0  1010     2842        1   2     -17         -1000 udevd
[ 3086]     0  3086    23289       42   8     -17         -1000 auditd
[ 3174]     0  3174    16556       26   1     -17         -1000 sshd
[ 3253]     0  3253    20217       24   2       0             0 master
[ 3265]     0  3265    29216       22   0       0             0 crond
[ 3474]     0  3474   249065      355   0       0             0 dsm_sa_datamgrd
[ 3684]     0  3684    73206       86   1       0             0 dsm_sa_eventmgr
[ 3685]     0  3685   115300        9   0       0             0 dsm_sa_datamgrd
[ 3712]     0  3712   109424      428   7       0             0 dsm_sa_snmpd
[ 3757]     0  3757   159830       53   2       0             0 dsm_om_shrsvcd
[ 3781]     0  3781     1016        2   2       0             0 mingetty
[ 3783]     0  3783     1016        2   8       0             0 mingetty
[ 3789]     0  3789     1016        2   9       0             0 mingetty
[ 3795]     0  3795     1016        2  11       0             0 mingetty
[ 3798]     0  3798     1016        2   6       0             0 mingetty
[ 3800]     0  3800     1016        2   2       0             0 mingetty
[ 6104]     0  6104     2841        1   0     -17         -1000 udevd
[ 6106]     0  6106     2841        1   0     -17         -1000 udevd
[24530]     0 24530    26550        7   1       0             0 mysqld_safe
[31635]     0 31635    80599      197   0       0             0 bacula-fd
[38121]     0 38121    55171    26624   0       0             0 puppetd
[ 4546]     0  4546    62368     1465   0       0             0 rsyslogd
[19221]     0 19221     1530        1   3       0             0 collectdmon
[19222]     0 19222   443447     3024   4       0             0 collectd
[ 1460]   500  1460   125934       60   1       0             0 icinga2
[ 1473]   500  1473   688892     1098   1       0             0 icinga2
[18124]    89 18124    20280       19   0       0             0 qmgr
[41035]     0 41035    25232       27   2       0             0 rhnsd
[41116]     0 41116    52038      401   0       0             0 osad
[11895]    27 11895 18030627  8921592   1       0             0 mysqld
[41068]     0 41068    24993      253   2       0             0 sshd
[41070]     0 41070    25142      405   0       0             0 sshd
[14243]     0 14243    24993      244   0       0             0 sshd
[14245]     0 14245    27110      120   1       0             0 bash
[22904]     0 22904    24993      253   2       0             0 sshd
[22906]     0 22906    27108      124   2       0             0 bash
[25586]    38 25586     7684      154   0       0             0 ntpd
[58468]    89 58468    20237      221   1       0             0 pickup
Out of memory: Kill process 11895 (mysqld) score 292 or sacrifice child
Killed process 11895, UID 27, (mysqld) total-vm:72122508kB, anon-rss:35686344kB, file-rss:48kB

输出cat /proc/self/mountinfo

16 21 0:3 / /proc rw,relatime - proc proc rw
17 21 0:0 / /sys rw,relatime - sysfs sysfs rw
18 21 0:5 / /dev rw,relatime - devtmpfs devtmpfs rw,size=66042096k,nr_inodes=16510524,mode=755
19 18 0:11 / /dev/pts rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
20 18 0:16 / /dev/shm rw,relatime - tmpfs tmpfs rw
21 1 253:0 / / rw,relatime - ext4 /dev/mapper/vg_ods055-lv_root rw,barrier=1,data=ordered
22 16 0:15 / /proc/bus/usb rw,relatime - usbfs /proc/bus/usb rw
23 21 8:1 / /boot rw,relatime - ext4 /dev/sda1 rw,barrier=1,data=ordered
24 21 253:3 / /home rw,relatime - ext4 /dev/mapper/vg_ods055-lv_home rw,barrier=1,data=ordered
25 21 253:2 / /var/lib/mysql rw,relatime - ext4 /dev/mapper/vg_ods055_mysql-mysql_data rw,barrier=1,data=ordered
26 16 0:17 / /proc/sys/fs/binfmt_misc rw,relatime - binfmt_misc none rw

输出lsmod

Module                  Size  Used by
tcp_diag                1041  0
inet_diag               8735  1 tcp_diag
vfat                   10584  0
fat                    54992  1 vfat
usb_storage            49228  0
mpt3sas               191659  1
mpt2sas               189883  1
scsi_transport_sas     35588  2 mpt3sas,mpt2sas
raid_class              4388  2 mpt3sas,mpt2sas
mptctl                 31785  1
mptbase                93647  1 mptctl
dell_rbu                9414  0
ipv6                  335525  144
ipmi_devintf            7729  2
sg                     29318  0
joydev                 10480  0
power_meter             9009  0
acpi_ipmi               3745  1 power_meter
ipmi_si                44751  2 acpi_ipmi
ipmi_msghandler        38701  3 ipmi_devintf,acpi_ipmi,ipmi_si
iTCO_wdt                7115  0
iTCO_vendor_support     3056  1 iTCO_wdt
tg3                   161289  0
ptp                     9614  1 tg3
pps_core               10690  1 ptp
dcdbas                  8707  0
sb_edac                17888  0
edac_core              46645  3 sb_edac
lpc_ich                12963  0
mfd_core                1895  1 lpc_ich
shpchp                 29130  0
ext4                  378683  4
jbd2                   93252  1 ext4
mbcache                 8193  1 ext4
sd_mod                 37030  5
crc_t10dif              1209  1 sd_mod
sr_mod                 15049  0
cdrom                  39085  1 sr_mod
megaraid_sas          109375  5
wmi                     6287  0
ahci                   42738  0
dm_mirror              14384  0
dm_region_hash         12085  1 dm_mirror
dm_log                  9930  2 dm_mirror,dm_region_hash
dm_mod                 99168  14 dm_mirror,dm_log

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_ods055-lv_root
                       50G  9.7G   37G  21% /
tmpfs                  63G     0   63G   0% /dev/shm
/dev/sda1             477M   57M  395M  13% /boot
/dev/mapper/vg_ods055-lv_home
                       81G   56M   77G   1% /home
/dev/mapper/vg_ods055_mysql-mysql_data
                      1.1T  700G  344G  68% /var/lib/mysql

输出cat /proc/meminfo

MemTotal:       132108204 kB
MemFree:          380404 kB
Buffers:          214256 kB
Cached:         16000720 kB
SwapCached:        22964 kB
Active:         26156736 kB
Inactive:        9329756 kB
Active(anon):   16971328 kB
Inactive(anon):  2300216 kB
Active(file):    9185408 kB
Inactive(file):  7029540 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4194300 kB
SwapFree:        4039224 kB
Dirty:               816 kB
Writeback:             0 kB
AnonPages:      19261368 kB
Mapped:            17860 kB
Shmem:                20 kB
Slab:             662020 kB
SReclaimable:     617680 kB
SUnreclaim:        44340 kB
KernelStack:        7008 kB
PageTables:        43768 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    22869968 kB
Committed_AS:   69472720 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      484276 kB
VmallocChunk:   34291771192 kB
HardwareCorrupted:     0 kB
AnonHugePages:  16736256 kB
HugePages_Total:   46268
HugePages_Free:    46268
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        5056 kB
DirectMap2M:     2045952 kB
DirectMap1G:    132120576 kB

输出cat /proc/zoneinfo

Node 0, zone      DMA
  pages free     3934
        min      2
        low      2
        high     3
        scanned  0
        spanned  4095
        present  3835
    nr_free_pages 3934
    nr_inactive_anon 0
    nr_active_anon 0
    nr_inactive_file 0
    nr_active_file 0
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 0
    nr_mapped    0
    nr_file_pages 0
    nr_dirty     0
    nr_writeback 0
    nr_slab_reclaimable 0
    nr_slab_unreclaimable 0
    nr_page_table_pages 0
    nr_kernel_stack 0
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 0
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     0
    numa_hit     1
    numa_miss    0
    numa_foreign 0
    numa_interleave 0
    numa_local   0
    numa_other   1
    nr_anon_transparent_hugepages 0
        protection: (0, 2955, 64565, 64565)
  pagesets
    cpu: 0
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 1
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 2
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 3
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 4
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 5
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 6
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 7
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 8
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 9
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 10
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
    cpu: 11
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 8
  all_unreclaimable: 1
  prev_priority:     12
  start_pfn:         1
  inactive_ratio:    1
Node 0, zone    DMA32
  pages free     62344
        min      515
        low      643
        high     772
        scanned  0
        spanned  1044480
        present  756520
    nr_free_pages 62344
    nr_inactive_anon 86498
    nr_active_anon 226241
    nr_inactive_file 14165
    nr_active_file 14186
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 5484
    nr_mapped    4
    nr_file_pages 28412
    nr_dirty     0
    nr_writeback 0
    nr_slab_reclaimable 80093
    nr_slab_unreclaimable 251
    nr_page_table_pages 250
    nr_kernel_stack 0
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 2329633
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     0
    numa_hit     30968949
    numa_miss    12412436
    numa_foreign 0
    numa_interleave 0
    numa_local   30968183
    numa_other   12413202
    nr_anon_transparent_hugepages 600
        protection: (0, 0, 61610, 61610)
  pagesets
    cpu: 0
              count: 21
              high:  186
              batch: 31
  vm stats threshold: 48
    cpu: 1
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 48
    cpu: 2
              count: 75
              high:  186
              batch: 31
  vm stats threshold: 48
    cpu: 3
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 48
    cpu: 4
              count: 199
              high:  186
              batch: 31
  vm stats threshold: 48
    cpu: 5
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 48
    cpu: 6
              count: 16
              high:  186
              batch: 31
  vm stats threshold: 48
    cpu: 7
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 48
    cpu: 8
              count: 24
              high:  186
              batch: 31
  vm stats threshold: 48
    cpu: 9
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 48
    cpu: 10
              count: 167
              high:  186
              batch: 31
  vm stats threshold: 48
    cpu: 11
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 48
  all_unreclaimable: 0
  prev_priority:     12
  start_pfn:         4096
  inactive_ratio:    4
Node 0, zone   Normal
  pages free     13426
        min      10740
        low      13425
        high     16110
        scanned  0
        spanned  15990784
        present  15772160
    nr_free_pages 13426
    nr_inactive_anon 277373
    nr_active_anon 2623458
    nr_inactive_file 422756
    nr_active_file 652474
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 304740
    nr_mapped    2764
    nr_file_pages 1079208
    nr_dirty     32
    nr_writeback 0
    nr_slab_reclaimable 46034
    nr_slab_unreclaimable 6793
    nr_page_table_pages 6298
    nr_kernel_stack 334
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 8578494
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     5
    numa_hit     788900341
    numa_miss    44641935
    numa_foreign 291882873
    numa_interleave 678929
    numa_local   788587089
    numa_other   44955187
    nr_anon_transparent_hugepages 5067
        protection: (0, 0, 0, 0)
  pagesets
    cpu: 0
              count: 37
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 1
              count: 1
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 2
              count: 30
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 3
              count: 3
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 4
              count: 114
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 5
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 6
              count: 161
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 7
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 8
              count: 85
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 9
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 10
              count: 184
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 11
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 80
  all_unreclaimable: 0
  prev_priority:     12
  start_pfn:         1048576
  inactive_ratio:    24
Node 1, zone   Normal
  pages free     15668
        min      11269
        low      14086
        high     16903
        scanned  0
        spanned  16777216
        present  16547840
    nr_free_pages 15668
    nr_inactive_anon 210225
    nr_active_anon 1405678
    nr_inactive_file 1312467
    nr_active_file 1626153
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 332657
    nr_mapped    1691
    nr_file_pages 2940312
    nr_dirty     97
    nr_writeback 0
    nr_slab_reclaimable 27961
    nr_slab_unreclaimable 4145
    nr_page_table_pages 4417
    nr_kernel_stack 104
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 7441636
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     0
    numa_hit     568448792
    numa_miss    291882873
    numa_foreign 57054371
    numa_interleave 678973
    numa_local   567389682
    numa_other   292941983
    nr_anon_transparent_hugepages 2505
        protection: (0, 0, 0, 0)
  pagesets
    cpu: 0
              count: 44
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 1
              count: 98
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 2
              count: 171
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 3
              count: 28
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 4
              count: 55
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 5
              count: 32
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 6
              count: 100
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 7
              count: 204
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 8
              count: 30
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 9
              count: 154
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 10
              count: 30
              high:  186
              batch: 31
  vm stats threshold: 80
    cpu: 11
              count: 91
              high:  186
              batch: 31
  vm stats threshold: 80
  all_unreclaimable: 0
  prev_priority:     12
  start_pfn:         17039360
  inactive_ratio:    25

答案1

你的问题在这里:

HugePages_Total:   46268
HugePages_Free:    46268
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        5056 kB

您已启用一些大页面分配但并未使用它们。这些大页面占用了您很大一部分内存。请禁用此分配。

跑步

sysctl -w vm.nr_hugepages=0

然后检查/etc/sysctl.conf并删除您已设置的分配。

这应该可以释放您浪费的大约 90GiB 内存。

相关内容