当仍有可用内存时,Linux 会随机出现 OOM

当仍有可用内存时,Linux 会随机出现 OOM

我随机遇到 OOM 问题(大约每月一次),但似乎有足够的可用内存。

我不明白原因以及如何解决它......

vm.min_free_kbytes = 90112

Linux xxx-alpha 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux

内存监视器

slabtop

 Active / Total Objects (% used)    : 45902519 / 54189870 (84.7%)
 Active / Total Slabs (% used)      : 1260240 / 1260240 (100.0%)
 Active / Total Caches (% used)     : 151 / 187 (80.7%)
 Active / Total Size (% used)       : 9241442.17K / 10856556.48K (85.1%)
 Minimum / Average / Maximum Object : 0.01K / 0.20K / 23.25K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
43615236 37957660  87%    0.19K 1038467       42   8307736K dentry
2180640 1579066  72%    0.07K  38940       56    155760K range_seg_cache
706048 538395  76%    0.01K   1379      512      5516K kmalloc-8
644992 395397  61%    0.03K   5039      128     20156K kmalloc-32
635150 294932  46%    0.32K  12963       49    207408K arc_buf_hdr_t_full
546840 332510  60%    0.09K  13020       42     52080K kmalloc-96
dmesg

[Sun Jan 10 00:05:35 2021] elasticsearch[7 invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
[Sun Jan 10 00:05:35 2021] elasticsearch[7 cpuset=67e9bddb581923bf7602fbcce820867ff13dd19355d458112bee6300e0a2cbd8 mems_allowed=0-1
[Sun Jan 10 00:05:35 2021] CPU: 45 PID: 9599 Comm: elasticsearch[7 Tainted: P           OE     4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
[Sun Jan 10 00:05:35 2021] Hardware name: Ericsson CRU 0101/S2BS-MB, BIOS S2B_3B20 05/30/2019
[Sun Jan 10 00:05:35 2021] Call Trace:
[Sun Jan 10 00:05:35 2021]  dump_stack+0x5c/0x80
[Sun Jan 10 00:05:35 2021]  dump_header+0x6b/0x283
[Sun Jan 10 00:05:35 2021]  ? do_try_to_free_pages+0x2ec/0x370
[Sun Jan 10 00:05:35 2021]  oom_kill_process.cold.30+0xb/0x1cf
[Sun Jan 10 00:05:35 2021]  ? oom_badness+0x23/0x140
[Sun Jan 10 00:05:35 2021]  out_of_memory+0x1a5/0x430
[Sun Jan 10 00:05:35 2021]  __alloc_pages_slowpath+0xbd8/0xcb0
[Sun Jan 10 00:05:35 2021]  __alloc_pages_nodemask+0x28b/0x2b0
[Sun Jan 10 00:05:35 2021]  filemap_fault+0x3bd/0x780
[Sun Jan 10 00:05:35 2021]  ? filemap_map_pages+0x1ed/0x3a0
[Sun Jan 10 00:05:35 2021]  __do_fault+0x36/0x130
[Sun Jan 10 00:05:35 2021]  __handle_mm_fault+0xe6c/0x1270
[Sun Jan 10 00:05:35 2021]  handle_mm_fault+0xd6/0x200
[Sun Jan 10 00:05:35 2021]  __do_page_fault+0x249/0x4f0
[Sun Jan 10 00:05:35 2021]  ? page_fault+0x8/0x30
[Sun Jan 10 00:05:35 2021]  page_fault+0x1e/0x30
[Sun Jan 10 00:05:35 2021] RIP: 0033:0x7f68612bc435
[Sun Jan 10 00:05:35 2021] Code: 1f 00 00 49 c1 e0 03 41 8b 48 18 45 8b 48 1c 41 3b c9 0f 8d 47 21 00 00 4d 8b 50 10 8b d9 ff c3 41 89 58 18 49 8b fa 4c 63 d1 <46> 0f be 14 17 45 85 d2 0f 8c 07 0d 00 00 41 8b ca 83 e1 01 41 88
[Sun Jan 10 00:05:35 2021] RSP: 002b:00007e876f44e050 EFLAGS: 00010207
[Sun Jan 10 00:05:35 2021] RAX: 0000000000000002 RBX: 00000000000e37a6 RCX: 00000000000e37a5
[Sun Jan 10 00:05:35 2021] RDX: 0000000684825b90 RSI: 000000000032bce7 RDI: 00007f625cad5441
[Sun Jan 10 00:05:35 2021] RBP: 00000000d3887259 R08: 0000000684825bd0 R09: 00000000002f8d73
[Sun Jan 10 00:05:35 2021] R10: 00000000000e37a5 R11: 0000000684825b90 R12: 0000000000000000
[Sun Jan 10 00:05:35 2021] R13: 00000006829fb780 R14: 00000000d0904b72 R15: 00007f66dc05f800
[Sun Jan 10 00:05:35 2021] Mem-Info:
[Sun Jan 10 00:05:35 2021] active_anon:12218942 inactive_anon:93895 isolated_anon:0
                            active_file:1203 inactive_file:1345 isolated_file:6
                            unevictable:0 dirty:15 writeback:21 unstable:0
                            slab_reclaimable:1710574 slab_unreclaimable:1562977
                            mapped:26078 shmem:177912 pagetables:59851 bounce:0
                            free:49188 free_pcp:9835 free_cma:0
[Sun Jan 10 00:05:35 2021] Node 0 active_anon:31064964kB inactive_anon:48776kB active_file:284kB inactive_file:392kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:18116kB dirty:4kB writeback:4kB shmem:128568kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[Sun Jan 10 00:05:35 2021] Node 1 active_anon:17810804kB inactive_anon:326804kB active_file:4528kB inactive_file:4988kB unevictable:0kB isolated(anon):0kB isolated(file):24kB mapped:86196kB dirty:56kB writeback:80kB shmem:583080kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[Sun Jan 10 00:05:35 2021] Node 0 DMA free:15904kB min:20kB low:32kB high:44kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 1848 32052 32052 32052
[Sun Jan 10 00:05:35 2021] Node 0 DMA32 free:123288kB min:2556kB low:4420kB high:6284kB active_anon:1651524kB inactive_anon:2372kB active_file:116kB inactive_file:56kB unevictable:0kB writepending:0kB present:1959556kB managed:1892668kB mlocked:0kB kernel_stack:212kB pagetables:2500kB bounce:0kB free_pcp:248kB local_pcp:0kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 0 30203 30203 30203
[Sun Jan 10 00:05:35 2021] Node 0 Normal free:22820kB min:42348kB low:73276kB high:104204kB active_anon:29413440kB inactive_anon:46404kB active_file:156kB inactive_file:336kB unevictable:0kB writepending:8kB present:31457280kB managed:30928584kB mlocked:0kB kernel_stack:21416kB pagetables:78456kB bounce:0kB free_pcp:6260kB local_pcp:0kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 0 0 0 0
[Sun Jan 10 00:05:35 2021] Node 1 Normal free:34740kB min:45184kB low:78184kB high:111184kB active_anon:17810804kB inactive_anon:326804kB active_file:5176kB inactive_file:5044kB unevictable:0kB writepending:136kB present:33554432kB managed:33004524kB mlocked:0kB kernel_stack:47908kB pagetables:158448kB bounce:0kB free_pcp:32832kB local_pcp:1324kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 0 0 0 0
[Sun Jan 10 00:05:35 2021] Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 3*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB
[Sun Jan 10 00:05:35 2021] Node 0 DMA32: 99*4kB (UMH) 15*8kB (UMH) 3483*16kB (UMH) 2052*32kB (UMEH) 3*64kB (MH) 1*128kB (H) 3*256kB (H) 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 123508kB
[Sun Jan 10 00:05:35 2021] Node 0 Normal: 882*4kB (UMEH) 33*8kB (MEH) 34*16kB (ME) 751*32kB (UME) 3*64kB (UH) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 28560kB
[Sun Jan 10 00:05:35 2021] Node 1 Normal: 8333*4kB (MH) 26*8kB (H) 1*16kB (H) 1*32kB (H) 3*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 33780kB
[Sun Jan 10 00:05:35 2021] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Sun Jan 10 00:05:35 2021] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sun Jan 10 00:05:35 2021] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Sun Jan 10 00:05:35 2021] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sun Jan 10 00:05:35 2021] 181392 total pagecache pages
[Sun Jan 10 00:05:35 2021] 0 pages in swap cache
[Sun Jan 10 00:05:35 2021] Swap cache stats: add 0, delete 0, find 0/0
[Sun Jan 10 00:05:35 2021] Free swap  = 0kB
[Sun Jan 10 00:05:35 2021] Total swap = 0kB
[Sun Jan 10 00:05:35 2021] 16746816 pages RAM
[Sun Jan 10 00:05:35 2021] 0 pages HighMem/MovableOnly
[Sun Jan 10 00:05:35 2021] 286396 pages reserved
[Sun Jan 10 00:05:35 2021] 0 pages hwpoisoned
[Sun Jan 10 00:05:35 2021] Tasks state (memory values in pages):
[Sun Jan 10 00:05:35 2021] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Sun Jan 10 00:05:35 2021] [    611]     0   611    60450    40093   503808        0             0 systemd-journal
[Sun Jan 10 00:05:35 2021] [    635]     0   635     5616      348    61440        0         -1000 systemd-udevd
[Sun Jan 10 00:05:35 2021] [    814]     0   814      770       56    45056        0             0 mdadm
[Sun Jan 10 00:05:35 2021] [   9990]     0  9990    41684      221    69632        0             0 zed
[Sun Jan 10 00:05:35 2021] [   9991]     0  9991     2557       73    61440        0             0 cron
[Sun Jan 10 00:05:35 2021] [   9993]   104  9993     2222      187    57344        0          -900 dbus-daemon
[Sun Jan 10 00:05:35 2021] [   9994]     0  9994     4991      386    77824        0             0 systemd-logind
[Sun Jan 10 00:05:35 2021] [  10013]     0 10013   200898     2220    65536        0             0 supervisord
[Sun Jan 10 00:05:35 2021] [  10020]     0 10020    28023     1915   114688        0             0 unattended-upgr
[Sun Jan 10 00:05:35 2021] [  10028]     0 10028  1405091     6815   843776        0             0 containerd
[Sun Jan 10 00:05:35 2021] [  10033]     0 10033  1307161    15321   995328        0          -500 dockerd
[Sun Jan 10 00:05:35 2021] [  10040]     0 10040     3963      236    69632        0         -1000 sshd
[Sun Jan 10 00:05:35 2021] [  10074]     0 10074     4934     1407    73728        0             0 haproxy
[Sun Jan 10 00:05:35 2021] [  12823]     0 12823     1403       28    45056        0             0 agetty
[Sun Jan 10 00:05:35 2021] [  13945]   107 13945     4935      210    81920        0             0 exim4
[Sun Jan 10 00:05:35 2021] [  19622]     0 19622     5382      445    81920        0             0 systemd
[Sun Jan 10 00:05:35 2021] [  19623]     0 19623     5779      630    73728        0             0 (sd-pam)
[Sun Jan 10 00:05:35 2021] [  24193]     0 24193    31559     4572   143360        0             0 prom-node-expor
[Sun Jan 10 00:05:35 2021] [   9763]     0  9763    61147     1433    65536        0             0 nginx
[Sun Jan 10 00:05:35 2021] [   4864]     0  4864    58959      222    94208        0             0 polkitd
[Sun Jan 10 00:05:35 2021] [  19619]     0 19619   155613      481   126976        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  19633]     0 19633    27276      212    73728        0          -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [  19686]   472 19686    46091     7685   315392        0             0 grafana-server
[Sun Jan 10 00:05:35 2021] [  34921]     0 34921   820113      471   462848        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  34929]     0 34929    26924      202    73728        0          -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [  34948] 65534 34948  5391852    43302 21938176        0             0 prometheus
[Sun Jan 10 00:05:35 2021] [  12393]   108 12393    19145      168    65536        0             0 ntpd
[Sun Jan 10 00:05:35 2021] [   5862]     0  5862     4947     2284    90112        0             0 tmux: server
[Sun Jan 10 00:05:35 2021] [   5863]     0  5863     2209      151    61440        0             0 bash
[Sun Jan 10 00:05:35 2021] [  33048]     0 33048     2249      215    57344        0             0 bash
[Sun Jan 10 00:05:35 2021] [  29476]     0 29476     2201      154    53248        0             0 bash
[Sun Jan 10 00:05:35 2021] [  31493]     0 31493      597       46    45056        0             0 mysqld_safe
[Sun Jan 10 00:05:35 2021] [  45444]     0 45444     5315     1563    86016        0             0 mysql
[Sun Jan 10 00:05:35 2021] [  13606]     0 13606     2209      156    61440        0             0 bash
[Sun Jan 10 00:05:35 2021] [  35429]     0 35429     2179      152    49152        0             0 bash
[Sun Jan 10 00:05:35 2021] [  26987]     0 26987   137180      482   118784        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  27040]     0 27040   985866     1166   544768        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  27049]     0 27049    27276      940    86016        0          -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [  27074]     0 27074     4947       74    81920        0             0 wrapper
[Sun Jan 10 00:05:35 2021] [  28817]     0 28817     1098       22    53248        0             0 runsvdir
[Sun Jan 10 00:05:35 2021] [  29114]     0 29114     1060       19    49152        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  29117]     0 29117     1096       20    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  29120]     0 29120    16378      178   167936        0             0 sshd
[Sun Jan 10 00:05:35 2021] [  34657]     0 34657     1060       18    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  34660]   997 34660    18749     2493   143360        0             0 redis-server
[Sun Jan 10 00:05:35 2021] [  35797]     0 35797     1096       22    57344        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  37303]     0 37303     1060       18    49152        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  37305]   998 37305   760003      445   475136        0             0 gitaly-wrapper
[Sun Jan 10 00:05:35 2021] [  37311]   998 37311   759550    23379  1101824        0             0 gitaly
[Sun Jan 10 00:05:35 2021] [  37587]   998 37587   728803    17562   888832        0             0 ruby
[Sun Jan 10 00:05:35 2021] [  37588]   998 37588   778727    39586  1073152        0             0 ruby
[Sun Jan 10 00:05:35 2021] [  37595]     0 37595     1096       21    57344        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  39290]     0 39290     1060       19    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  39299]   996 39299    12338     2782   131072        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  39329]   996 39329    12392      920   110592        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  39330]   996 39330    12374      831   110592        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  39331]   996 39331    12338      543   102400        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  39332]   996 39332    12473     1007   118784        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  39333]   996 39333     8152      553    94208        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  39334]   996 39334    12451      709   118784        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  40268]     0 40268     1096       20    49152        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  41373]     0 41373     1060       18    49152        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  41375]   998 41375   330686   207154  2560000        0             0 bundle
[Sun Jan 10 00:05:35 2021] [  42757]     0 42757     1096       22    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  43230]     0 43230     1060       19    45056        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  43232]   998 43232    34288     3288   274432        0             0 ruby
[Sun Jan 10 00:05:35 2021] [  43261]   998 43261   891610   247976  6795264        0             0 bundle
[Sun Jan 10 00:05:35 2021] [  44871]     0 44871     1096       21    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  45716]     0 45716     1060       18    49152        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  46557]     0 46557     1096       20    57344        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  47976]     0 47976     1060       18    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  47978]     0 47978    11158      287   114688        0             0 nginx
[Sun Jan 10 00:05:35 2021] [  48665]     0 48665     1096       20    57344        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [   1841]     0  1841     1060       18    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [   2211]     0  2211     1096       20    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [   4501]     0  4501     1060       18    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [   4503]   998  4503   237878   114014  1216512        0             0 gitlab-exporter
[Sun Jan 10 00:05:35 2021] [   5326]     0  5326     1096       21    57344        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [   6712]     0  6712     1060       19    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [   7193]     0  7193     1096       21    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [   9114]     0  9114     1060       18    49152        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  11360]     0 11360     1096       23    57344        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  13794]     0 13794     1060       19    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  15522]     0 15522     1096       21    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  17228]     0 17228     1060       19    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  18181]     0 18181     1096       22    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  20227]     0 20227     1060       18    49152        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  21159]     0 21159     1096       21    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  23043]   996 23043    15811     3066   151552        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  23048]   996 23048    15008     1972   143360        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  23052]   996 23052    15801     3037   151552        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  29245]   996 29245    14829     1534   143360        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  30217]   996 30217    14830     1535   143360        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  30363]   996 30363    15482     2791   147456        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  31925]   998 31925   727762     7223   606208        0             0 gitlab-workhors
[Sun Jan 10 00:05:35 2021] [  32066]   997 32066   640030     2595   475136        0             0 redis_exporter
[Sun Jan 10 00:05:35 2021] [  32276]   992 32276  1962869    75117  2211840        0             0 prometheus
[Sun Jan 10 00:05:35 2021] [  32438]   992 32438   696610     2725   503808        0             0 alertmanager
[Sun Jan 10 00:05:35 2021] [  32620]   996 32620   691549     4155   540672        0             0 postgres_export
[Sun Jan 10 00:05:35 2021] [  32626]   996 32626    21203     6899   180224        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  32693]   992 32693  1181528     5796   720896        0             0 grafana-server
[Sun Jan 10 00:05:35 2021] [  33814]     0 33814     4939       59    86016        0             0 gitlab-ctl
[Sun Jan 10 00:05:35 2021] [  33828]     0 33828    37816     6500   335872        0             0 omnibus-ctl
[Sun Jan 10 00:05:35 2021] [  34377]     0 34377     1126       18    57344        0             0 sh
[Sun Jan 10 00:05:35 2021] [  34380]     0 34380     1595       20    53248        0             0 xargs
[Sun Jan 10 00:05:35 2021] [  34381]     0 34381     1528       31    57344        0             0 tail
[Sun Jan 10 00:05:35 2021] [  35947]     0 35947     2179      145    49152        0             0 bash
[Sun Jan 10 00:05:35 2021] [  23562]   998 23562  9032150  7162562 62947328        0             0 mysqld
[Sun Jan 10 00:05:35 2021] [  44618]     0 44618   137180      481   122880        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  44627]     0 44627    27276      195    77824        0          -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [  44706]  2000 44706  9185087   552797  6643712        0             0 java
[Sun Jan 10 00:05:35 2021] [  41452]     0 41452   155613      480   126976        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  41460]     0 41460    27276      338    73728        0          -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [  41475]     0 41475    27276      255    73728        0          -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [  41494]   999 41494   327076    55842   933888        0             0 mongod
[Sun Jan 10 00:05:35 2021] [  41503]  1000 41503 422008290  1906347 96919552        0             0 java
[Sun Jan 10 00:05:35 2021] [  43137]     0 43137   137180      481   122880        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  43150]     0 43150   230017     1307   176128        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  43164]     0 43164   155613      479   131072        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  43178]     0 43178   137180      483   131072        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  43192]     0 43192   192959     1068   159744        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  43208]     0 43208   155613      478   135168        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  43234]     0 43234    27276      998    77824        0          -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [  43357]  1100 43357      571       20    45056        0             0 tini
[Sun Jan 10 00:05:35 2021] [  43798]  1100 43798     1372       78    49152        0             0 docker-entrypoi
[Sun Jan 10 00:05:35 2021] [  43828]  1100 43828 11417937  1054338 12394496        0             0 java
[Sun Jan 10 00:05:35 2021] [  23584]   106 23584     5178     1676    77824        0             0 haproxy
[Sun Jan 10 00:05:35 2021] [  23691]    33 23691    61703     1991    73728        0             0 nginx
[Sun Jan 10 00:05:35 2021] [  23692]    33 23692    61653     1844    73728        0             0 nginx
[Sun Jan 10 00:05:35 2021] [  24208]     0 24208   180714     7702   192512        0             0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [  24216]     0 24216   180650     6978   204800        0             0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [  24243]     0 24243   180842     7627   200704        0             0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [  24256]     0 24256   180906     7859   204800        0             0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [  24279]     0 24279   180778     7917   204800        0             0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [  24303]     0 24303   181146    10773   221184        0             0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [  48493]   998 48493   460417   215657  3211264        0             0 bundle
[Sun Jan 10 00:05:35 2021] [  48615]   996 48615    14829     1524   143360        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  20931]   998 20931   464001   225770  3223552        0             0 bundle
[Sun Jan 10 00:05:35 2021] [  20977]   996 20977    14829     1493   143360        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  46187]   998 46187   447326   220269  3133440        0             0 bundle
[Sun Jan 10 00:05:35 2021] [  46206]   996 46206    14829     1550   143360        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  27018]   998 27018   452737   226133  3174400        0             0 bundle
[Sun Jan 10 00:05:35 2021] [  27069]   996 27069    14829     1530   143360        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  10726]   996 10726    15611     3032   151552        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  43795]   996 43795    15497     2895   147456        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  39677]     0 39677     1126       17    49152        0             0 gitlab-logrotat
[Sun Jan 10 00:05:35 2021] [  46365]     0 46365     1094       17    53248        0             0 sleep
[Sun Jan 10 00:05:35 2021] [   4140]   996  4140    15522     3019   151552        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  27524]     0 27524   156033      358   143360        0             0 rsyslogd
[Sun Jan 10 00:05:35 2021] [  23561]     0 23561     2785       98    57344        0             0 cron
[Sun Jan 10 00:05:35 2021] [  23572]     0 23572      597       16    45056        0             0 sh
[Sun Jan 10 00:05:35 2021] [  23573]     0 23573     2092       61    45056        0             0 ctl
[Sun Jan 10 00:05:35 2021] [  23839]     0 23839    18370     1260   131072        0             0 php
[Sun Jan 10 00:05:35 2021] [  24848]     0 24848     1358       42    53248        0             0 sshd
[Sun Jan 10 00:05:35 2021] Out of memory: Kill process 23562 (mysqld) score 436 or sacrifice child
[Sun Jan 10 00:05:35 2021] Killed process 23562 (mysqld) total-vm:36128600kB, anon-rss:28650288kB, file-rss:0kB, shmem-rss:0kB
[Sun Jan 10 00:05:37 2021] oom_reaper: reaped process 23562 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

答案1

有一个解释在即使内存可用,系统也无法分配内存

至于如何解决问题,我建议首先检查是否存在内存泄漏或可以停止的未使用进程,一旦您知道从内存的角度来看一切都运行良好,剩下的就是增加物理内存,如果不可能添加一些交换,因为它可以释放一些内存(可能会产生性能成本)

答案2

我问过很多人,但还没有得到好的解决办法。

我高度怀疑这个问题与Linux内核的dentries/inode缓存有关:当可用内存不足时,内核不会尝试回收缓存内存。

每次运行echo 2 > /proc/sys/vm/drop_caches内存free就变成8G左右。然后下降到4G左右,持续几十分钟左右,然后又下降到2G左右,最后可能会变成不到1G,有时还不到400M。

仅对2影响/proc/sys/vm/drop_caches空闲内存,1没有影响。

我不得不使用一个临时解决方案(虽然我认为这很糟糕......),echo 2 > /proc/sys/vm/drop_caches每15分钟运行一次......太悲伤了。

相关内容