我随机遇到 OOM 问题（大约每月一次），但似乎有足够的可用内存。
我不明白原因以及如何解决它......
vm.min_free_kbytes = 90112
Linux xxx-alpha 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux
slabtop
Active / Total Objects (% used) : 45902519 / 54189870 (84.7%)
Active / Total Slabs (% used) : 1260240 / 1260240 (100.0%)
Active / Total Caches (% used) : 151 / 187 (80.7%)
Active / Total Size (% used) : 9241442.17K / 10856556.48K (85.1%)
Minimum / Average / Maximum Object : 0.01K / 0.20K / 23.25K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
43615236 37957660 87% 0.19K 1038467 42 8307736K dentry
2180640 1579066 72% 0.07K 38940 56 155760K range_seg_cache
706048 538395 76% 0.01K 1379 512 5516K kmalloc-8
644992 395397 61% 0.03K 5039 128 20156K kmalloc-32
635150 294932 46% 0.32K 12963 49 207408K arc_buf_hdr_t_full
546840 332510 60% 0.09K 13020 42 52080K kmalloc-96
dmesg
[Sun Jan 10 00:05:35 2021] elasticsearch[7 invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
[Sun Jan 10 00:05:35 2021] elasticsearch[7 cpuset=67e9bddb581923bf7602fbcce820867ff13dd19355d458112bee6300e0a2cbd8 mems_allowed=0-1
[Sun Jan 10 00:05:35 2021] CPU: 45 PID: 9599 Comm: elasticsearch[7 Tainted: P OE 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
[Sun Jan 10 00:05:35 2021] Hardware name: Ericsson CRU 0101/S2BS-MB, BIOS S2B_3B20 05/30/2019
[Sun Jan 10 00:05:35 2021] Call Trace:
[Sun Jan 10 00:05:35 2021] dump_stack+0x5c/0x80
[Sun Jan 10 00:05:35 2021] dump_header+0x6b/0x283
[Sun Jan 10 00:05:35 2021] ? do_try_to_free_pages+0x2ec/0x370
[Sun Jan 10 00:05:35 2021] oom_kill_process.cold.30+0xb/0x1cf
[Sun Jan 10 00:05:35 2021] ? oom_badness+0x23/0x140
[Sun Jan 10 00:05:35 2021] out_of_memory+0x1a5/0x430
[Sun Jan 10 00:05:35 2021] __alloc_pages_slowpath+0xbd8/0xcb0
[Sun Jan 10 00:05:35 2021] __alloc_pages_nodemask+0x28b/0x2b0
[Sun Jan 10 00:05:35 2021] filemap_fault+0x3bd/0x780
[Sun Jan 10 00:05:35 2021] ? filemap_map_pages+0x1ed/0x3a0
[Sun Jan 10 00:05:35 2021] __do_fault+0x36/0x130
[Sun Jan 10 00:05:35 2021] __handle_mm_fault+0xe6c/0x1270
[Sun Jan 10 00:05:35 2021] handle_mm_fault+0xd6/0x200
[Sun Jan 10 00:05:35 2021] __do_page_fault+0x249/0x4f0
[Sun Jan 10 00:05:35 2021] ? page_fault+0x8/0x30
[Sun Jan 10 00:05:35 2021] page_fault+0x1e/0x30
[Sun Jan 10 00:05:35 2021] RIP: 0033:0x7f68612bc435
[Sun Jan 10 00:05:35 2021] Code: 1f 00 00 49 c1 e0 03 41 8b 48 18 45 8b 48 1c 41 3b c9 0f 8d 47 21 00 00 4d 8b 50 10 8b d9 ff c3 41 89 58 18 49 8b fa 4c 63 d1 <46> 0f be 14 17 45 85 d2 0f 8c 07 0d 00 00 41 8b ca 83 e1 01 41 88
[Sun Jan 10 00:05:35 2021] RSP: 002b:00007e876f44e050 EFLAGS: 00010207
[Sun Jan 10 00:05:35 2021] RAX: 0000000000000002 RBX: 00000000000e37a6 RCX: 00000000000e37a5
[Sun Jan 10 00:05:35 2021] RDX: 0000000684825b90 RSI: 000000000032bce7 RDI: 00007f625cad5441
[Sun Jan 10 00:05:35 2021] RBP: 00000000d3887259 R08: 0000000684825bd0 R09: 00000000002f8d73
[Sun Jan 10 00:05:35 2021] R10: 00000000000e37a5 R11: 0000000684825b90 R12: 0000000000000000
[Sun Jan 10 00:05:35 2021] R13: 00000006829fb780 R14: 00000000d0904b72 R15: 00007f66dc05f800
[Sun Jan 10 00:05:35 2021] Mem-Info:
[Sun Jan 10 00:05:35 2021] active_anon:12218942 inactive_anon:93895 isolated_anon:0
active_file:1203 inactive_file:1345 isolated_file:6
unevictable:0 dirty:15 writeback:21 unstable:0
slab_reclaimable:1710574 slab_unreclaimable:1562977
mapped:26078 shmem:177912 pagetables:59851 bounce:0
free:49188 free_pcp:9835 free_cma:0
[Sun Jan 10 00:05:35 2021] Node 0 active_anon:31064964kB inactive_anon:48776kB active_file:284kB inactive_file:392kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:18116kB dirty:4kB writeback:4kB shmem:128568kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[Sun Jan 10 00:05:35 2021] Node 1 active_anon:17810804kB inactive_anon:326804kB active_file:4528kB inactive_file:4988kB unevictable:0kB isolated(anon):0kB isolated(file):24kB mapped:86196kB dirty:56kB writeback:80kB shmem:583080kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[Sun Jan 10 00:05:35 2021] Node 0 DMA free:15904kB min:20kB low:32kB high:44kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 1848 32052 32052 32052
[Sun Jan 10 00:05:35 2021] Node 0 DMA32 free:123288kB min:2556kB low:4420kB high:6284kB active_anon:1651524kB inactive_anon:2372kB active_file:116kB inactive_file:56kB unevictable:0kB writepending:0kB present:1959556kB managed:1892668kB mlocked:0kB kernel_stack:212kB pagetables:2500kB bounce:0kB free_pcp:248kB local_pcp:0kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 0 30203 30203 30203
[Sun Jan 10 00:05:35 2021] Node 0 Normal free:22820kB min:42348kB low:73276kB high:104204kB active_anon:29413440kB inactive_anon:46404kB active_file:156kB inactive_file:336kB unevictable:0kB writepending:8kB present:31457280kB managed:30928584kB mlocked:0kB kernel_stack:21416kB pagetables:78456kB bounce:0kB free_pcp:6260kB local_pcp:0kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 0 0 0 0
[Sun Jan 10 00:05:35 2021] Node 1 Normal free:34740kB min:45184kB low:78184kB high:111184kB active_anon:17810804kB inactive_anon:326804kB active_file:5176kB inactive_file:5044kB unevictable:0kB writepending:136kB present:33554432kB managed:33004524kB mlocked:0kB kernel_stack:47908kB pagetables:158448kB bounce:0kB free_pcp:32832kB local_pcp:1324kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 0 0 0 0
[Sun Jan 10 00:05:35 2021] Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 3*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB
[Sun Jan 10 00:05:35 2021] Node 0 DMA32: 99*4kB (UMH) 15*8kB (UMH) 3483*16kB (UMH) 2052*32kB (UMEH) 3*64kB (MH) 1*128kB (H) 3*256kB (H) 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 123508kB
[Sun Jan 10 00:05:35 2021] Node 0 Normal: 882*4kB (UMEH) 33*8kB (MEH) 34*16kB (ME) 751*32kB (UME) 3*64kB (UH) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 28560kB
[Sun Jan 10 00:05:35 2021] Node 1 Normal: 8333*4kB (MH) 26*8kB (H) 1*16kB (H) 1*32kB (H) 3*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 33780kB
[Sun Jan 10 00:05:35 2021] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Sun Jan 10 00:05:35 2021] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sun Jan 10 00:05:35 2021] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Sun Jan 10 00:05:35 2021] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sun Jan 10 00:05:35 2021] 181392 total pagecache pages
[Sun Jan 10 00:05:35 2021] 0 pages in swap cache
[Sun Jan 10 00:05:35 2021] Swap cache stats: add 0, delete 0, find 0/0
[Sun Jan 10 00:05:35 2021] Free swap = 0kB
[Sun Jan 10 00:05:35 2021] Total swap = 0kB
[Sun Jan 10 00:05:35 2021] 16746816 pages RAM
[Sun Jan 10 00:05:35 2021] 0 pages HighMem/MovableOnly
[Sun Jan 10 00:05:35 2021] 286396 pages reserved
[Sun Jan 10 00:05:35 2021] 0 pages hwpoisoned
[Sun Jan 10 00:05:35 2021] Tasks state (memory values in pages):
[Sun Jan 10 00:05:35 2021] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[Sun Jan 10 00:05:35 2021] [ 611] 0 611 60450 40093 503808 0 0 systemd-journal
[Sun Jan 10 00:05:35 2021] [ 635] 0 635 5616 348 61440 0 -1000 systemd-udevd
[Sun Jan 10 00:05:35 2021] [ 814] 0 814 770 56 45056 0 0 mdadm
[Sun Jan 10 00:05:35 2021] [ 9990] 0 9990 41684 221 69632 0 0 zed
[Sun Jan 10 00:05:35 2021] [ 9991] 0 9991 2557 73 61440 0 0 cron
[Sun Jan 10 00:05:35 2021] [ 9993] 104 9993 2222 187 57344 0 -900 dbus-daemon
[Sun Jan 10 00:05:35 2021] [ 9994] 0 9994 4991 386 77824 0 0 systemd-logind
[Sun Jan 10 00:05:35 2021] [ 10013] 0 10013 200898 2220 65536 0 0 supervisord
[Sun Jan 10 00:05:35 2021] [ 10020] 0 10020 28023 1915 114688 0 0 unattended-upgr
[Sun Jan 10 00:05:35 2021] [ 10028] 0 10028 1405091 6815 843776 0 0 containerd
[Sun Jan 10 00:05:35 2021] [ 10033] 0 10033 1307161 15321 995328 0 -500 dockerd
[Sun Jan 10 00:05:35 2021] [ 10040] 0 10040 3963 236 69632 0 -1000 sshd
[Sun Jan 10 00:05:35 2021] [ 10074] 0 10074 4934 1407 73728 0 0 haproxy
[Sun Jan 10 00:05:35 2021] [ 12823] 0 12823 1403 28 45056 0 0 agetty
[Sun Jan 10 00:05:35 2021] [ 13945] 107 13945 4935 210 81920 0 0 exim4
[Sun Jan 10 00:05:35 2021] [ 19622] 0 19622 5382 445 81920 0 0 systemd
[Sun Jan 10 00:05:35 2021] [ 19623] 0 19623 5779 630 73728 0 0 (sd-pam)
[Sun Jan 10 00:05:35 2021] [ 24193] 0 24193 31559 4572 143360 0 0 prom-node-expor
[Sun Jan 10 00:05:35 2021] [ 9763] 0 9763 61147 1433 65536 0 0 nginx
[Sun Jan 10 00:05:35 2021] [ 4864] 0 4864 58959 222 94208 0 0 polkitd
[Sun Jan 10 00:05:35 2021] [ 19619] 0 19619 155613 481 126976 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 19633] 0 19633 27276 212 73728 0 -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [ 19686] 472 19686 46091 7685 315392 0 0 grafana-server
[Sun Jan 10 00:05:35 2021] [ 34921] 0 34921 820113 471 462848 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 34929] 0 34929 26924 202 73728 0 -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [ 34948] 65534 34948 5391852 43302 21938176 0 0 prometheus
[Sun Jan 10 00:05:35 2021] [ 12393] 108 12393 19145 168 65536 0 0 ntpd
[Sun Jan 10 00:05:35 2021] [ 5862] 0 5862 4947 2284 90112 0 0 tmux: server
[Sun Jan 10 00:05:35 2021] [ 5863] 0 5863 2209 151 61440 0 0 bash
[Sun Jan 10 00:05:35 2021] [ 33048] 0 33048 2249 215 57344 0 0 bash
[Sun Jan 10 00:05:35 2021] [ 29476] 0 29476 2201 154 53248 0 0 bash
[Sun Jan 10 00:05:35 2021] [ 31493] 0 31493 597 46 45056 0 0 mysqld_safe
[Sun Jan 10 00:05:35 2021] [ 45444] 0 45444 5315 1563 86016 0 0 mysql
[Sun Jan 10 00:05:35 2021] [ 13606] 0 13606 2209 156 61440 0 0 bash
[Sun Jan 10 00:05:35 2021] [ 35429] 0 35429 2179 152 49152 0 0 bash
[Sun Jan 10 00:05:35 2021] [ 26987] 0 26987 137180 482 118784 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 27040] 0 27040 985866 1166 544768 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 27049] 0 27049 27276 940 86016 0 -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [ 27074] 0 27074 4947 74 81920 0 0 wrapper
[Sun Jan 10 00:05:35 2021] [ 28817] 0 28817 1098 22 53248 0 0 runsvdir
[Sun Jan 10 00:05:35 2021] [ 29114] 0 29114 1060 19 49152 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 29117] 0 29117 1096 20 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 29120] 0 29120 16378 178 167936 0 0 sshd
[Sun Jan 10 00:05:35 2021] [ 34657] 0 34657 1060 18 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 34660] 997 34660 18749 2493 143360 0 0 redis-server
[Sun Jan 10 00:05:35 2021] [ 35797] 0 35797 1096 22 57344 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 37303] 0 37303 1060 18 49152 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 37305] 998 37305 760003 445 475136 0 0 gitaly-wrapper
[Sun Jan 10 00:05:35 2021] [ 37311] 998 37311 759550 23379 1101824 0 0 gitaly
[Sun Jan 10 00:05:35 2021] [ 37587] 998 37587 728803 17562 888832 0 0 ruby
[Sun Jan 10 00:05:35 2021] [ 37588] 998 37588 778727 39586 1073152 0 0 ruby
[Sun Jan 10 00:05:35 2021] [ 37595] 0 37595 1096 21 57344 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 39290] 0 39290 1060 19 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 39299] 996 39299 12338 2782 131072 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 39329] 996 39329 12392 920 110592 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 39330] 996 39330 12374 831 110592 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 39331] 996 39331 12338 543 102400 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 39332] 996 39332 12473 1007 118784 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 39333] 996 39333 8152 553 94208 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 39334] 996 39334 12451 709 118784 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 40268] 0 40268 1096 20 49152 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 41373] 0 41373 1060 18 49152 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 41375] 998 41375 330686 207154 2560000 0 0 bundle
[Sun Jan 10 00:05:35 2021] [ 42757] 0 42757 1096 22 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 43230] 0 43230 1060 19 45056 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 43232] 998 43232 34288 3288 274432 0 0 ruby
[Sun Jan 10 00:05:35 2021] [ 43261] 998 43261 891610 247976 6795264 0 0 bundle
[Sun Jan 10 00:05:35 2021] [ 44871] 0 44871 1096 21 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 45716] 0 45716 1060 18 49152 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 46557] 0 46557 1096 20 57344 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 47976] 0 47976 1060 18 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 47978] 0 47978 11158 287 114688 0 0 nginx
[Sun Jan 10 00:05:35 2021] [ 48665] 0 48665 1096 20 57344 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 1841] 0 1841 1060 18 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 2211] 0 2211 1096 20 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 4501] 0 4501 1060 18 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 4503] 998 4503 237878 114014 1216512 0 0 gitlab-exporter
[Sun Jan 10 00:05:35 2021] [ 5326] 0 5326 1096 21 57344 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 6712] 0 6712 1060 19 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 7193] 0 7193 1096 21 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 9114] 0 9114 1060 18 49152 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 11360] 0 11360 1096 23 57344 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 13794] 0 13794 1060 19 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 15522] 0 15522 1096 21 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 17228] 0 17228 1060 19 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 18181] 0 18181 1096 22 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 20227] 0 20227 1060 18 49152 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 21159] 0 21159 1096 21 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 23043] 996 23043 15811 3066 151552 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 23048] 996 23048 15008 1972 143360 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 23052] 996 23052 15801 3037 151552 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 29245] 996 29245 14829 1534 143360 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 30217] 996 30217 14830 1535 143360 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 30363] 996 30363 15482 2791 147456 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 31925] 998 31925 727762 7223 606208 0 0 gitlab-workhors
[Sun Jan 10 00:05:35 2021] [ 32066] 997 32066 640030 2595 475136 0 0 redis_exporter
[Sun Jan 10 00:05:35 2021] [ 32276] 992 32276 1962869 75117 2211840 0 0 prometheus
[Sun Jan 10 00:05:35 2021] [ 32438] 992 32438 696610 2725 503808 0 0 alertmanager
[Sun Jan 10 00:05:35 2021] [ 32620] 996 32620 691549 4155 540672 0 0 postgres_export
[Sun Jan 10 00:05:35 2021] [ 32626] 996 32626 21203 6899 180224 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 32693] 992 32693 1181528 5796 720896 0 0 grafana-server
[Sun Jan 10 00:05:35 2021] [ 33814] 0 33814 4939 59 86016 0 0 gitlab-ctl
[Sun Jan 10 00:05:35 2021] [ 33828] 0 33828 37816 6500 335872 0 0 omnibus-ctl
[Sun Jan 10 00:05:35 2021] [ 34377] 0 34377 1126 18 57344 0 0 sh
[Sun Jan 10 00:05:35 2021] [ 34380] 0 34380 1595 20 53248 0 0 xargs
[Sun Jan 10 00:05:35 2021] [ 34381] 0 34381 1528 31 57344 0 0 tail
[Sun Jan 10 00:05:35 2021] [ 35947] 0 35947 2179 145 49152 0 0 bash
[Sun Jan 10 00:05:35 2021] [ 23562] 998 23562 9032150 7162562 62947328 0 0 mysqld
[Sun Jan 10 00:05:35 2021] [ 44618] 0 44618 137180 481 122880 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 44627] 0 44627 27276 195 77824 0 -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [ 44706] 2000 44706 9185087 552797 6643712 0 0 java
[Sun Jan 10 00:05:35 2021] [ 41452] 0 41452 155613 480 126976 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 41460] 0 41460 27276 338 73728 0 -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [ 41475] 0 41475 27276 255 73728 0 -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [ 41494] 999 41494 327076 55842 933888 0 0 mongod
[Sun Jan 10 00:05:35 2021] [ 41503] 1000 41503 422008290 1906347 96919552 0 0 java
[Sun Jan 10 00:05:35 2021] [ 43137] 0 43137 137180 481 122880 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 43150] 0 43150 230017 1307 176128 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 43164] 0 43164 155613 479 131072 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 43178] 0 43178 137180 483 131072 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 43192] 0 43192 192959 1068 159744 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 43208] 0 43208 155613 478 135168 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 43234] 0 43234 27276 998 77824 0 -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [ 43357] 1100 43357 571 20 45056 0 0 tini
[Sun Jan 10 00:05:35 2021] [ 43798] 1100 43798 1372 78 49152 0 0 docker-entrypoi
[Sun Jan 10 00:05:35 2021] [ 43828] 1100 43828 11417937 1054338 12394496 0 0 java
[Sun Jan 10 00:05:35 2021] [ 23584] 106 23584 5178 1676 77824 0 0 haproxy
[Sun Jan 10 00:05:35 2021] [ 23691] 33 23691 61703 1991 73728 0 0 nginx
[Sun Jan 10 00:05:35 2021] [ 23692] 33 23692 61653 1844 73728 0 0 nginx
[Sun Jan 10 00:05:35 2021] [ 24208] 0 24208 180714 7702 192512 0 0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [ 24216] 0 24216 180650 6978 204800 0 0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [ 24243] 0 24243 180842 7627 200704 0 0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [ 24256] 0 24256 180906 7859 204800 0 0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [ 24279] 0 24279 180778 7917 204800 0 0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [ 24303] 0 24303 181146 10773 221184 0 0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [ 48493] 998 48493 460417 215657 3211264 0 0 bundle
[Sun Jan 10 00:05:35 2021] [ 48615] 996 48615 14829 1524 143360 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 20931] 998 20931 464001 225770 3223552 0 0 bundle
[Sun Jan 10 00:05:35 2021] [ 20977] 996 20977 14829 1493 143360 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 46187] 998 46187 447326 220269 3133440 0 0 bundle
[Sun Jan 10 00:05:35 2021] [ 46206] 996 46206 14829 1550 143360 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 27018] 998 27018 452737 226133 3174400 0 0 bundle
[Sun Jan 10 00:05:35 2021] [ 27069] 996 27069 14829 1530 143360 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 10726] 996 10726 15611 3032 151552 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 43795] 996 43795 15497 2895 147456 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 39677] 0 39677 1126 17 49152 0 0 gitlab-logrotat
[Sun Jan 10 00:05:35 2021] [ 46365] 0 46365 1094 17 53248 0 0 sleep
[Sun Jan 10 00:05:35 2021] [ 4140] 996 4140 15522 3019 151552 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 27524] 0 27524 156033 358 143360 0 0 rsyslogd
[Sun Jan 10 00:05:35 2021] [ 23561] 0 23561 2785 98 57344 0 0 cron
[Sun Jan 10 00:05:35 2021] [ 23572] 0 23572 597 16 45056 0 0 sh
[Sun Jan 10 00:05:35 2021] [ 23573] 0 23573 2092 61 45056 0 0 ctl
[Sun Jan 10 00:05:35 2021] [ 23839] 0 23839 18370 1260 131072 0 0 php
[Sun Jan 10 00:05:35 2021] [ 24848] 0 24848 1358 42 53248 0 0 sshd
[Sun Jan 10 00:05:35 2021] Out of memory: Kill process 23562 (mysqld) score 436 or sacrifice child
[Sun Jan 10 00:05:35 2021] Killed process 23562 (mysqld) total-vm:36128600kB, anon-rss:28650288kB, file-rss:0kB, shmem-rss:0kB
[Sun Jan 10 00:05:37 2021] oom_reaper: reaped process 23562 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
答案1
有一个解释在即使内存可用，系统也无法分配内存。
至于如何解决问题，我建议首先检查是否存在内存泄漏或可以停止的未使用进程，一旦您知道从内存的角度来看一切都运行良好，剩下的就是增加物理内存，如果不可能添加一些交换，因为它可以释放一些内存（可能会产生性能成本）
答案2
我问过很多人，但还没有得到好的解决办法。
我高度怀疑这个问题与Linux内核的dentries/inode缓存有关：当可用内存不足时，内核不会尝试回收缓存内存。
每次运行echo 2 > /proc/sys/vm/drop_caches内存free就变成8G左右。然后下降到4G左右，持续几十分钟左右，然后又下降到2G左右，最后可能会变成不到1G，有时还不到400M。
仅对2影响/proc/sys/vm/drop_caches空闲内存，1没有影响。
我不得不使用一个临时解决方案（虽然我认为这很糟糕......），echo 2 > /proc/sys/vm/drop_caches每15分钟运行一次......太悲伤了。