我正在尝试找出导致内存过载 OOM killer 执行其操作的原因。这种情况大约每三个小时发生一次，平均实时客户端数量约为 300，我无法找出导致此原因的触发因素，因为 oom-killer 可以从不同的进程调用，从服务器主机进程到 beamium 和 noderig，服务器主机进程也会填满内存，即使客户端可能与服务器断开连接/它们的数量正在减少。正在寻找额外的知识，因为我没有主意了，有什么方法可以进行更深入的调试吗？也许问题出在交换上？免费-h信息
系统日志：
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:366872)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:502088)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:502080)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:501048)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:501040)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:501032)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:501024)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:502104)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:502096)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:367496)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:336808)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:336792)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:336784)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:336768)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:848456)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:845352)
Mar 31 00:26:44 pashamachine kernel: Write-error on swap-device (259:1:464)
Mar 31 00:26:47 pashamachine server[5514]: [N] IPaddress:62588 has been disconnected.
Mar 31 00:26:58 pashamachine server[5514]: [N] IPaddress:60275 has been disconnected.
Mar 31 00:27:11 pashamachine kernel: mysqld invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0
Mar 31 00:27:11 pashamachine kernel: mysqld cpuset=/ mems_allowed=0
Mar 31 00:27:11 pashamachine kernel: CPU: 4 PID: 2517 Comm: mysqld Not tainted 4.9.168-xxxx-std-ipv6-64 #665790
Mar 31 00:27:11 pashamachine kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./E3C242D4U2-2T, BIOS L0.09E 03/14/2019
Mar 31 00:27:11 pashamachine kernel: ffffb5138692b9e0 ffffffffaf679ef7 ffffb5138692bb98 ffff9f160a8cc380
Mar 31 00:27:11 pashamachine kernel: ffffb5138692ba58 ffffffffaf22fd97 0000000000000000 0000000000000000
Mar 31 00:27:11 pashamachine kernel: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Mar 31 00:27:11 pashamachine kernel: Call Trace:
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf679ef7>] dump_stack+0x4d/0x66
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf22fd97>] dump_header+0x76/0x1f1
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf1bf37e>] oom_kill_process+0x20e/0x3e0
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf1bf87b>] out_of_memory+0x11b/0x4a0
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf1c3f44>] __alloc_pages_slowpath+0x994/0xb80
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf1c42e7>] __alloc_pages_nodemask+0x147/0x1d0
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf20d59e>] alloc_pages_current+0x9e/0x150
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf1bad93>] __page_cache_alloc+0xa3/0xe0
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf1bbc38>] ? pagecache_get_page+0x28/0x220
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf1bc12d>] filemap_fault+0x2fd/0x4a0
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf30cec1>] ext4_filemap_fault+0x31/0x50
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf1e7a53>] __do_fault+0xa3/0x1a0
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf1eb43f>] handle_mm_fault+0xb6f/0x1120
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf09ea4d>] __do_page_fault+0x22d/0x450
Mar 31 00:27:11 pashamachine kernel: [<ffffffffaf09ecac>] do_page_fault+0xc/0x10
Mar 31 00:27:11 pashamachine kernel: [<ffffffffafeac282>] page_fault+0x22/0x30
Mar 31 00:27:11 pashamachine kernel: Mem-Info:
Mar 31 00:27:11 pashamachine kernel: active_anon:7602864 inactive_anon:447235 isolated_anon:0
Mar 31 00:27:11 pashamachine kernel: active_file:162 inactive_file:347 isolated_file:0
Mar 31 00:27:11 pashamachine kernel: unevictable:1 dirty:0 writeback:0 unstable:0
Mar 31 00:27:11 pashamachine kernel: slab_reclaimable:5973 slab_unreclaimable:19315
Mar 31 00:27:11 pashamachine kernel: mapped:964 shmem:79951 pagetables:17086 bounce:0
Mar 31 00:27:11 pashamachine kernel: free:50740 free_pcp:387 free_cma:0
Mar 31 00:27:11 pashamachine kernel: Node 0 active_anon:30411456kB inactive_anon:1788940kB active_file:728kB inactive_file:680kB unevictable:4kB isolated(anon):0kB isolated(file):0kB mapped:4028kB dirty:0kB writeback:0kB shmem:319804kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 9269248kB writeback_tmp:0kB unstable:0kB pages_scanned:178 all_unreclaimable? no
Mar 31 00:27:11 pashamachine kernel: Node 0 DMA free:15896kB min:32kB low:44kB high:56kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15896kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Mar 31 00:27:11 pashamachine kernel: lowmem_reserve[]: 0 2017 31924 31924
Mar 31 00:27:11 pashamachine kernel: Node 0 DMA32 free:123888kB min:4268kB low:6332kB high:8396kB active_anon:1767176kB inactive_anon:241356kB active_file:220kB inactive_file:600kB unevictable:0kB writepending:0kB present:2140472kB managed:2140468kB mlocked:0kB slab_reclaimable:252kB slab_unreclaimable:208kB kernel_stack:64kB pagetables:3928kB bounce:0kB free_pcp:764kB local_pcp:0kB free_cma:0kB
Mar 31 00:27:11 pashamachine kernel: lowmem_reserve[]: 0 0 29907 29907
Mar 31 00:27:11 pashamachine kernel: Node 0 Normal free:63224kB min:63280kB low:93904kB high:124528kB active_anon:28644280kB inactive_anon:1547584kB active_file:360kB inactive_file:524kB unevictable:4kB writepending:0kB present:31178752kB managed:30628620kB mlocked:4kB slab_reclaimable:23640kB slab_unreclaimable:77052kB kernel_stack:7088kB pagetables:64416kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Mar 31 00:27:11 pashamachine kernel: lowmem_reserve[]: 0 0 0 0
Mar 31 00:27:11 pashamachine kernel: Node 0 DMA: 2*4kB (U) 2*8kB (U) 0*16kB 2*32kB (U) 3*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB
Mar 31 00:27:11 pashamachine kernel: Node 0 DMA32: 14*4kB (UME) 50*8kB (UM) 88*16kB (UME) 124*32kB (UME) 68*64kB (UME) 48*128kB (UME) 19*256kB (UE) 12*512kB (UE) 7*1024kB (UE) 2*2048kB (UM) 21*4096kB (UMH) = 124616kB
Mar 31 00:27:11 pashamachine kernel: Node 0 Normal: 473*4kB (UME) 261*8kB (UMEH) 269*16kB (UMEH) 234*32kB (UMEH) 170*64kB (UMEH) 81*128kB (UMEH) 53*256kB (UMEH) 23*512kB (UME) 1*1024kB (H) 0*2048kB 0*4096kB = 63388kB
Mar 31 00:27:11 pashamachine kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Mar 31 00:27:11 pashamachine kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Mar 31 00:27:11 pashamachine kernel: 231789 total pagecache pages
Mar 31 00:27:11 pashamachine kernel: 151263 pages in swap cache
Mar 31 00:27:11 pashamachine kernel: Swap cache stats: add 839809, delete 688546, find 3766057/3789646
Mar 31 00:27:11 pashamachine kernel: Free swap = 0kB
Mar 31 00:27:11 pashamachine kernel: Total swap = 1046520kB
Mar 31 00:27:11 pashamachine kernel: 8333804 pages RAM
Mar 31 00:27:11 pashamachine kernel: 0 pages HighMem/MovableOnly
Mar 31 00:27:11 pashamachine kernel: 137558 pages reserved
Mar 31 00:27:11 pashamachine kernel: 0 pages hwpoisoned
Mar 31 00:27:11 pashamachine kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Mar 31 00:27:11 pashamachine kernel: [ 480] 0 480 9091 900 15 3 79 0 systemd-journal
Mar 31 00:27:11 pashamachine kernel: [ 508] 0 508 4697 14 9 3 289 -1000 systemd-udevd
Mar 31 00:27:11 pashamachine kernel: [ 513] 101 513 3563 57 10 3 105 0 systemd-network
Mar 31 00:27:11 pashamachine kernel: [ 681] 0 681 810 62 5 3 28 0 mdadm
Mar 31 00:27:11 pashamachine kernel: [ 743] 100 743 21772 29 12 3 99 0 systemd-timesyn
Mar 31 00:27:11 pashamachine kernel: [ 773] 106 773 255875 8927 77 4 4618 0 named
Mar 31 00:27:11 pashamachine kernel: [ 780] 0 780 2561 65 8 3 63 0 irqbalance
Mar 31 00:27:11 pashamachine kernel: [ 798] 0 798 2437 25 8 3 42 0 cron
Mar 31 00:27:11 pashamachine kernel: [ 802] 0 802 2964 39 9 3 105 0 systemd-logind
Mar 31 00:27:11 pashamachine kernel: [ 806] 107 806 2183 120 7 3 44 -900 dbus-daemon
Mar 31 00:27:11 pashamachine kernel: [ 812] 108 812 23477 2583 50 4 10441 0 beamium
Mar 31 00:27:11 pashamachine kernel: [ 837] 0 837 6834 32 16 3 157 -1000 sshd
Mar 31 00:27:11 pashamachine kernel: [ 842] 0 842 498754 22631 120 7 4345 0 noderig
Mar 31 00:27:11 pashamachine kernel: [ 846] 0 846 1656 0 7 3 30 0 agetty
Mar 31 00:27:11 pashamachine kernel: [ 848] 0 848 1575 0 7 3 116 0 login
Mar 31 00:27:11 pashamachine kernel: [ 850] 0 850 1656 0 8 3 30 0 agetty
Mar 31 00:27:11 pashamachine kernel: [ 1679] 0 1679 2013 1 6 3 145 0 screen
Mar 31 00:27:11 pashamachine kernel: [ 1680] 0 1680 2059 1 7 3 135 0 bash
Mar 31 00:27:11 pashamachine kernel: [ 2296] 0 2296 2059 1 7 3 144 0 bash
Mar 31 00:27:11 pashamachine kernel: [30306] 0 30306 2015 1 7 3 149 0 screen
Mar 31 00:27:11 pashamachine kernel: [30307] 0 30307 2059 1 7 3 137 0 bash
Mar 31 00:27:11 pashamachine kernel: [30308] 109 30308 2983591 1376774 3615 16 237632 0 mysqld
Mar 31 00:27:11 pashamachine kernel: [ 6887] 0 6887 57103 401 16 4 83 0 rsyslogd
Mar 31 00:27:11 pashamachine kernel: [ 5514] 0 5514 8497142 6404668 12981 48 0 0 server
Mar 31 00:27:11 pashamachine kernel: [ 7436] 0 7436 6954 229 17 3 0 0 sshd
Mar 31 00:27:11 pashamachine kernel: [ 7443] 0 7443 608 24 5 3 0 0 sftp-server
Mar 31 00:27:11 pashamachine kernel: [ 7753] 0 7753 6955 253 17 3 0 0 sshd
Mar 31 00:27:11 pashamachine kernel: [ 7759] 0 7759 2059 140 7 3 0 0 bash
Mar 31 00:27:11 pashamachine kernel: [ 8685] 0 8685 3769 206 11 3 0 0 top
Mar 31 00:27:11 pashamachine kernel: Out of memory: Kill process 5514 (server) score 736 or sacrifice child
Mar 31 00:27:11 pashamachine kernel: Killed process 5514 (server) total-vm:33988568kB, anon-rss:25618672kB, file-rss:0kB, shmem-rss:0kB