我花了好长时间才弄清楚。Apache 每隔几天就会挂起大约 20 分钟，然后“恢复正常”。这种情况发生在中午，也发生在半夜。该服务器是一个强大的 Web 服务器，有 4 个 CPU、8GB RAM 和另外 12GB 交换空间。我在错误日志中看不到任何突出的内容。有人能从这些日志中看到任何表明问题因为我所看到的一切都像问题的结果或者症状！
系统日志
Dec 10 05:25:02 admin kernel: [397885.197196] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23622 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:03 admin kernel: [397886.194931] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23623 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:05 admin kernel: [397888.198941] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23624 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:08 admin kernel: [397891.641472] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=967 DF PROTO=TCP SPT=52268 DPT=80 WINDOW=2045 RES=0x00 ACK FIN URGP=0
Dec 10 05:25:08 admin kernel: [397891.974137] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=18.206.39.189 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=116 ID=28012 DF PROTO=TCP SPT=50916 DPT=443 WINDOW=0 RES=0x00 ACK RST URGP=0
Dec 10 05:25:08 admin kernel: [397891.974230] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=18.206.39.189 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=116 ID=28013 DF PROTO=TCP SPT=50866 DPT=80 WINDOW=1021 RES=0x00 ACK FIN URGP=0
Dec 10 05:25:09 admin kernel: [397892.210857] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23625 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:09 admin CRON[8325]: (root) CMD (sh /etc/apache2/websitesCron/apacheTest.sh)
Dec 10 05:25:09 admin CRON[8326]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)
Dec 10 05:25:10 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 388
Dec 10 05:25:10 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 388
Dec 10 05:25:17 admin kernel: [397900.226738] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23626 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:25 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 401
Dec 10 05:25:25 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 401
Dec 10 05:25:27 admin kernel: [397910.285120] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=985 DF PROTO=TCP SPT=52268 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0
Dec 10 05:25:33 admin kernel: [397916.242478] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23627 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:34 admin sendmail[8390]: 1BABPXbK008390: from=root, size=425, class=0, nrcpts=1, msgid=<[email protected]>, bodytype=8BITMIME, relay=root@localhost
Dec 10 05:25:34 admin sendmail[8390]: 1BABPXbK008390: to=root, delay=00:00:01, mailer=relay, pri=30425, stat=queued
Dec 10 05:25:40 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 413
Dec 10 05:25:40 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 413
Dec 10 05:25:55 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 419
Dec 10 05:25:55 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 419
Dec 10 05:26:04 admin CRON[8459]: (root) CMD (cp /var/spool/cron/crontabs/root /var/www/crontab/root && chmod 777 /var/www/crontab/root && chown zak:zak /var/www/crontab/root)
Dec 10 05:26:05 admin kernel: [397948.305893] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23628 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:26:10 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 423
Dec 10 05:26:10 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 423
Dec 10 05:26:25 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 428
Dec 10 05:26:25 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 428
Dec 10 05:26:26 admin kernel: [397969.985342] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=990 DF PROTO=TCP SPT=62843 DPT=80 WINDOW=2045 RES=0x00 ACK FIN URGP=0
Dec 10 05:26:29 admin kernel: [397972.391234] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=995 DF PROTO=TCP SPT=62843 DPT=80 WINDOW=2045 RES=0x00 ACK FIN URGP=0
Dec 10 05:26:34 admin kernel: [397977.203821] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=999 DF PROTO=TCP SPT=62843 DPT=80 WINDOW=2045 RES=0x00 ACK FIN URGP=0
Dec 10 05:26:40 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 436
Dec 10 05:26:40 admin kernel: [397983.346629] apache2 invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0
Dec 10 05:26:40 admin kernel: [397983.346635] apache2 cpuset=/ mems_allowed=0
Dec 10 05:26:40 admin kernel: [397983.346644] CPU: 0 PID: 8270 Comm: apache2 Not tainted 4.4.0-119-generic #143-Ubuntu
Dec 10 05:26:40 admin kernel: [397983.346646] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Dec 10 05:26:40 admin kernel: [397983.346649] 0000000000000286 553e9d17649eeffd ffff8800138a7b10 ffffffff81400443
Dec 10 05:26:40 admin kernel: [397983.346652] ffff8800138a7cc8 ffff8800b8e5aa00 ffff8800138a7b80 ffffffff8121086e
Dec 10 05:26:40 admin kernel: [397983.346655] ffff88023fc1ad70 ffff88023fc1ad60 ffffea0006a80b00 0000000100000001
Dec 10 05:26:40 admin kernel: [397983.346657] Call Trace:
Dec 10 05:26:40 admin kernel: [397983.346670] [<ffffffff81400443>] dump_stack+0x63/0x90
Dec 10 05:26:40 admin kernel: [397983.346677] [<ffffffff8121086e>] dump_header+0x5a/0x1c5
Dec 10 05:26:40 admin kernel: [397983.346682] [<ffffffff81196f32>] oom_kill_process+0x202/0x3c0
Dec 10 05:26:40 admin kernel: [397983.346685] [<ffffffff81197359>] out_of_memory+0x219/0x460
Dec 10 05:26:40 admin kernel: [397983.346689] [<ffffffff8119d3a5>] __alloc_pages_slowpath.constprop.88+0x965/0xb00
Dec 10 05:26:40 admin kernel: [397983.346692] [<ffffffff8119d7c8>] __alloc_pages_nodemask+0x288/0x2a0
Dec 10 05:26:40 admin kernel: [397983.346694] [<ffffffff8119d87b>] alloc_kmem_pages_node+0x4b/0xc0
Dec 10 05:26:40 admin kernel: [397983.346699] [<ffffffff8108077e>] copy_process+0x1be/0x1bb0
Dec 10 05:26:40 admin kernel: [397983.346703] [<ffffffff811a44f7>] ? lru_cache_add_active_or_unevictable+0x27/0xa0
Dec 10 05:26:40 admin kernel: [397983.346707] [<ffffffff811c6178>] ? handle_mm_fault+0xcc8/0x1820
Dec 10 05:26:40 admin kernel: [397983.346709] [<ffffffff81082300>] _do_fork+0x80/0x360
Dec 10 05:26:40 admin kernel: [397983.346712] [<ffffffff81082689>] SyS_clone+0x19/0x20
Dec 10 05:26:40 admin kernel: [397983.346717] [<ffffffff8184f708>] entry_SYSCALL_64_fastpath+0x1c/0xbb
Dec 10 05:26:40 admin kernel: [397983.346718] Mem-Info:
Dec 10 05:26:40 admin kernel: [397983.346723] active_anon:1463690 inactive_anon:290454 isolated_anon:132
Dec 10 05:26:40 admin kernel: [397983.346723] active_file:17208 inactive_file:13369 isolated_file:0
Dec 10 05:26:40 admin kernel: [397983.346723] unevictable:913 dirty:0 writeback:987 unstable:0
Dec 10 05:26:40 admin kernel: [397983.346723] slab_reclaimable:19387 slab_unreclaimable:41949
Dec 10 05:26:40 admin kernel: [397983.346723] mapped:25866 shmem:18736 pagetables:141112 bounce:0
Dec 10 05:26:40 admin kernel: [397983.346723] free:26269 free_pcp:0 free_cma:0
Dec 10 05:26:40 admin kernel: [397983.346728] Node 0 DMA free:15864kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 10 05:26:40 admin kernel: [397983.346734] lowmem_reserve[]: 0 2937 7928 7928 7928
Dec 10 05:26:40 admin kernel: [397983.346737] Node 0 DMA32 free:44712kB min:24988kB low:31232kB high:37480kB active_anon:2037256kB inactive_anon:515680kB active_file:24624kB inactive_file:19936kB unevictable:1228kB isolated(anon):348kB isolated(file):0kB present:3129216kB managed:3048436kB mlocked:1228kB dirty:0kB writeback:2620kB mapped:38304kB shmem:28716kB slab_reclaimable:25488kB slab_unreclaimable:70356kB kernel_stack:15136kB pagetables:261600kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec 10 05:26:40 admin kernel: [397983.346743] lowmem_reserve[]: 0 0 4990 4990 4990
Dec 10 05:26:40 admin kernel: [397983.346745] Node 0 Normal free:44500kB min:42460kB low:53072kB high:63688kB active_anon:3817504kB inactive_anon:646136kB active_file:44208kB inactive_file:33540kB unevictable:2424kB isolated(anon):180kB isolated(file):0kB present:5242880kB managed:5110492kB mlocked:2424kB dirty:0kB writeback:1328kB mapped:65160kB shmem:46228kB slab_reclaimable:52060kB slab_unreclaimable:97432kB kernel_stack:13728kB pagetables:302848kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec 10 05:26:40 admin kernel: [397983.346750] lowmem_reserve[]: 0 0 0 0 0
Dec 10 05:26:40 admin kernel: [397983.346752] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15864kB
Dec 10 05:26:40 admin kernel: [397983.346762] Node 0 DMA32: 6731*4kB (UM) 2257*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44980kB
Dec 10 05:26:40 admin kernel: [397983.346768] Node 0 Normal: 10918*4kB (UMEH) 49*8kB (UMH) 0*16kB 1*32kB (H) 1*64kB (H) 3*128kB (H) 0*256kB 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 45056kB
Dec 10 05:26:40 admin kernel: [397983.346777] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 10 05:26:40 admin kernel: [397983.346779] 68538 total pagecache pages
Dec 10 05:26:40 admin kernel: [397983.346781] 18641 pages in swap cache
Dec 10 05:26:40 admin kernel: [397983.346783] Swap cache stats: add 383468753, delete 383450112, find 229519152/378197444
Dec 10 05:26:40 admin kernel: [397983.346784] Free swap = 1252280kB
Dec 10 05:26:42 admin kernel: [397983.346785] Total swap = 11717628kB
Dec 10 05:26:42 admin kernel: [397983.346787] 2097022 pages RAM
Dec 10 05:26:42 admin kernel: [397983.346788] 0 pages HighMem/MovableOnly
Dec 10 05:26:42 admin kernel: [397983.346789] 53314 pages reserved
Dec 10 05:26:42 admin kernel: [397983.346790] 0 pages cma reserved
Dec 10 05:26:42 admin kernel: [397983.346791] 0 pages hwpoisoned
Dec 10 05:26:42 admin kernel: [397983.346793] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Dec 10 05:26:42 admin kernel: [397983.346800] [ 409] 0 409 10976 822 24 3 602 0 systemd-journal
Dec 10 05:26:42 admin kernel: [397983.346803] [ 451] 0 451 23693 83 17 3 50 0 lvmetad
Dec 10 05:26:42 admin kernel: [397983.346805] [ 470] 0 470 11415 242 22 3 485 -1000 systemd-udevd
Dec 10 05:26:42 admin kernel: [397983.346808] [ 784] 0 784 47527 409 52 3 223 0 vmtoolsd
Dec 10 05:26:42 admin kernel: [397983.346811] [ 838] 0 838 5884 0 16 3 51 0 rpc.idmapd
Dec 10 05:26:42 admin kernel: [397983.346813] [ 842] 100 842 25081 132 20 3 52 0 systemd-timesyn
Dec 10 05:26:42 admin kernel: [397983.346816] [ 934] 0 934 11906 140 27 3 109 0 rpcbind
Dec 10 05:26:42 admin kernel: [397983.346818] [ 948] 0 948 151028 166 28 4 213 0 lxcfs
Dec 10 05:26:42 admin kernel: [397983.346821] [ 953] 0 953 7252 456 19 3 48 0 cron
Dec 10 05:26:42 admin kernel: [397983.346823] [ 959] 0 959 1099 335 8 3 40 0 acpid
Dec 10 05:26:42 admin kernel: [397983.346826] [ 961] 0 961 7165 239 19 3 57 0 systemd-logind
Dec 10 05:26:42 admin kernel: [397983.346828] [ 967] 108 967 10725 566 25 3 76 -900 dbus-daemon
Dec 10 05:26:42 admin kernel: [397983.346831] [ 1047] 0 1047 21359 342 32 3 347 0 VGAuthService
Dec 10 05:26:42 admin kernel: [397983.346833] [ 1049] 0 1049 68974 456 37 3 224 0 accounts-daemon
Dec 10 05:26:42 admin kernel: [397983.346835] [ 1051] 0 1051 6511 391 18 3 47 0 atd
Dec 10 05:26:42 admin kernel: [397983.346838] [ 1056] 104 1056 64099 457 29 3 367 0 rsyslogd
Dec 10 05:26:42 admin kernel: [397983.346841] [ 1148] 0 1148 3343 51 11 3 23 0 mdadm
Dec 10 05:26:42 admin kernel: [397983.346843] [ 1179] 0 1179 6011 251 16 3 89 0 vsftpd
Dec 10 05:26:42 admin kernel: [397983.346845] [ 1189] 0 1189 69278 506 40 4 122 0 polkitd
Dec 10 05:26:42 admin kernel: [397983.346848] [ 1194] 0 1194 1305 358 9 3 61 0 iscsid
Dec 10 05:26:42 admin kernel: [397983.346850] [ 1195] 0 1195 1430 879 10 3 0 -17 iscsid
Dec 10 05:26:42 admin kernel: [397983.346853] [ 1207] 0 1207 9494 0 22 3 190 0 rpc.mountd
Dec 10 05:26:42 admin kernel: [397983.346856] [ 1221] 0 1221 16378 339 36 3 196 -1000 sshd
Dec 10 05:26:42 admin kernel: [397983.346859] [ 1296] 0 1296 16458 453 37 4 128 0 login
Dec 10 05:26:42 admin kernel: [397983.346861] [ 1326] 0 1326 4905 284 14 3 39 0 irqbalance
Dec 10 05:26:42 admin kernel: [397983.346864] [ 1420] 0 1420 13514 229 30 3 2438 0 munin-node
Dec 10 05:26:42 admin kernel: [397983.346866] [ 1501] 0 1501 26199 440 51 4 470 0 sendmail-mta
Dec 10 05:26:42 admin kernel: [397983.346869] [ 1681] 0 1681 109282 257 171 3 1936 0 php-fpm7.0
Dec 10 05:26:42 admin kernel: [397983.346871] [ 1687] 33 1687 109282 184 152 3 1961 0 php-fpm7.0
Dec 10 05:26:42 admin kernel: [397983.346873] [ 1688] 33 1688 109282 184 152 3 1961 0 php-fpm7.0
Dec 10 05:26:42 admin kernel: [397983.346876] [16727] 1000 16727 11330 422 25 3 217 0 systemd
Dec 10 05:26:42 admin kernel: [397983.346878] [16731] 1000 16731 52186 0 37 3 501 0 (sd-pam)
Dec 10 05:26:42 admin kernel: [397983.346881] [16734] 1000 16734 5900 357 18 4 725 0 bash
Dec 10 05:26:42 admin kernel: [397983.346884] [11281] 119 11281 69349 594 88 3 1257 0 freshclam
Dec 10 05:26:42 admin kernel: [397983.346886] [32663] 0 32663 23229 459 48 3 258 0 sshd
Dec 10 05:26:42 admin kernel: [397983.346889] [32740] 1000 32740 23668 373 48 3 743 0 sshd
Dec 10 05:26:42 admin kernel: [397983.346892] [22943] 0 22943 23229 330 51 3 272 0 sshd
Dec 10 05:26:42 admin kernel: [397983.346894] [23020] 1000 23020 23229 332 49 3 247 0 sshd
Dec 10 05:26:42 admin kernel: [397983.346897] [23023] 1000 23023 3220 393 12 3 51 0 sftp-server
Dec 10 05:26:42 admin kernel: [397983.346899] [24027] 1000 24027 3220 381 12 3 67 0 sftp-server
Dec 10 05:26:42 admin kernel: [397983.346902] [27215] 1000 27215 3220 378 12 3 95 0 sftp-server
Dec 10 05:26:42 admin kernel: [397983.346904] [13006] 0 13006 168834 2419 280 3 29858 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346907] [15772] 33 15772 385874 34986 583 4 67131 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346909] [15773] 33 15773 381008 40856 579 4 53799 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346912] [15776] 33 15776 297940 35288 395 4 53305 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346915] [15777] 33 15777 303389 37051 415 4 49856 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346919] [15779] 33 15779 303184 39806 403 4 46606 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346922] [15782] 33 15782 304265 46518 404 4 43859 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346925] [15786] 33 15786 371258 41084 557 4 49806 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346929] [15787] 33 15787 302813 34962 403 4 53776 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346931] [15789] 33 15789 380748 35019 567 4 52932 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346934] [15799] 33 15799 384260 39566 561 4 55415 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346936] [15800] 33 15800 374314 29895 544 4 52106 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346939] [15801] 33 15801 380377 34701 565 4 53467 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346941] [15802] 33 15802 306345 35364 424 4 57003 0 apache2
........
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 570
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 570
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 556
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 556
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 539
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 539
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 525
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 525
......
Dec 10 06:27:15 admin kernel: [401618.309140] Out of memory: Kill process 15801 (apache2) score 14 or sacrifice child
Dec 10 06:27:15 admin kernel: [401618.309960] Killed process 15801 (apache2) total-vm:1521508kB, anon-rss:23236kB, file-rss:0kB
Dec 10 06:27:21 admin CRON[12717]: (root) CMD (cp /var/spool/cron/crontabs/root /var/www/crontab/root && chmod 777 /var/www/crontab/root && chown zak:zak /var/www/crontab/root)
Dec 10 06:27:21 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 459
Dec 10 06:27:21 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 459
Dec 10 06:27:29 admin kernel: [401632.351612] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=209.85.238.216 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=42747 PROTO=TCP SPT=49490 DPT=443 WINDOW=243 RES=0x00 ACK RST URGP=0
Dec 10 06:27:30 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 475
Dec 10 06:27:30 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 475
Dec 10 06:27:43 admin apache2[12635]: *
Dec 10 06:27:43 admin systemd[1]: Stopped LSB: Apache2 web server.
Dec 10 06:27:43 admin systemd[1]: Starting LSB: Apache2 web server...
Dec 10 06:27:43 admin apache2[12749]: * Starting Apache httpd web server apache2
Dec 10 06:27:45 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 370
Dec 10 06:27:45 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 370
Dec 10 06:27:49 admin systemd[1]: Starting Daily apt upgrade and clean activities...
Dec 10 06:27:49 admin kernel: [401652.463319] [UFW BLOCK] IN=ens32 OUT=
MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=123.183.224.66 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=41274 PROTO=TCP SPT=45696 DPT=443 WINDOW=243 RES=0x00 ACK RST URGP=0
Dec 10 06:27:54 admin apache2[12749]: [Fri Dec 10 06:27:54.747509 2021] [proxy_html:notice] [pid 12805] AH01425: I18n support in mod_proxy_html requires mod_xml2enc. Without it, non-ASCII characters in proxied pages are likely to display incorrectly.
c 10 06:28:09 admin kernel: [401672.878972] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=76.99.197.116 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=52726 PROTO=TCP SPT=55476 DPT=443 WINDOW=248 RES=0x00 ACK RST URGP=0
Dec 10 06:28:15 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 225
Dec 10 06:28:15 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 225
Dec 10 06:28:30 admin kernel: [401693.571187] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=185.191.171.2 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=57 ID=0 DF PROTO=TCP SPT=42580 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
Dec 10 06:28:30 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 176
Dec 10 06:28:30 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 176
Dec 10 06:28:45 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 139
Dec 10 06:28:45 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 139
Dec 10 06:28:49 admin kernel: [401712.942365] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=95.217.225.110 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=44271 PROTO=TCP SPT=36124 DPT=443 WINDOW=254 RES=0x00 ACK RST URGP=0
Dec 10 06:29:00 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 109
Dec 10 06:29:00 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 109
Dec 10 06:29:01 admin CRON[13228]: (root) CMD (cp /var/spool/cron/crontabs/root /var/www/crontab/root && chmod 777 /var/www/crontab/root && chown zak:zak /var/www/crontab/root)
Dec 10 06:29:15 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 87
Dec 10 06:29:15 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 87
Dec 10 06:29:30 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 69
Dec 10 06:29:30 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 69
Dec 10 06:29:45 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 55
Dec 10 06:29:45 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 55
Dec 10 06:29:53 admin kernel: [401776.578209] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=116.179.37.124 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=47773 PROTO=TCP SPT=52871 DPT=443 WINDOW=246 RES=0x00 ACK RST URGP=0
Dec 10 06:30:00 admin sm-mta[1501]: accepting connections again for daemon MTA-v4
Dec 10 06:30:00 admin sm-mta[1501]: accepting connections again for daemon MSP-v4
Apache 错误日志
[Fri Dec 10 05:23:33.884301 2021] [pagespeed:warn] [pid 7829] [mod_pagespeed 1.13.35.2-0 @7829] Fetch timed out: https://|-REMOVED-|/css/fontawesome.css (connecting to:10.2.6.80) (4) waiting for 50 ms
[Fri Dec 10 05:23:33.927235 2021] [pagespeed:error] [pid 7881] [mod_pagespeed 1.13.35.2-0 @7881] Slow write operation on file /var/cache/mod_pagespeed/v3/|-REMOVED-|/https,3A/,2F|-REMOVED-|/css/animate.css+stylesheet-1625614552.css+Contact-Us-1625614552.css.pagespeed.cc.0HlzBptJqh.css,.tempC6gpkp: 2974.19ms; configure SlowFileLatencyUs to change threshold\n
[Fri Dec 10 05:23:34.012843 2021] [pagespeed:warn] [pid 15909] [mod_pagespeed 1.13.35.2-0 @15909] Fetch timed out: https://|-REMOVED-|/imageserver/confirm/buttons.png (connecting to:10.2.6.80) (1) waiting for 50 ms
[Fri Dec 10 05:23:34.036279 2021] [pagespeed:warn] [pid 7368] [mod_pagespeed 1.13.35.2-0 @7368] Fetch timed out: https://|-REMOVED-|/css/images/layers-2x.png (connecting to:10.2.6.80) (1) waiting for 50 ms
[Fri Dec 10 05:23:34.190221 2021] [pagespeed:warn] [pid 7615] [mod_pagespeed 1.13.35.2-0 @7615] Fetch timed out: https://|-REMOVED-|/imageserver/confirm/ie.png (connecting to:10.2.6.80) (1) waiting for 50 ms
[Fri Dec 10 05:23:34.291732 2021] [pagespeed:error] [pid 7314] [mod_pagespeed 1.13.35.2-0 @7314] Slow ReadFile operation on file /var/cache/mod_pagespeed/v3/|-REMOVED-|/https,3A/,2F|-REMOVED-|/css/presidential-solaris-1.css,: 333.858ms; configure SlowFileLatencyUs to change threshold\n
.........
[Fri Dec 10 06:26:59.176654 2021] [core:warn] [pid 13006] AH00045: child process 7180 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619326 2021] [core:warn] [pid 13006] AH00045: child process 7181 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619410 2021] [core:warn] [pid 13006] AH00045: child process 7182 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619449 2021] [core:warn] [pid 13006] AH00045: child process 7183 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619492 2021] [core:warn] [pid 13006] AH00045: child process 7184 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619529 2021] [core:warn] [pid 13006] AH00045: child process 7185 still did not exit, sending a SIGTERM
.......
[Fri Dec 10 06:27:23.205977 2021] [core:error] [pid 13006] AH00047: could not make child process 7184 exit, attempting to continue anyway
[Fri Dec 10 06:27:23.206200 2021] [core:error] [pid 13006] AH00047: could not make child process 7326 exit, attempting to continue anyway
[Fri Dec 10 06:27:23.206283 2021] [core:error] [pid 13006] AH00047: could not make child process 7338 exit, attempting to continue anyway
[Fri Dec 10 06:27:23.206418 2021] [core:error] [pid 13006] AH00047: could not make child process 7353 exit, attempting to continue anyway
[Fri Dec 10 06:27:23.206562 2021] [core:error] [pid 13006] AH00047: could not make child process 7362 exit, attempting to continue anyway
.........
Fri Dec 10 06:27:23.323233 2021] [mpm_prefork:notice] [pid 13006] AH00169: caught SIGTERM, shutting down
停机时间如下所示：
我们的停机时间检测器捕获到的确切时间是：
向下：2021-12-10 05:26:33 UTC-6
向上：2021-12-10 05:45:02 UTC-6
答案1
好吧，对于任何处于我这种情况的人来说，在未来都会遇到这种情况。这是一个解决方案，尽管是一个黑客解决方案。
我注意到，如果我关闭 PageSpeed，问题就不大了。我查看了我的平均 Apache 请求，发现它是 62MB。我将其除以用于运行所有其他资源的剩余 RAM。我还剩下 6GB。我将 Max Server 和 Max Client 从 256 更改为 128——这有帮助——但并没有“解决”它。
然后我注意到 pagespeed 并没有像我预期的那样快速地将物理文件存储在 /var/cache/mod_pagespeed 中。它而是使用 RAM 空间，即使使用文件内部的行conf：
ModPagespeedFileCachePath "/var/cache/mod_pagespeed/"
然后我有了一个想法......当 RAM 无法访问时，pagespeed 在正常 apache 重启期间会做什么......它存储在本地/var/cache/目录中的是什么。
因为我的 Web 配置是静态的，配置文件永远不会改变。我将其写入service apache reload每 30 分钟运行一次的 cron。现在 Pagespeed 会填满磁盘var/cache，而不会填满 RAM，直到崩溃。
笔记 如果您像我们一样运行 1000 多个网站，请确保您有足够的空间来缓存每个网站的物理文件，最终将占用数 GB 的空间，但这就是 PageSpeed 的工作方式。否则，您的磁盘很快就会被填满。我的磁盘使用量在几小时内从 102GB 增加到 165GB，在 1TB 的驱动器上，这微不足道，但如果我有 200GB 的驱动器，这可能会令人担忧。