我的服务器每天都会宕机，有时一天会宕机两三次。每次宕机时间在 2 到 15 分钟之间
我很确定这是由于某种分布式 DDOS 或暴力破解造成的，但使用 Cloudflare 似乎还不够
如果有人能查看这些并确认是 apache（传入流量）导致了崩溃，那将会有很大的帮助
Oct 1 07:34:34 flylive kernel: httpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Oct 1 07:34:34 flylive kernel: httpd cpuset=/ mems_allowed=0
Oct 1 07:34:34 flylive kernel: CPU: 1 PID: 6948 Comm: httpd Kdump: loaded Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Oct 1 07:34:34 flylive kernel: Hardware name: Xen HVM domU, BIOS 4.1.5 11/28/2013
Oct 1 07:34:34 flylive kernel: Call Trace:
Oct 1 07:34:34 flylive kernel: [<ffffffffb9f7ffa5>] dump_stack+0x19/0x1b
Oct 1 07:34:34 flylive kernel: [<ffffffffb9f7a8c3>] dump_header+0x90/0x229
Oct 1 07:34:34 flylive kernel: [<ffffffffb9b0dadb>] ? cred_has_capability+0x6b/0x120
Oct 1 07:34:34 flylive kernel: [<ffffffffb99c251e>] oom_kill_process+0x25e/0x3f0
Oct 1 07:34:34 flylive kernel: [<ffffffffb9b0dbbe>] ? selinux_capable+0x2e/0x40
Oct 1 07:34:34 flylive kernel: [<ffffffffb99c2d76>] out_of_memory+0x4b6/0x4f0
Oct 1 07:34:34 flylive kernel: [<ffffffffb9f7b3e0>] __alloc_pages_slowpath+0x5db/0x729
Oct 1 07:34:34 flylive kernel: [<ffffffffb99c91f6>] __alloc_pages_nodemask+0x436/0x450
Oct 1 07:34:34 flylive kernel: [<ffffffffb9a18ea8>] alloc_pages_current+0x98/0x110
Oct 1 07:34:34 flylive kernel: [<ffffffffb99be427>] __page_cache_alloc+0x97/0xb0
Oct 1 07:34:34 flylive kernel: [<ffffffffb99c0fe0>] filemap_fault+0x270/0x420
Oct 1 07:34:34 flylive kernel: [<ffffffffc0339706>] ext4_filemap_fault+0x36/0x50 [ext4]
Oct 1 07:34:34 flylive kernel: [<ffffffffb99edf6a>] __do_fault.isra.61+0x8a/0x100
Oct 1 07:34:34 flylive kernel: [<ffffffffb99ee51c>] do_read_fault.isra.63+0x4c/0x1b0
Oct 1 07:34:34 flylive kernel: [<ffffffffb99f5d80>] handle_mm_fault+0xa20/0xfb0
Oct 1 07:34:34 flylive kernel: [<ffffffffb9e32731>] ? sock_aio_read+0x21/0x30
Oct 1 07:34:34 flylive kernel: [<ffffffffb9a4c633>] ? do_sync_read+0x93/0xe0
Oct 1 07:34:34 flylive kernel: [<ffffffffb9f8d653>] __do_page_fault+0x213/0x500
Oct 1 07:34:34 flylive kernel: [<ffffffffb9f8d975>] do_page_fault+0x35/0x90
Oct 1 07:34:34 flylive kernel: [<ffffffffb9f89ac9>] ? error_swapgs+0xaa/0xc0
Oct 1 07:34:34 flylive kernel: [<ffffffffb9f89778>] page_fault+0x28/0x30
Oct 1 07:34:34 flylive kernel: Mem-Info:
Oct 1 07:34:34 flylive kernel: active_anon:878702 inactive_anon:27289 isolated_anon:0#012 active_file:0 inactive_file:749 isolated_file:78#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:6602 slab_unreclaimable:10598#012 mapped:30050 shmem:78419 pagetables:6291 bounce:0#012 free:20812 free_pcp:0 free_cma:0
Oct 1 07:34:34 flylive kernel: Node 0 DMA free:15324kB min:276kB low:344kB high:412kB active_anon:296kB inactive_anon:92kB active_file:0kB inactive_file:24kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:132kB shmem:132kB slab_reclaimable:28kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:39 all_unreclaimable? yes
Oct 1 07:34:34 flylive kernel: lowmem_reserve[]: 0 3580 3765 3765
Oct 1 07:34:34 flylive kernel: Node 0 DMA32 free:64680kB min:64000kB low:80000kB high:96000kB active_anon:3381472kB inactive_anon:103448kB active_file:0kB inactive_file:3104kB unevictable:0kB isolated(anon):0kB isolated(file):312kB present:3915776kB managed:3669344kB mlocked:0kB dirty:0kB writeback:0kB mapped:114384kB shmem:302416kB slab_reclaimable:20252kB slab_unreclaimable:33768kB kernel_stack:7120kB pagetables:23420kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4822 all_unreclaimable? yes
Oct 1 07:34:34 flylive kernel: lowmem_reserve[]: 0 0 184 184
Oct 1 07:34:34 flylive kernel: Node 0 Normal free:3244kB min:3304kB low:4128kB high:4956kB active_anon:133040kB inactive_anon:5616kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:258048kB managed:189324kB mlocked:0kB dirty:0kB writeback:0kB mapped:5684kB shmem:11128kB slab_reclaimable:6128kB slab_unreclaimable:8592kB kernel_stack:2304kB pagetables:1720kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 1 07:34:34 flylive kernel: lowmem_reserve[]: 0 0 0 0
Oct 1 07:34:34 flylive kernel: Node 0 DMA: 3*4kB (EM) 1*8kB (M) 3*16kB (U) 2*32kB (UE) 3*64kB (UE) 1*128kB (E) 2*256kB (UE) 2*512kB (EM) 3*1024kB (UEM) 1*2048kB (E) 2*4096kB (M) = 15300kB
Oct 1 07:34:34 flylive kernel: Node 0 DMA32: 757*4kB (UEM) 502*8kB (UEM) 400*16kB (UEM) 417*32kB (UEM) 257*64kB (UEM) 91*128kB (UEM) 17*256kB (UEM) 5*512kB (EM) 3*1024kB (M) 0*2048kB 0*4096kB = 64868kB
Oct 1 07:34:34 flylive kernel: Node 0 Normal: 223*4kB (UEM) 69*8kB (EM) 33*16kB (UEM) 24*32kB (UM) 13*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3572kB
Oct 1 07:34:34 flylive kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 1 07:34:34 flylive kernel: 79172 total pagecache pages
Oct 1 07:34:34 flylive kernel: 0 pages in swap cache
Oct 1 07:34:34 flylive kernel: Swap cache stats: add 0, delete 0, find 0/0
Oct 1 07:34:34 flylive kernel: Free swap = 0kB
Oct 1 07:34:34 flylive kernel: Total swap = 0kB
Oct 1 07:34:34 flylive kernel: 1047453 pages RAM
Oct 1 07:34:34 flylive kernel: 0 pages HighMem/MovableOnly
Oct 1 07:34:34 flylive kernel: 78810 pages reserved
Oct 1 07:34:34 flylive kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Oct 1 07:34:34 flylive kernel: [ 384] 0 384 13969 4769 34 0 0 systemd-journal
Oct 1 07:34:34 flylive kernel: [ 386] 0 386 74209 2594 66 0 0 nova-agent
Oct 1 07:34:34 flylive kernel: [ 407] 0 407 29795 369 29 0 0 lvmetad
Oct 1 07:34:34 flylive kernel: [ 420] 0 420 12083 592 25 0 -1000 systemd-udevd
Oct 1 07:34:34 flylive kernel: [ 534] 0 534 13883 109 26 0 -1000 auditd
Oct 1 07:34:34 flylive kernel: [ 556] 0 556 13203 188 31 0 0 smartd
Oct 1 07:34:34 flylive kernel: [ 557] 996 557 2145 37 9 0 0 lsmd
Oct 1 07:34:34 flylive kernel: [ 558] 0 558 22642 214 47 0 0 rngd
Oct 1 07:34:34 flylive kernel: [ 560] 0 560 5386 73 17 0 0 irqbalance
Oct 1 07:34:34 flylive kernel: [ 561] 997 561 153256 1595 63 0 0 polkitd
Oct 1 07:34:34 flylive kernel: [ 565] 0 565 57041 438 63 0 0 abrtd
Oct 1 07:34:34 flylive kernel: [ 566] 0 566 56465 370 61 0 0 abrt-watch-log
Oct 1 07:34:34 flylive kernel: [ 570] 0 570 6796 289 19 0 0 systemd-logind
Oct 1 07:34:34 flylive kernel: [ 571] 81 571 16620 197 32 0 -900 dbus-daemon
Oct 1 07:34:34 flylive kernel: [ 582] 995 582 30102 122 28 0 0 chronyd
Oct 1 07:34:34 flylive kernel: [ 606] 0 606 90793 6722 95 0 0 firewalld
Oct 1 07:34:34 flylive kernel: [ 1061] 0 1061 108533 2196 45 0 0 rackspace-monit
Oct 1 07:34:34 flylive kernel: [ 1065] 0 1065 145994 3269 101 0 0 tuned
Oct 1 07:34:34 flylive kernel: [ 1296] 0 1296 23245 340 43 0 0 master
Oct 1 07:34:34 flylive kernel: [ 1303] 89 1303 23315 349 45 0 0 qmgr
Oct 1 07:34:34 flylive kernel: [ 1331] 0 1331 120817 3635 109 0 0 rsyslogd
Oct 1 07:34:34 flylive kernel: [ 1332] 0 1332 28231 256 59 0 -1000 sshd
Oct 1 07:34:34 flylive kernel: [ 1357] 0 1357 6477 52 19 0 0 atd
Oct 1 07:34:34 flylive kernel: [ 1358] 0 1358 31598 160 19 0 0 crond
Oct 1 07:34:34 flylive kernel: [ 1624] 0 1624 3022 169 11 0 0 xe-daemon
Oct 1 07:34:34 flylive kernel: [24548] 27 24548 28355 73 12 0 0 mysqld_safe
Oct 1 07:34:34 flylive kernel: [24727] 27 24727 663068 125770 322 0 0 mysqld
Oct 1 07:34:34 flylive kernel: [27028] 0 27028 13322 143 27 0 0 vsftpd
Oct 1 07:34:34 flylive kernel: [20795] 0 20795 27552 34 10 0 0 agetty
Oct 1 07:34:34 flylive kernel: [15951] 0 15951 172780 12691 260 0 0 httpd
Oct 1 07:34:34 flylive kernel: [15952] 48 15952 79019 1410 149 0 0 httpd
Oct 1 07:34:34 flylive kernel: [15957] 48 15957 888056 174687 748 0 0 httpd
Oct 1 07:34:34 flylive kernel: [16176] 48 16176 855193 177632 751 0 0 httpd
Oct 1 07:34:34 flylive kernel: [25271] 48 25271 855272 180205 835 0 0 httpd
Oct 1 07:34:34 flylive kernel: [28247] 993 28247 665696 12110 202 0 0 agent
Oct 1 07:34:34 flylive kernel: [28248] 993 28248 228276 3979 63 0 0 trace-agent
Oct 1 07:34:34 flylive kernel: [28249] 993 28249 252846 5151 81 0 0 process-agent
Oct 1 07:34:34 flylive kernel: [18206] 0 18206 31372 61 18 0 0 anacron
Oct 1 07:34:34 flylive kernel: [28219] 0 28219 28321 54 12 0 0 run-parts
Oct 1 07:34:34 flylive kernel: [28223] 0 28223 88040 3622 122 0 0 yum-cron
Oct 1 07:34:34 flylive kernel: [28224] 0 28224 28411 36 11 0 0 awk
Oct 1 07:34:34 flylive kernel: [ 6903] 48 6903 827221 120930 655 0 0 httpd
Oct 1 07:34:34 flylive kernel: [21093] 48 21093 829126 97392 555 0 0 httpd
Oct 1 07:34:34 flylive kernel: [ 3204] 0 3204 45104 229 45 0 0 crond
Oct 1 07:34:34 flylive kernel: [ 3206] 0 3206 28321 53 12 0 0 run-parts
Oct 1 07:34:34 flylive kernel: [ 3221] 0 3221 320000 12911 166 0 0 yum-cron
Oct 1 07:34:34 flylive kernel: [ 3222] 0 3222 28411 36 12 0 0 awk
Oct 1 07:34:34 flylive kernel: [ 4189] 89 4189 23271 333 46 0 0 pickup
Oct 1 07:34:34 flylive kernel: [ 6209] 0 6209 2922 54 11 0 0 xe-update-guest
Oct 1 07:34:34 flylive kernel: [ 6211] 0 6211 35820 178 26 0 0 crond
Oct 1 07:34:34 flylive kernel: [ 6217] 0 6217 3298 27 11 0 0 mkdir
Oct 1 07:34:34 flylive kernel: Out of memory: Kill process 25271 (httpd) score 186 or sacrifice child
Oct 1 07:34:34 flylive kernel: Killed process 25271 (httpd), UID 48, total-vm:3421088kB, anon-rss:628616kB, file-rss:0kB, shmem-rss:92204kB
Oct 1 07:34:34 flylive kernel: dbus-daemon invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=-900
Oct 1 07:34:34 flylive kernel: dbus-daemon cpuset=/ mems_allowed=0
Oct 1 07:34:34 flylive kernel: CPU: 3 PID: 571 Comm: dbus-daemon Kdump: loaded Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Oct 1 07:34:34 flylive kernel: Hardware name: Xen HVM domU, BIOS 4.1.5 11/28/2013
Oct 1 07:34:34 flylive kernel: Call Trace:
Oct 1 07:34:34 flylive kernel: [<ffffffffb9f7ffa5>] dump_stack+0x19/0x1b
Oct 1 07:34:34 flylive kernel: [<ffffffffb9f7a8c3>] dump_header+0x90/0x229
Oct 1 07:34:34 flylive kernel: [<ffffffffb9b0dadb>] ? cred_has_capability+0x6b/0x120
Oct 1 07:34:34 flylive kernel: [<ffffffffb99c251e>] oom_kill_process+0x25e/0x3f0
Oct 1 07:34:34 flylive kernel: [<ffffffffb9b0dbbe>] ? selinux_capable+0x2e/0x40
Oct 1 07:34:34 flylive kernel: [<ffffffffb99c2d76>] out_of_memory+0x4b6/0x4f0
Oct 1 07:34:34 flylive kernel: [<ffffffffb9f7b3e0>] __alloc_pages_slowpath+0x5db/0x729
Oct 1 07:34:34 flylive kernel: [<ffffffffb99c91f6>] __alloc_pages_nodemask+0x436/0x450
Oct 1 07:34:34 flylive kernel: [<ffffffffb9a18ea8>] alloc_pages_current+0x98/0x110
Oct 1 07:34:34 flylive kernel: [<ffffffffb99be427>] __page_cache_alloc+0x97/0xb0
Oct 1 07:34:34 flylive kernel: [<ffffffffb99c0fe0>] filemap_fault+0x270/0x420
Oct 1 07:34:34 flylive kernel: [<ffffffffc0339706>] ext4_filemap_fault+0x36/0x50 [ext4]
Oct 1 07:34:34 flylive kernel: [<ffffffffb99edf6a>] __do_fault.isra.61+0x8a/0x100
Oct 1 07:34:34 flylive kernel: [<ffffffffb99ee51c>] do_read_fault.isra.63+0x4c/0x1b0
Oct 1 07:34:34 flylive kernel: [<ffffffffb99f5d80>] handle_mm_fault+0xa20/0xfb0
Oct 1 07:34:34 flylive kernel: [<ffffffffb9a9c029>] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0
Oct 1 07:34:34 flylive kernel: [<ffffffffb9f8d653>] __do_page_fault+0x213/0x500
Oct 1 07:34:34 flylive kernel: [<ffffffffb9f8d975>] do_page_fault+0x35/0x90
Oct 1 07:34:34 flylive kernel: [<ffffffffb9f89ac9>] ? error_swapgs+0xaa/0xc0
Oct 1 07:34:34 flylive kernel: [<ffffffffb9f89778>] page_fault+0x28/0x30
Oct 1 07:34:34 flylive kernel: Mem-Info:
Oct 1 07:34:34 flylive kernel: active_anon:878702 inactive_anon:27289 isolated_anon:0#012 active_file:24 inactive_file:0 isolated_file:78#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:6602 slab_unreclaimable:10598#012 mapped:29995 shmem:78419 pagetables:6291 bounce:0#012 free:21434 free_pcp:202 free_cma:0
Oct 1 07:34:34 flylive kernel: Node 0 DMA free:15324kB min:276kB low:344kB high:412kB active_anon:296kB inactive_anon:92kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:132kB shmem:132kB slab_reclaimable:28kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 1 07:34:34 flylive kernel: lowmem_reserve[]: 0 3580 3765 3765
Oct 1 07:34:34 flylive kernel: Node 0 DMA32 free:67168kB min:64000kB low:80000kB high:96000kB active_anon:3381472kB inactive_anon:103448kB active_file:96kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):312kB present:3915776kB managed:3669344kB mlocked:0kB dirty:0kB writeback:0kB mapped:114164kB shmem:302416kB slab_reclaimable:20252kB slab_unreclaimable:33768kB kernel_stack:7120kB pagetables:23420kB unstable:0kB bounce:0kB free_pcp:808kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:156 all_unreclaimable? yes
Oct 1 07:34:34 flylive kernel: lowmem_reserve[]: 0 0 184 184
Oct 1 07:34:34 flylive kernel: Node 0 Normal free:3244kB min:3304kB low:4128kB high:4956kB active_anon:133040kB inactive_anon:5616kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:258048kB managed:189324kB mlocked:0kB dirty:0kB writeback:0kB mapped:5684kB shmem:11128kB slab_reclaimable:6128kB slab_unreclaimable:8592kB kernel_stack:2304kB pagetables:1720kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 1 07:34:34 flylive kernel: lowmem_reserve[]: 0 0 0 0
Oct 1 07:34:34 flylive kernel: Node 0 DMA: 3*4kB (EM) 1*8kB (M) 4*16kB (UM) 2*32kB (UE) 3*64kB (UE) 1*128kB (E) 2*256kB (UE) 2*512kB (EM) 3*1024kB (UEM) 1*2048kB (E) 2*4096kB (M) = 15316kB
Oct 1 07:34:34 flylive kernel: Node 0 DMA32: 834*4kB (UEM) 536*8kB (UEM) 446*16kB (UEM) 451*32kB (UEM) 263*64kB (UEM) 91*128kB (UEM) 16*256kB (UEM) 5*512kB (EM) 3*1024kB (M) 0*2048kB 0*4096kB = 67400kB
Oct 1 07:34:34 flylive kernel: Node 0 Normal: 223*4kB (UEM) 69*8kB (EM) 33*16kB (UEM) 27*32kB (UM) 13*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3668kB
Oct 1 07:34:34 flylive kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 1 07:34:34 flylive kernel: 78450 total pagecache pages
Oct 1 07:34:34 flylive kernel: 0 pages in swap cache
Oct 1 07:34:34 flylive kernel: Swap cache stats: add 0, delete 0, find 0/0
Oct 1 07:34:34 flylive kernel: Free swap = 0kB
Oct 1 07:34:34 flylive kernel: Total swap = 0kB
Oct 1 07:34:34 flylive kernel: 1047453 pages RAM
Oct 1 07:34:34 flylive kernel: 0 pages HighMem/MovableOnly
Oct 1 07:34:34 flylive kernel: 78810 pages reserved
Oct 1 07:34:34 flylive kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Oct 1 07:34:34 flylive kernel: [ 384] 0 384 13969 4769 34 0 0 systemd-journal
Oct 1 07:34:34 flylive kernel: [ 386] 0 386 74209 2594 66 0 0 nova-agent
Oct 1 07:34:34 flylive kernel: [ 407] 0 407 29795 369 29 0 0 lvmetad
Oct 1 07:34:34 flylive kernel: [ 420] 0 420 12083 5
.....
这是同一天发生的另一个例子
Oct 1 12:18:25 flylive kernel: httpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Oct 1 12:18:25 flylive kernel: httpd cpuset=/ mems_allowed=0
Oct 1 12:18:25 flylive kernel: CPU: 2 PID: 16132 Comm: httpd Kdump: loaded Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Oct 1 12:18:25 flylive kernel: Hardware name: Xen HVM domU, BIOS 4.1.5 11/28/2013
Oct 1 12:18:25 flylive kernel: Call Trace:
Oct 1 12:18:25 flylive kernel: [<ffffffffb9f7ffa5>] dump_stack+0x19/0x1b
Oct 1 12:18:25 flylive kernel: [<ffffffffb9f7a8c3>] dump_header+0x90/0x229
Oct 1 12:18:25 flylive kernel: [<ffffffffb9b0dadb>] ? cred_has_capability+0x6b/0x120
Oct 1 12:18:25 flylive kernel: [<ffffffffb99c251e>] oom_kill_process+0x25e/0x3f0
Oct 1 12:18:25 flylive kernel: [<ffffffffb9b0dbbe>] ? selinux_capable+0x2e/0x40
Oct 1 12:18:25 flylive kernel: [<ffffffffb99c2d76>] out_of_memory+0x4b6/0x4f0
Oct 1 12:18:25 flylive kernel: [<ffffffffb9f7b3e0>] __alloc_pages_slowpath+0x5db/0x729
Oct 1 12:18:25 flylive kernel: [<ffffffffb99c91f6>] __alloc_pages_nodemask+0x436/0x450
Oct 1 12:18:25 flylive kernel: [<ffffffffb9a18ea8>] alloc_pages_current+0x98/0x110
Oct 1 12:18:25 flylive kernel: [<ffffffffb99be427>] __page_cache_alloc+0x97/0xb0
Oct 1 12:18:25 flylive kernel: [<ffffffffb99c0fe0>] filemap_fault+0x270/0x420
Oct 1 12:18:25 flylive kernel: [<ffffffffc0339706>] ext4_filemap_fault+0x36/0x50 [ext4]
Oct 1 12:18:25 flylive kernel: [<ffffffffb99edf6a>] __do_fault.isra.61+0x8a/0x100
Oct 1 12:18:25 flylive kernel: [<ffffffffb99ee51c>] do_read_fault.isra.63+0x4c/0x1b0
Oct 1 12:18:25 flylive kernel: [<ffffffffb99f5d80>] handle_mm_fault+0xa20/0xfb0
Oct 1 12:18:25 flylive kernel: [<ffffffffb9a4e49e>] ? do_readv_writev+0x19e/0x260
Oct 1 12:18:25 flylive kernel: [<ffffffffb9f8d653>] __do_page_fault+0x213/0x500
Oct 1 12:18:25 flylive kernel: [<ffffffffb9f8d975>] do_page_fault+0x35/0x90
Oct 1 12:18:25 flylive kernel: [<ffffffffb9f89ac9>] ? error_swapgs+0xaa/0xc0
Oct 1 12:18:25 flylive kernel: [<ffffffffb9f89778>] page_fault+0x28/0x30
Oct 1 12:18:25 flylive kernel: Mem-Info:
Oct 1 12:18:25 flylive kernel: active_anon:880510 inactive_anon:26048 isolated_anon:0#012 active_file:980 inactive_file:1374 isolated_file:40#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:6605 slab_unreclaimable:10583#012 mapped:30569 shmem:78464 pagetables:5854 bounce:0#012 free:21148 free_pcp:583 free_cma:0
Oct 1 12:18:25 flylive kernel: Node 0 DMA free:15336kB min:276kB low:344kB high:412kB active_anon:312kB inactive_anon:88kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:132kB shmem:132kB slab_reclaimable:28kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:24kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:40 all_unreclaimable? yes
Oct 1 12:18:25 flylive kernel: lowmem_reserve[]: 0 3580 3765 3765
Oct 1 12:18:25 flylive kernel: Node 0 DMA32 free:66092kB min:64000kB low:80000kB high:96000kB active_anon:3388036kB inactive_anon:98532kB active_file:3980kB inactive_file:5876kB unevictable:0kB isolated(anon):0kB isolated(file):160kB present:3915776kB managed:3669344kB mlocked:0kB dirty:0kB writeback:0kB mapped:116456kB shmem:302596kB slab_reclaimable:20220kB slab_unreclaimable:33640kB kernel_stack:5968kB pagetables:21748kB unstable:0kB bounce:0kB free_pcp:2088kB local_pcp:168kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 1 12:18:25 flylive kernel: lowmem_reserve[]: 0 0 184 184
Oct 1 12:18:25 flylive kernel: Node 0 Normal free:3164kB min:3304kB low:4128kB high:4956kB active_anon:133692kB inactive_anon:5572kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:258048kB managed:189324kB mlocked:0kB dirty:0kB writeback:0kB mapped:5688kB shmem:11128kB slab_reclaimable:6172kB slab_unreclaimable:8660kB kernel_stack:2304kB pagetables:1644kB unstable:0kB bounce:0kB free_pcp:244kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 1 12:18:25 flylive kernel: lowmem_reserve[]: 0 0 0 0
Oct 1 12:18:25 flylive kernel: Node 0 DMA: 1*4kB (E) 0*8kB 5*16kB (UM) 3*32kB (UEM) 3*64kB (UE) 1*128kB (E) 2*256kB (UE) 2*512kB (EM) 3*1024kB (UEM) 1*2048kB (E) 2*4096kB (M) = 15348kB
Oct 1 12:18:25 flylive kernel: Node 0 DMA32: 1001*4kB (UEM) 609*8kB (UEM) 551*16kB (UEM) 459*32kB (UEM) 277*64kB (UEM) 92*128kB (UEM) 11*256kB (UE) 3*512kB (E) 0*1024kB 0*2048kB 0*4096kB = 66236kB
Oct 1 12:18:25 flylive kernel: Node 0 Normal: 143*4kB (UEM) 61*8kB (UEM) 35*16kB (UEM) 34*32kB (UEM) 12*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3476kB
Oct 1 12:18:25 flylive kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 1 12:18:25 flylive kernel: 81118 total pagecache pages
Oct 1 12:18:25 flylive kernel: 0 pages in swap cache
Oct 1 12:18:25 flylive kernel: Swap cache stats: add 0, delete 0, find 0/0
Oct 1 12:18:25 flylive kernel: Free swap = 0kB
Oct 1 12:18:25 flylive kernel: Total swap = 0kB
Oct 1 12:18:25 flylive kernel: 1047453 pages RAM
Oct 1 12:18:25 flylive kernel: 0 pages HighMem/MovableOnly
Oct 1 12:18:25 flylive kernel: 78810 pages reserved
Oct 1 12:18:25 flylive kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Oct 1 12:18:25 flylive kernel: [ 384] 0 384 13969 5091 34 0 0 systemd-journal
Oct 1 12:18:25 flylive kernel: [ 386] 0 386 74209 2594 66 0 0 nova-agent
Oct 1 12:18:25 flylive kernel: [ 407] 0 407 29795 369 29 0 0 lvmetad
Oct 1 12:18:25 flylive kernel: [ 420] 0 420 12083 592 25 0 -1000 systemd-udevd
Oct 1 12:18:25 flylive kernel: [ 534] 0 534 13883 109 26 0 -1000 auditd
Oct 1 12:18:25 flylive kernel: [ 556] 0 556 13203 188 31 0 0 smartd
Oct 1 12:18:25 flylive kernel: [ 557] 996 557 2145 37 9 0 0 lsmd
Oct 1 12:18:25 flylive kernel: [ 558] 0 558 22642 214 47 0 0 rngd
Oct 1 12:18:25 flylive kernel: [ 560] 0 560 5385 72 17 0 0 irqbalance
Oct 1 12:18:25 flylive kernel: [ 561] 997 561 153256 1595 63 0 0 polkitd
Oct 1 12:18:25 flylive kernel: [ 565] 0 565 57041 438 63 0 0 abrtd
Oct 1 12:18:25 flylive kernel: [ 566] 0 566 56465 370 61 0 0 abrt-watch-log
Oct 1 12:18:25 flylive kernel: [ 570] 0 570 6796 289 19 0 0 systemd-logind
Oct 1 12:18:25 flylive kernel: [ 571] 81 571 16620 197 32 0 -900 dbus-daemon
Oct 1 12:18:25 flylive kernel: [ 582] 995 582 30102 122 28 0 0 chronyd
Oct 1 12:18:25 flylive kernel: [ 606] 0 606 90793 6722 95 0 0 firewalld
Oct 1 12:18:25 flylive kernel: [ 1061] 0 1061 108534 2198 45 0 0 rackspace-monit
Oct 1 12:18:25 flylive kernel: [ 1065] 0 1065 145994 3269 101 0 0 tuned
Oct 1 12:18:25 flylive kernel: [ 1296] 0 1296 23245 340 43 0 0 master
Oct 1 12:18:25 flylive kernel: [ 1303] 89 1303 23315 349 45 0 0 qmgr
Oct 1 12:18:25 flylive kernel: [ 1331] 0 1331 120817 3887 110 0 0 rsyslogd
Oct 1 12:18:25 flylive kernel: [ 1332] 0 1332 28231 256 59 0 -1000 sshd
Oct 1 12:18:25 flylive kernel: [ 1357] 0 1357 6477 52 19 0 0 atd
Oct 1 12:18:25 flylive kernel: [ 1358] 0 1358 31598 160 19 0 0 crond
Oct 1 12:18:25 flylive kernel: [ 1624] 0 1624 3022 169 11 0 0 xe-daemon
Oct 1 12:18:25 flylive kernel: [24548] 27 24548 28355 73 12 0 0 mysqld_safe
Oct 1 12:18:25 flylive kernel: [24727] 27 24727 663068 125773 322 0 0 mysqld
Oct 1 12:18:25 flylive kernel: [27028] 0 27028 13322 143 27 0 0 vsftpd
Oct 1 12:18:25 flylive kernel: [20795] 0 20795 27552 34 10 0 0 agetty
Oct 1 12:18:25 flylive kernel: [15951] 0 15951 172780 12692 260 0 0 httpd
Oct 1 12:18:25 flylive kernel: [15952] 48 15952 79019 1410 149 0 0 httpd
Oct 1 12:18:25 flylive kernel: [15957] 48 15957 888056 187683 770 0 0 httpd
Oct 1 12:18:25 flylive kernel: [16176] 48 16176 855193 194798 836 0 0 httpd
Oct 1 12:18:25 flylive kernel: [28247] 993 28247 665824 12375 202 0 0 agent
Oct 1 12:18:25 flylive kernel: [28248] 993 28248 228276 3989 63 0 0 trace-agent
Oct 1 12:18:25 flylive kernel: [28249] 993 28249 252846 5139 81 0 0 process-agent
Oct 1 12:18:25 flylive kernel: [ 6903] 48 6903 829270 156330 758 0 0 httpd
Oct 1 12:18:25 flylive kernel: [21093] 48 21093 845510 154373 726 0 0 httpd
Oct 1 12:18:25 flylive kernel: [32719] 0 32719 45104 229 45 0 0 crond
Oct 1 12:18:25 flylive kernel: [32724] 0 32724 28321 44 12 0 0 bash
Oct 1 12:18:25 flylive kernel: [32726] 0 32726 40929 802 35 0 0 python
Oct 1 12:18:25 flylive kernel: [ 483] 0 483 45104 229 45 0 0 crond
Oct 1 12:18:25 flylive kernel: [ 485] 0 485 28321 54 12 0 0 run-parts
Oct 1 12:18:25 flylive kernel: [ 497] 0 497 314045 56125 247 0 0 yum-cron
Oct 1 12:18:25 flylive kernel: [ 498] 0 498 28411 36 13 0 0 awk
Oct 1 12:18:25 flylive kernel: [ 1319] 89 1319 23271 333 45 0 0 pickup
Oct 1 12:18:25 flylive kernel: [ 2153] 0 2153 2923 60 12 0 0 xe-update-guest
Oct 1 12:18:25 flylive kernel: [ 2154] 0 2154 44078 214 42 0 0 crond
Oct 1 12:18:25 flylive kernel: Out of memory: Kill process 16176 (httpd) score 201 or sacrifice child
Oct 1 12:18:25 flylive kernel: Killed process 16176 (httpd), UID 48, total-vm:3420772kB, anon-rss:678440kB, file-rss:0kB, shmem-rss:100752kB
我只是想添加这张来自 Cloudflare 的截图，显示服务器停机期间请求量急剧上升。看来 Cloudflare 无法阻止所有请求（我猜是因为网站仍然停机了几分钟）
答案1
正如其中一条评论所提到的，问题在于服务器内存不足。
这是一个典型的配置错误。
您正在托管网站或其他可公开访问的服务互联网。任何流量都是可以预料到的。有太多“疯狂”的东西：脚本恶意软件、机器人、“digg 效果”、有针对性的攻击。
但在任何情况下，任何流量都不应导致内存不足的情况。OOM（内存不足）会导致 Linux 搜索最“肥胖”过程它会查找以释放内存。
10月1日 07:34:34 flylive 内核：内存不足：终止进程 25271 (httpd) 得分 186 或牺牲子进程
你可能分配了过多的 Apache 工作进程，因此占用了过多的 RAM。如果是这样，你可能需要限制它们合理的数字将更好地对应于您实际拥有的 RAM。
如果你想保证避免这样的停机，你绝对需要配置交换空间作为一种安全措施，它将扩展 Linux 可用的虚拟内存。但请注意，交换是一件缓慢的事情，因为它本质上是“慢速磁盘上的 RAM”。
结论
- 设置足够的交换空间以防止完全停机
- 配置 swappinness 内核参数为例如 10
- 观察
free -h输出或交换使用情况。如果服务器使用大量交换，则表明您的服务器配置不正确，您需要进一步限制 Apache 进程等内容。