我试图找出为什么我的服务器平均负载高达 150,有时甚至达到 200。
这是在 VMware ESXi 虚拟机上运行的 Ubuntu 服务器,具有 8 个 CPU 和 26 GB RAM。该服务器是一个邮件服务器(使用 Kerio Mailserver),拥有大约 500 个用户帐户。服务器在工作时间(从上午 8:30 到下午 6:30 左右)承受如此高的负载,Kerio Mailserver 是一个类似 Exchange 的邮件服务器,用户使用连接器同步他们的 Outlook。
这是我一直在尝试查找高负载原因时使用的一些命令的结果:
顶部:
top - 16:25:32 up 46 days, 20:59, 2 users, load average: 178.44, 164.61, 156.84
Tasks: 241 total, 1 running, 240 sleeping, 0 stopped, 0 zombie
%Cpu(s): 45.8 us, 53.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.5 si, 0.0 st
KiB Mem: 24689476 total, 24234796 used, 454680 free, 675324 buffers
KiB Swap: 23436284 total, 763136 used, 22673148 free. 14616960 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1253 root 20 0 13.889g 6.918g 22680 S 99.3 29.4 190621:37 mailserver
64 root 20 0 0 0 0 S 0.2 0.0 257:29.22 kswapd0
30482 root 20 0 25036 3160 2516 R 0.1 0.0 0:01.33 top
8227 root 20 0 318856 12232 11288 S 0.0 0.0 15:39.14 smbd
1 root 20 0 36408 3144 1884 S 0.0 0.0 0:06.40 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.49 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 2:32.48 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root 20 0 0 0 0 S 0.0 0.0 81:48.05 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root rt 0 0 0 0 S 0.0 0.0 0:05.96 migration/0
10 root rt 0 0 0 0 S 0.0 0.0 0:12.15 watchdog/0
11 root rt 0 0 0 0 S 0.0 0.0 0:11.93 watchdog/1
12 root rt 0 0 0 0 S 0.0 0.0 0:06.02 migration/1
13 root 20 0 0 0 0 S 0.0 0.0 2:17.19 ksoftirqd/1
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
16 root rt 0 0 0 0 S 0.0 0.0 0:10.84 watchdog/2
17 root rt 0 0 0 0 S 0.0 0.0 0:06.29 migration/2
18 root 20 0 0 0 0 S 0.0 0.0 2:22.20 ksoftirqd/2
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0
20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H
21 root rt 0 0 0 0 S 0.0 0.0 0:10.75 watchdog/3
22 root rt 0 0 0 0 S 0.0 0.0 0:06.34 migration/3
23 root 20 0 0 0 0 S 0.0 0.0 2:07.07 ksoftirqd/3
24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0
25 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H
26 root rt 0 0 0 0 S 0.0 0.0 0:11.49 watchdog/4
27 root rt 0 0 0 0 S 0.0 0.0 0:06.34 migration/4
28 root 20 0 0 0 0 S 0.0 0.0 1:50.66 ksoftirqd/4
30 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:0H
31 root rt 0 0 0 0 S 0.0 0.0 0:11.48 watchdog/5
32 root rt 0 0 0 0 S 0.0 0.0 0:06.45 migration/5
33 root 20 0 0 0 0 S 0.0 0.0 2:04.74 ksoftirqd/5
34 root 20 0 0 0 0 S 0.0 0.0 1:30.98 kworker/5:0
35 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:0H
36 root rt 0 0 0 0 S 0.0 0.0 0:11.22 watchdog/6
37 root rt 0 0 0 0 S 0.0 0.0 0:06.40 migration/6
38 root 20 0 0 0 0 S 0.0 0.0 2:23.44 ksoftirqd/6
40 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/6:0H
41 root rt 0 0 0 0 S 0.0 0.0 0:11.06 watchdog/7
42 root rt 0 0 0 0 S 0.0 0.0 0:06.50 migration/7
43 root 20 0 0 0 0 S 0.0 0.0 2:06.70 ksoftirqd/7
45 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/7:0H
46 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
47 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
48 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 perf
49 root 20 0 0 0 0 S 0.0 0.0 0:08.92 khungtaskd
IOSTAT
Linux 4.4.0-124-generic (mardom-mail) 08/06/2018 _x86_64_ (8 CPU)
08/06/2018 04:26:33 PM
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.86 88.05 95.18 148.71 1809.49 2014.93 31.36 1.03 4.21 6.02 3.05 0.82 20.08
08/06/2018 04:26:35 PM
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 1.50 164.00 255.00 234.00 3744.00 2028.00 23.61 2.06 4.21 6.00 2.26 1.35 66.00
08/06/2018 04:26:37 PM
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 160.50 211.00 25.50 3164.00 1188.00 36.80 1.30 5.45 5.90 1.80 2.88 68.00
08/06/2018 04:26:39 PM
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.50 207.00 129.00 63.50 2808.00 1942.00 49.35 1.16 6.06 8.48 1.13 3.35 64.40
* sdb 是存储邮件数据的设备 *
系统控制:
fs.file-max = 2097152
vm.swappiness = 10
vm.dirty_ratio = 60
vm.dirty_background_ratio = 2
net.ipv4.tcp_synack_retries = 2
net.ipv4.ip_local_port_range = 2000 65535
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15
net.core.rmem_default = 31457280
net.core.rmem_max = 12582912
net.core.wmem_default = 31457280
net.core.wmem_max = 12582912
net.ipv4.tcp_max_syn_backlog = 4096
net.core.somaxconn = 4096
net.core.netdev_max_backlog = 65536
net.core.optmem_max = 25165824
net.ipv4.tcp_mem = 65536 131072 262144
net.ipv4.udp_mem = 65536 131072 262144
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.udp_rmem_min = 16384
net.ipv4.tcp_wmem = 8192 65536 16777216
net.ipv4.udp_wmem_min = 16384
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_rmem = 20240 87380 16582912
net.ipv4.tcp_wmem = 20240 87380 16582912
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_no_metrics_save = 1
net.core.netdev_max_backlog = 8000
NETSTAT-S 这是“netstat -s”命令输出的一部分:
TcpExt:
147823 SYN cookies sent
90353 SYN cookies received
649892 invalid SYN cookies received
35112 resets received for embryonic SYN_RECV sockets
598 packets pruned from receive queue because of socket buffer overrun
1248 ICMP packets dropped because they were out-of-window
1750991 TCP sockets finished time wait in fast timer
115289 TCP sockets finished time wait in slow timer
198679 passive connections rejected because of time stamp
313672 packets rejects in established connections because of timestamp
30746108 delayed acks sent
52041 delayed acks further delayed because of locked socket
Quick ack mode was activated 3386942 times
708473 times the listen queue of a socket overflowed
1060673 SYNs to LISTEN sockets dropped
3845729 packets directly queued to recvmsg prequeue.
356530 bytes directly in process context from backlog
352524121 bytes directly received in process context from prequeue
477210889 packet headers predicted
269376 packets header predicted and directly queued to user
863932645 acknowledgments not containing data payload received
1164901897 predicted acknowledgments
5140 times recovered from packet loss due to fast retransmit
3122668 times recovered from packet loss by selective acknowledgements
17 bad SACK blocks received
Detected reordering 2283 times using FACK
Detected reordering 2792 times using SACK
Detected reordering 79 times using reno fast retransmit
Detected reordering 7587 times using time stamp
11657 congestion windows fully recovered without slow start
7574 congestion windows partially recovered using Hoe heuristic
113083 congestion windows recovered without slow start by DSACK
1749877 congestion windows recovered without slow start after partial ack
TCPLostRetransmit: 505007
446 timeouts after reno fast retransmit
69083 timeouts after SACK recovery
37057 timeouts in loss state
10089153 fast retransmits
193583 forward retransmits
842022 retransmits in slow start
5758947 other TCP timeouts
TCPLossProbes: 16066532
TCPLossProbeRecovery: 213644
551 classic Reno fast retransmits failed
81107 SACK retransmits failed
74 times receiver scheduled too late for direct processing
9695 packets collapsed in receive queue due to low socket buffer
3556447 DSACKs sent for old packets
99902 DSACKs sent for out of order packets
11030608 DSACKs received
47032 DSACKs for out of order packets received
2171835 connections reset due to unexpected data
1370307 connections reset due to early user close
1984601 connections aborted due to timeout
TCPSACKDiscard: 52
TCPDSACKIgnoredOld: 38928
TCPDSACKIgnoredNoUndo: 4151872
TCPSpuriousRTOs: 50607
TCPSackShifted: 11911393
TCPSackMerged: 12335047
TCPSackShiftFallback: 12395942
IPReversePathFilter: 1
TCPReqQFullDoCookies: 161423
TCPRetransFail: 123
TCPRcvCoalesce: 306299180
TCPOFOQueue: 6851749
TCPOFOMerge: 94743
TCPChallengeACK: 74466
TCPSYNChallenge: 6119
TCPFastOpenCookieReqd: 3
TCPSpuriousRtxHostQueues: 1761
TCPAutoCorking: 59800735
TCPFromZeroWindowAdv: 52396
TCPToZeroWindowAdv: 52396
TCPWantZeroWindowAdv: 416614
TCPSynRetrans: 2205685
TCPOrigDataSent: -230466592
TCPHystartTrainDetect: 950609
TCPHystartTrainCwnd: 17827670
TCPHystartDelayDetect: 649377
TCPHystartDelayCwnd: 27358928
TCPACKSkippedSynRecv: 227586
TCPACKSkippedPAWS: 32292
TCPACKSkippedSeq: 30707
TCPACKSkippedFinWait2: 7
TCPACKSkippedTimeWait: 48
TCPACKSkippedChallenge: 1494
TCPWinProbe: 279293
TCPKeepAlive: 1685
当我运行“netstat -nat | wc -l”时,我得到 2988 个连接。
正如您在“top”输出中看到的,“mailserver”进程消耗了> 90%的CPU,但这也可能导致高负载?
** 更新 ** 作为更新,我注意到当我阻止某些 VLAN 连接到服务器(具有大约 60 台或更多计算机的 VLAN)时,负载开始下降。这可能是与网络相关的问题吗?或者服务器的总体容量?