我在使用 Debian 7 (LAMP 堆栈) 时遇到以下问题:
top - 12:16:08 up 16:36, 4 users, load average: 27.97, 27.66, 27.42
Tasks: 240 total, 2 running, 238 sleeping, 0 stopped, 0 zombie
%Cpu(s): 8.5 us, 3.3 sy, 86.8 ni, 1.0 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem: 32980448 total, 32278828 used, 701620 free, 264368 buffers
KiB Swap: 11882492 total, 0 used, 11882492 free, 29265456 cached
因此,即使所有服务(http、postfix、bind)都关闭(并且网络完全被 iptables 阻塞),平均负载也始终约为 24。它接近 100% 的良好负载和约 27 la,但正在运行的进程只有 3-5 个。网站可以运行,但现在根本没有空闲。
有什么想法吗?我如何确定哪些进程正在“niced”运行?
顺便说一下,服务器有 12 个核心,总共有 24 个线程...这可能是个 bug,每个核心都“挂起”了
更新 进程列表:
top - 12:29:46 up 16:49, 3 users, load average: 27.55, 27.30, 27.35
Tasks: 234 total, 4 running, 229 sleeping, 0 stopped, 1 zombie
%Cpu(s): 10.1 us, 4.2 sy, 84.5 ni, 0.7 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem: 32980448 total, 32363708 used, 616740 free, 266392 buffers
KiB Swap: 11882492 total, 0 used, 11882492 free, 29357760 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17760 mysql 20 0 2278m 972m 8544 S 71 3.0 330:59.07 mysqld
31139 www-data 20 0 81896 4524 2076 S 4 0.0 4:27.55 nginx
31140 www-data 20 0 81776 3256 1064 R 3 0.0 4:23.99 nginx
52110 web12543 20 0 240m 14m 3328 R 3 0.0 0:00.09 apache2
33669 root 20 0 236m 7484 580 S 1 0.0 0:01.18 apache2
40600 root 20 0 236m 7484 580 S 1 0.0 0:00.76 apache2
43386 root 20 0 236m 7484 580 S 1 0.0 0:00.57 apache2
43645 root 20 0 236m 7480 576 S 1 0.0 0:00.54 apache2
44197 root 20 0 236m 7484 580 S 1 0.0 0:00.49 apache2
46111 root 20 0 236m 7480 576 S 1 0.0 0:00.38 apache2
47041 root 20 0 236m 7480 576 S 1 0.0 0:00.33 apache2
47644 root 20 0 236m 7480 576 S 1 0.0 0:00.28 apache2
48683 root 20 0 0 0 0 Z 1 0.0 0:00.23 apache2
48765 root 20 0 236m 7480 576 S 1 0.0 0:00.23 apache2
49749 root 20 0 236m 7480 576 S 1 0.0 0:00.14 apache2
49833 root 20 0 236m 7480 576 S 1 0.0 0:00.14 apache2
49931 root 20 0 236m 7480 576 S 1 0.0 0:00.13 apache2
49935 root 20 0 236m 7480 576 R 1 0.0 0:00.12 apache2
50691 root 20 0 236m 7480 576 S 1 0.0 0:00.09 apache2
50692 root 20 0 236m 7480 576 S 1 0.0 0:00.08 apache2
50786 root 20 0 236m 7480 576 S 1 0.0 0:00.08 apache2
50787 root 20 0 236m 7484 580 S 1 0.0 0:00.07 apache2
51487 root 20 0 236m 7480 576 S 1 0.0 0:00.04 apache2
51588 root 20 0 25320 1584 1080 R 1 0.0 0:00.05 top
31138 www-data 20 0 81824 3436 1064 S 0 0.0 4:18.99 nginx
45688 root 20 0 236m 7480 576 S 0 0.0 0:00.42 apache2
46535 root 20 0 236m 7480 576 S 0 0.0 0:00.35 apache2
46869 root 20 0 236m 7480 576 S 0 0.0 0:00.34 apache2
49933 root 20 0 236m 7480 576 S 0 0.0 0:00.12 apache2
49934 root 20 0 236m 7480 576 S 0 0.0 0:00.12 apache2
51311 root 20 0 236m 7480 576 S 0 0.0 0:00.03 apache2
52085 web12543 20 0 237m 11m 3272 S 0 0.0 0:00.01 apache2
62551 root 20 0 25492 1864 1192 S 0 0.0 0:07.21 top
1 root 20 0 12708 844 704 S 0 0.0 0:04.20 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:00.92 ksoftirqd/0
5 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:0
6 root rt 0 0 0 0 S 0 0.0 0:00.38 migration/0
7 root rt 0 0 0 0 S 0 0.0 0:00.30 watchdog/0
8 root rt 0 0 0 0 S 0 0.0 0:00.20 migration/1
9 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/1:0
10 root 20 0 0 0 0 S 0 0.0 0:00.26 ksoftirqd/1
12 root rt 0 0 0 0 S 0 0.0 0:00.21 watchdog/1
13 root rt 0 0 0 0 S 0 0.0 0:00.19 migration/2
14 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/2:0
15 root 20 0 0 0 0 S 0 0.0 0:00.24 ksoftirqd/2
16 root rt 0 0 0 0 S 0 0.0 0:00.16 watchdog/2
17 root rt 0 0 0 0 S 0 0.0 0:00.14 migration/3
18 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/3:0
19 root 20 0 0 0 0 S 0 0.0 0:00.18 ksoftirqd/3
20 root rt 0 0 0 0 S 0 0.0 0:00.13 watchdog/3
21 root rt 0 0 0 0 S 0 0.0 0:00.14 migration/4
23 root 20 0 0 0 0 S 0 0.0 0:00.22 ksoftirqd/4
24 root rt 0 0 0 0 S 0 0.0 0:00.13 watchdog/4
25 root rt 0 0 0 0 S 0 0.0 0:00.14 migration/5
26 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/5:0
27 root 20 0 0 0 0 S 0 0.0 0:00.25 ksoftirqd/5
28 root rt 0 0 0 0 S 0 0.0 0:00.12 watchdog/5
29 root rt 0 0 0 0 S 0 0.0 0:00.32 migration/6
31 root 20 0 0 0 0 S 0 0.0 0:00.79 ksoftirqd/6
32 root rt 0 0 0 0 S 0 0.0 0:00.15 watchdog/6
33 root rt 0 0 0 0 S 0 0.0 0:00.27 migration/7
34 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/7:0
35 root 20 0 0 0 0 S 0 0.0 0:00.28 ksoftirqd/7
36 root rt 0 0 0 0 S 0 0.0 0:00.15 watchdog/7
37 root rt 0 0 0 0 S 0 0.0 0:00.19 migration/8
39 root 20 0 0 0 0 S 0 0.0 0:00.25 ksoftirqd/8
40 root rt 0 0 0 0 S 0 0.0 0:00.16 watchdog/8
41 root rt 0 0 0 0 S 0 0.0 0:00.18 migration/9
43 root 20 0 0 0 0 S 0 0.0 0:00.28 ksoftirqd/9
44 root rt 0 0 0 0 S 0 0.0 0:00.16 watchdog/9
45 root rt 0 0 0 0 S 0 0.0 0:00.19 migration/10
46 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/10:0
47 root 20 0 0 0 0 S 0 0.0 0:00.21 ksoftirqd/10
48 root rt 0 0 0 0 S 0 0.0 0:00.16 watchdog/10
49 root rt 0 0 0 0 S 0 0.0 0:00.16 migration/11
50 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/11:0
51 root 20 0 0 0 0 S 0 0.0 0:00.18 ksoftirqd/11
52 root rt 0 0 0 0 S 0 0.0 0:00.16 watchdog/11
53 root rt 0 0 0 0 S 0 0.0 0:00.44 migration/12
54 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/12:0
55 root 20 0 0 0 0 S 0 0.0 0:00.10 ksoftirqd/12
56 root rt 0 0 0 0 S 0 0.0 0:00.12 watchdog/12
57 root rt 0 0 0 0 S 0 0.0 0:00.35 migration/13
59 root 20 0 0 0 0 S 0 0.0 0:00.08 ksoftirqd/13
60 root rt 0 0 0 0 S 0 0.0 0:00.12 watchdog/13
61 root rt 0 0 0 0 S 0 0.0 0:00.28 migration/14
62 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/14:0
63 root 20 0 0 0 0 S 0 0.0 0:00.10 ksoftirqd/14
64 root rt 0 0 0 0 S 0 0.0 0:00.11 watchdog/14
65 root rt 0 0 0 0 S 0 0.0 0:00.24 migration/15
66 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/15:0
67 root 20 0 0 0 0 S 0 0.0 0:00.05 ksoftirqd/15
68 root rt 0 0 0 0 S 0 0.0 0:00.11 watchdog/15
69 root rt 0 0 0 0 S 0 0.0 0:00.23 migration/16
70 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/16:0
71 root 20 0 0 0 0 S 0 0.0 0:00.07 ksoftirqd/16
72 root rt 0 0 0 0 S 0 0.0 0:00.11 watchdog/16
73 root rt 0 0 0 0 S 0 0.0 0:00.21 migration/17
74 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/17:0
75 root 20 0 0 0 0 S 0 0.0 0:00.11 ksoftirqd/17
76 root 20 0 0 0 0 S 0 0.0 0:04.63 kworker/8:1
79 root 20 0 0 0 0 S 0 0.0 0:04.42 kworker/5:1
80 root 20 0 0 0 0 S 0 0.0 0:04.36 kworker/4:1
81 root 20 0 0 0 0 S 0 0.0 0:04.20 kworker/3:1
82 root 20 0 0 0 0 S 0 0.0 0:04.33 kworker/2:1
83 root rt 0 0 0 0 S 0 0.0 0:00.12 watchdog/17
84 root rt 0 0 0 0 S 0 0.0 0:00.53 migration/18
85 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/18:0
86 root 20 0 0 0 0 S 0 0.0 0:00.06 ksoftirqd/18
87 root 20 0 0 0 0 S 0 0.0 0:06.01 kworker/1:1
88 root rt 0 0 0 0 S 0 0.0 0:00.14 watchdog/18
89 root rt 0 0 0 0 S 0 0.0 0:00.41 migration/19
91 root 20 0 0 0 0 S 0 0.0 0:00.08 ksoftirqd/19
92 root rt 0 0 0 0 S 0 0.0 0:00.17 watchdog/19
93 root rt 0 0 0 0 S 0 0.0 0:00.34 migration/20
94 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/20:0
95 root 20 0 0 0 0 S 0 0.0 0:00.06 ksoftirqd/20
96 root rt 0 0 0 0 S 0 0.0 0:00.14 watchdog/20
97 root rt 0 0 0 0 S 0 0.0 0:00.30 migration/21
98 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/21:0
99 root 20 0 0 0 0 S 0 0.0 0:00.12 ksoftirqd/21
100 root rt 0 0 0 0 S 0 0.0 0:00.14 watchdog/21
101 root rt 0 0 0 0 S 0 0.0 0:00.28 migration/22
102 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/22:0
103 root 20 0 0 0 0 S 0 0.0 0:00.06 ksoftirqd/22
104 root rt 0 0 0 0 S 0 0.0 0:00.14 watchdog/22
105 root rt 0 0 0 0 S 0 0.0 0:00.26 migration/23
106 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/23:0
107 root 20 0 0 0 0 S 0 0.0 0:00.07 ksoftirqd/23
108 root rt 0 0 0 0 S 0 0.0 0:00.14 watchdog/23
109 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
110 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
111 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs
112 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
113 root 20 0 0 0 0 S 0 0.0 0:00.14 sync_supers
114 root 20 0 0 0 0 S 0 0.0 0:00.00 bdi-default
115 root 0 -20 0 0 0 S 0 0.0 0:00.00 kintegrityd
116 root 0 -20 0 0 0 S 0 0.0 0:00.00 kblockd
117 root 20 0 0 0 0 S 0 0.0 0:00.06 khungtaskd
118 root 20 0 0 0 0 S 0 0.0 0:06.12 kswapd0
119 root 20 0 0 0 0 S 0 0.0 0:04.55 kswapd1
120 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd
121 root 39 19 0 0 0 S 0 0.0 0:00.00 khugepaged
122 root 20 0 0 0 0 S 0 0.0 0:00.00 fsnotify_mark
123 root 0 -20 0 0 0 S 0 0.0 0:00.00 crypto
126 root 20 0 0 0 0 S 0 0.0 0:04.37 kworker/15:1
129 root 20 0 0 0 0 S 0 0.0 0:04.41 kworker/13:1
145 root 20 0 0 0 0 S 0 0.0 0:04.76 kworker/9:1
190 root 20 0 0 0 0 S 0 0.0 0:04.60 kworker/12:1
191 root 20 0 0 0 0 S 0 0.0 0:04.40 kworker/17:1
192 root 20 0 0 0 0 S 0 0.0 0:04.19 kworker/16:1
193 root 20 0 0 0 0 S 0 0.0 0:04.84 kworker/11:1
194 root 20 0 0 0 0 S 0 0.0 0:04.80 kworker/10:1
376 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff
377 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd
461 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
463 root 20 0 0 0 0 S 0 0.0 0:00.02 hpsa
467 root 20 0 0 0 0 S 0 0.0 0:00.97 kworker/u:1
476 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
477 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2
480 root 20 0 0 0 0 S 0 0.0 0:04.88 kworker/23:1
481 root 20 0 0 0 0 S 0 0.0 0:04.76 kworker/19:1
484 root 20 0 0 0 0 S 0 0.0 0:04.81 kworker/18:1
486 root 20 0 0 0 0 S 0 0.0 0:04.75 kworker/20:1
489 root 20 0 0 0 0 S 0 0.0 0:04.75 kworker/22:1
497 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/19:2
507 root 20 0 0 0 0 S 0 0.0 0:04.85 kworker/21:1
508 root 0 -20 0 0 0 S 0 0.0 0:00.00 kdmflush
517 root 0 -20 0 0 0 S 0 0.0 0:00.00 kdmflush
524 root 20 0 0 0 0 S 0 0.0 0:04.36 kworker/14:2
533 root 20 0 0 0 0 S 0 0.0 0:29.16 jbd2/dm-0-8
534 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
675 root 20 0 23772 1692 816 S 0 0.0 0:00.16 udevd
1008 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/4:2
1075 root 0 -20 0 0 0 S 0 0.0 0:00.00 edac-poller
1080 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/8:2
1081 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/9:2
1099 root 0 -20 0 0 0 S 0 0.0 0:00.00 kpsmoused
2291 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/13:2
2637 root 20 0 0 0 0 S 0 0.0 0:44.75 flush-254:0
2730 root 20 0 118m 2348 1152 S 0 0.0 0:05.10 rsyslogd
2782 daemon 20 0 18736 368 208 S 0 0.0 0:00.00 atd
2851 root 20 0 8240 684 528 S 0 0.0 0:00.01 acpid
2878 messageb 20 0 31864 488 220 S 0 0.0 0:00.00 dbus-daemon
2923 root 20 0 51992 1308 688 S 0 0.0 0:00.00 sshd
3557 root 20 0 22468 1112 832 S 0 0.0 0:00.67 cron
3627 root 20 0 23584 1004 676 S 0 0.0 0:00.27 dovecot
3782 dovecot 20 0 15244 1068 884 S 0 0.0 0:00.10 anvil
3783 root 20 0 15240 1248 904 S 0 0.0 0:00.06 log
3830 root 20 0 53648 9912 2120 S 0 0.0 0:04.89 munin-node
4399 root 20 0 39248 2628 1860 S 0 0.0 0:00.22 ihttpd
4450 root 20 0 54220 1568 1164 S 0 0.0 0:00.02 login
4451 root 20 0 20384 992 816 S 0 0.0 0:00.00 getty
4452 root 20 0 20384 988 816 S 0 0.0 0:00.00 getty
4453 root 20 0 20384 988 816 S 0 0.0 0:00.00 getty
4454 root 20 0 20384 992 816 S 0 0.0 0:00.00 getty
4455 root 20 0 20384 988 816 S 0 0.0 0:00.00 getty
5721 web12542 20 0 237m 9860 2368 S 0 0.0 0:00.00 apache2
5724 mysql 20 0 2011m 649m 256 S 0 2.0 0:00.00 mysqld
17427 root 20 0 8304 776 628 S 0 0.0 0:00.02 mysqld_safe
23825 postfix 20 0 42008 2476 1904 S 0 0.0 0:00.01 pickup
28501 root 20 0 0 0 0 S 0 0.0 0:04.18 kworker/0:0
28689 root 20 0 23348 1080 552 S 0 0.0 0:00.00 udevd
31059 root 20 0 39944 2520 1944 S 0 0.0 0:00.75 master
31073 postfix 20 0 42592 3188 2032 S 0 0.0 0:00.94 qmgr
31093 root 20 0 236m 12m 5952 S 0 0.0 0:14.22 apache2
31137 root 20 0 81476 1836 336 S 0 0.0 0:00.00 nginx
31141 www-data 20 0 81900 3512 1064 S 0 0.0 3:47.78 nginx
31878 root 20 0 0 0 0 S 0 0.0 0:05.02 kworker/0:1
32472 postfix 20 0 42008 2364 1824 S 0 0.0 0:00.00 anvil
35437 postfix 20 0 45116 3864 2508 S 0 0.0 0:00.49 tlsmgr
35880 root 20 0 206m 12m 8804 S 0 0.0 0:09.27 ispmgr
37067 postfix 20 0 46828 4368 3316 S 0 0.0 0:00.01 smtp
44298 postfix 20 0 66532 5152 3496 S 0 0.0 0:00.01 smtpd
48002 postfix 20 0 42004 2368 1828 S 0 0.0 0:00.00 error
48004 postfix 20 0 42040 2380 1840 S 0 0.0 0:00.00 bounce
48972 root 20 0 236m 7480 576 S 0 0.0 0:00.12 apache2
48975 root 20 0 74876 5152 2940 S 0 0.0 0:00.82 sshd
49288 root 20 0 0 0 0 S 0 0.0 0:04.01 kworker/7:1
49471 root 20 0 21636 2384 1724 S 0 0.0 0:00.31 bash
50882 web12542 20 0 242m 16m 3140 S 0 0.1 0:06.45 apache2
51111 bind 20 0 332m 55m 2352 S 0 0.2 0:04.50 named
52083 web12543 20 0 236m 9228 2004 S 0 0.0 0:00.00 apache2
52114 root 20 0 0 0 0 S 0 0.0 0:00.15 kworker/6:2
54542 root 20 0 236m 7480 576 S 0 0.0 0:00.10 apache2
55209 postfix 20 0 42020 2388 1844 S 0 0.0 0:00.02 trivial-rewrite
56098 postfix 20 0 42340 3180 2352 S 0 0.0 0:00.03 cleanup
56171 web12542 20 0 240m 14m 3300 S 0 0.0 0:00.08 apache2
56214 root 20 0 23348 1124 584 S 0 0.0 0:00.00 udevd
56218 root 20 0 21560 2248 1648 S 0 0.0 0:00.01 bash
56282 root 20 0 73352 3732 2936 S 0 0.0 0:00.48 sshd
61268 root 20 0 21544 2280 1696 S 0 0.0 0:00.04 bash
63955 root 20 0 0 0 0 S 0 0.0 0:01.36 kworker/6:0
更新2 现在,两天后平均负载显示 0.00 - 一直如此!!!
top - 10:57:06 up 2 days, 15:17, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 377 total, 10 running, 366 sleeping, 0 stopped, 1 zombie
Cpu(s): 29.1%us, 11.8%sy, 57.9%ni, 0.2%id, 0.0%wa, 0.0%hi, 0.9%si, 0.0%st
Mem: 32980448k total, 31662152k used, 1318296k free, 293472k buffers
Swap: 11882492k total, 744k used, 11881748k free, 25885360k cached
Debian 有漏洞?!
更新3
root@rd1:~# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
40 0 2076 1233156 294124 25595008 0 0 26 35 0 6 94 5 1 0
29 0 2076 1240300 294124 25595200 0 0 0 428 22645 61952 86 14 0 0
28 0 2076 1233792 294124 25595456 0 0 36 364 12324 23591 92 8 0 0
34 0 2076 1251060 294124 25595608 0 0 0 556 13593 33874 91 8 1 0
39 0 2076 1223912 294124 25595752 0 0 0 2332 38084 89618 81 19 0 0
35 0 2076 1205256 294124 25595960 0 0 0 556 25791 62903 81 18 0 0
27 0 2076 1238748 294124 25596768 0 0 0 344 20367 46819 84 16 1 0
35 0 2076 1211688 294124 25597256 0 0 32 484 13068 28677 87 12 1 0
35 0 2076 1216440 294124 25597800 0 0 32 132 26783 75160 81 19 0 0
38 0 2076 1218932 294124 25598188 0 0 64 6952 19746 47230 83 16 0 0
30 0 2076 1244800 294124 25598728 0 0 0 4 16268 35489 87 13 0 0
33 0 2076 1229928 294124 25599048 0 0 184 404 10958 21926 88 11 1 0
30 0 2076 1237536 294124 25599436 0 0 32 592 12408 29603 89 11 0 0
30 1 2076 1234664 294124 25599788 0 0 12 416 13072 31855 88 12 0 0
32 0 2076 1247136 294124 25599924 0 0 12 1928 12717 23503 88 11 1 0
31 0 2076 1236484 294124 25600312 0 0 32 2880 12893 33544 88 12 0 0