服务器突然变得太慢,出现超时错误

服务器突然变得太慢,出现超时错误

我的服务器突然变慢了,我检查了 http 级别的所有网站的所有日志,但没有任何日志表明为什么会发生这种情况,如果有人有任何建议,请

我的设置:

Centos 6.8

Nginx 1.10

php-fpm 7.0

Mariadb 10.1

内存 32GB,剩余 13GB

一年后,问题依然存在

注意:我能够将负载减少到 10 秒的唯一方法是将所有缓存移至内存 tmpfs

- 启用 Nginx 缓存后,负载将跳回到 100 秒,要么将缓存文件夹添加到内存中,要么不添加(我无法理解,因为之前它运行良好)

缓存系统:Opcache Filecache(Codeigniter)Memcached Redis

netstat -an | wc -l
22445

netstat -an | grep :80 | wc -l
51196

netstat -ant | awk '{print $6}' | sort | uniq -c | sort -n
  1 established)
  1 FIN_WAIT1
  1 Foreign
  2 SYN_RECV
  3 LAST_ACK
  6 LISTEN
21144 ESTABLISHED
30908 TIME_WAIT

netstat -i 15

Kernel Interface table
Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500   0 218927712      0      0      0 257329662      0      0      0 BMRU
lo        65536   0 310876055      0      0      0 310876055      0      0      0 LRU
eth0       1500   0 218994531      0      0      0 257405202      0      0      0 BMRU
lo        65536   0 311006988      0      0      0 311006988      0      0      0 LRU
eth0       1500   0 219060371      0      0      0 257479672      0      0      0 BMRU
lo        65536   0 311139650      0      0      0 311139650      0      0      0 LRU
eth0       1500   0 219131683      0      0      0 257561956      0      0      0 BMRU
lo        65536   0 311270631      0      0      0 311270631      0      0      0 LRU
eth0       1500   0 219201250      0      0      0 257640969      0      0      0 BMRU
lo        65536   0 311407443      0      0      0 311407443      0      0      0 LRU
eth0       1500   0 219268062      0      0      0 257717388      0      0      0 BMRU
lo        65536   0 311534133      0      0      0 311534133      0      0      0 LRU
eth0       1500   0 219335153      0      0      0 257792649      0      0      0 BMRU
lo        65536   0 311666747      0      0      0 311666747      0      0      0 LRU

iostat 15

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.98    0.52    3.15    0.30    0.00   85.05

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               20.79 833.21       490.35   76292548   44898504

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          28.67    0.00    9.33    0.83    0.00   61.17

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              46.67        32.00       873.60        480      13104

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          27.82    0.00   10.18    0.15    0.00   61.85

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              25.47        33.60      1692.80        504      25392

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          28.84    0.00   10.21    0.06    0.00   60.89

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              11.27        32.53       712.00        488      10680

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          27.83    0.00   10.45    0.95    0.00   60.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              58.67        18.67      1861.33        280      27920

虚拟机状态 15

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
12  0  0 11291076 1118568 12297544    0    0    17    10    8    4 11  3 
85  0  0
3  0   0 11248336 1118568 12299980    0    0     9   399 29547 52458 27  8 65  0  0
11  0  0 11224000 1118568 12303336    0    0    26   683 29830 52331 27  
9 63  1  0
21  0  0 11193280 1118568 12306744    0    0    39   347 31722 67277 31 10 59  0  0
16  0  0 11158436 1118568 12310428    0    0    17   809 30677 62257 27 
10 62  1  0
10  0  0 11129768 1118568 12314088    0    0    32   472 32385 61133 30 10 60  0  0
7  2   0 11120616 1118572 12317448    0    0    18   761 31739 66259 29 10 60  0  0

顶部 -b -n 1 | 头部

top - 20:17:07 up 1 day,  1:12,  1 user,  load average: 3.95, 4.11, 4.29
Tasks: 641 total,  15 running, 626 sleeping,   0 stopped,   0 zombie
Cpu(s): 10.8%us,  2.9%sy,  0.5%ni, 85.3%id,  0.3%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  32865012k total, 21206392k used, 11658620k free,  1118552k buffers
Swap:  8388604k total,        0k used,  8388604k free, 12165180k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
27709 mysql     20   0 5189m 1.0g  13m S 559.9  3.1   1800:22 mysqld
2750 nginx     10 -10  437m  42m 1488 R 19.2  0.1  46:05.13 nginx
3135 nginx     10 -10  433m  41m 1496 S 15.4  0.1  46:45.90 nginx

物联网

Total DISK READ: 64.98 K/s | Total DISK WRITE: 667.90 K/s
TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
884 be/3 root        0.00 B/s  491.00 K/s  0.00 %  4.80 % [jbd2/sda5-8]
6091 be/4 nginx      64.98 K/s   64.98 K/s  0.00 %  0.16 % php-fpm: pool www
16943 be/4 mysql       0.00 B/s    3.61 K/s  0.00 %  0.00 % mysqld
27829 be/4 mysql       0.00 B/s    7.22 K/s  0.00 %  0.00 % mysqld
27862 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld 
27887 be/4 mysql       0.00 B/s  238.28 K/s  0.00 %  0.00 % mysqld 
2745 be/2 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % nginx: worker process
2747 be/2 nginx       0.00 B/s    7.22 K/s  0.00 %  0.00 % nginx: worker process
2748 be/2 nginx       0.00 B/s   10.83 K/s  0.00 %  0.00 % nginx: worker process
2750 be/2 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % nginx: worker process
2751 be/2 nginx       0.00 B/s    7.22 K/s  0.00 %  0.00 % nginx: worker process
2788 be/2 nginx       0.00 B/s   14.44 K/s  0.00 %  0.00 % nginx: worker process
2872 be/2 nginx       0.00 B/s   10.83 K/s  0.00 %  0.00 % nginx: worker process
2919 be/2 nginx       0.00 B/s    7.22 K/s  0.00 %  0.00 % nginx: worker process
2920 be/2 nginx       0.00 B/s   10.83 K/s  0.00 %  0.00 % nginx: worker process
2991 be/2 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % nginx: worker process
3038 be/2 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % nginx: worker process
3121 be/2 nginx       0.00 B/s    7.22 K/s  0.00 %  0.00 % nginx: worker process
3135 be/2 nginx       0.00 B/s   14.44 K/s  0.00 %  0.00 % nginx: worker process
3218 be/2 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % nginx: worker process
3256 be/2 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % nginx: worker process
3258 be/2 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % nginx: worker process
3316 be/2 nginx       0.00 B/s   10.83 K/s  0.00 %  0.00 % nginx: worker process
3325 be/2 nginx       0.00 B/s   14.44 K/s  0.00 %  0.00 % nginx: worker process
3390 be/2 nginx       0.00 B/s    7.22 K/s  0.00 %  0.00 % nginx: worker process
28264 be/4 mysql       0.00 B/s   57.76 K/s  0.00 %  0.00 % mysqld 
27959 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld 
28000 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
5892 be/4 nginx       0.00 B/s   25.27 K/s  0.00 %  0.00 % php-fpm: pool www
5918 be/4 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % php-fpm: pool www
5938 be/4 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % php-fpm: pool www
5958 be/4 nginx       0.00 B/s   28.88 K/s  0.00 %  0.00 % php-fpm: pool www
5981 be/4 nginx       0.00 B/s   43.32 K/s  0.00 %  0.00 % php-fpm: pool www
5990 be/4 nginx       0.00 B/s   57.76 K/s  0.00 %  0.00 % php-fpm: pool www
6037 be/4 nginx       0.00 B/s   28.88 K/s  0.00 %  0.00 % php-fpm: pool www
6062 be/4 nginx       0.00 B/s   21.66 K/s  0.00 %  0.00 % php-fpm: pool www
6114 be/4 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % php-fpm: pool www
1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
3 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
4 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
5 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/0]
6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
8 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/1]
9 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]
10 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/1]
11 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/2]
12 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/2]
13 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/2]
14 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/2]
15 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/3]
16 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/3]
17 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/3]
18 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/3]
19 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/4]
20 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/4]
21 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/4]
22 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/4]
23 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/5]
24 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/5]

Nginx 配置

user  nginx nginx; 
worker_processes  24; 
worker_rlimit_nofile 300000;


pid  /var/run/nginx.pid;

events {
use                 epoll;
worker_connections  20192;
multi_accept        on;
   }

http {
server_tokens off;
include       /etc/nginx/mime.types;
default_type  application/octet-stream;
set_real_ip_from 127.0.0.1;
real_ip_header X-Forwarded-For;
sendfile        on;
tcp_nopush      on;
charset UTF-8;


fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;

proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_connect_timeout  600s;
proxy_send_timeout  600s;
proxy_read_timeout  600s;
fastcgi_send_timeout 600s;
fastcgi_read_timeout 600s;

keepalive_timeout  60;
keepalive_disable   none;
max_ranges                1;
reset_timedout_connection on;
tcp_nodelay               on;

open_file_cache max=10000 inactive=5m;
open_file_cache_valid 10m;
open_file_cache_min_uses 5;
open_file_cache_errors off;

client_body_buffer_size  1m;   
client_header_buffer_size 1m;   
client_max_body_size 1m;  
large_client_header_buffers 2 1m;  

gzip  on;
gzip_comp_level 5;
gzip_min_length 1000;
gzip_proxied any;
gzip_vary on;
gzip_types
  application/atom+xml
  application/javascript
  application/json
  application/rss+xml
  application/vnd.ms-fontobject
  application/x-font-ttf
  application/x-web-app-manifest+json
  application/xhtml+xml
  application/xml
  font/opentype
  image/svg+xml
  image/x-icon
  text/css
  text/plain
  text/x-component;

 #limit_req_zone  $binary_remote_addr  zone=gulag:1m   rate=50r/m;

# Upstream to abstract backend connection(s) for PHP.
upstream php {
server unix:/tmp/php-fpm.sock;
server 127.0.0.1:9000;
}

include /etc/nginx/conf.d/*;
 }

php-fpm 配置 http://pasted.co/d6a2fa9c

mysql 配置

port = 3306
#bind-address    =  
default-storage-engine = ARIA 
aria-pagecache-buffer-size= 2048M 
key_buffer = 2024M 
query_cache_size = 64M 
query_cache_limit = 512M 
max_connections = 3000 
thread_cache_size = 512 
query_cache_min_res_unit = 0 
tmp_table_size = 32M 
max_heap_table_size = 32M 
table_cache=1024 
concurrent_insert=2
max_allowed_packet = 64M 
sort_buffer_size = 128K 
read_buffer_size = 512K 
read_rnd_buffer_size = 512K 
net_buffer_length = 2K 
thread_stack = 512K 
wait_timeout = 300 
table_definition_cache = 4000 
thread_handling = pool-of-threads 
host_cache_size = 2000 
skip_name_resolve 
thread-pool-max-threads= 1000

php.in http://pasted.co/523a81db

nginx 错误日志 除此之外什么也没有

[警告] 10063#10063:服务器名称冲突

mysql 错误日志

除了重启日志外也是空的

php-fpm 错误日志

[01-Jun-2016 21:54:00] WARNING: [pool www] child 9596, script '/var/www/index.php' (request: "POST /index.php") execution timed out (302.124548 sec), terminating 
[01-Jun-2016 21:54:00] WARNING: [pool www] child 9596 exited on signal 15 (SIGTERM) after 1203.573297 seconds from start 
[01-Jun-2016 21:54:00] NOTICE: [pool www] child 10116 started 
[01-Jun-2016 22:07:20] WARNING: [pool www] child 9450, script '/var/www/index.php' (request: "GET /index.php") execution timed out (331.919262 sec), terminating 
[01-Jun-2016 22:07:25] WARNING: [pool www] child 9450 exited on signal 15 (SIGTERM) after 2009.277917 seconds from start 
[01-Jun-2016 22:07:25] NOTICE: [pool www] >     >     child 10271 started

顶部

top - 22:46:35 up  1:20,  1 user,  load average: 306.50, 283.23,
290.78 Tasks: 801 total,  89 running, 712 sleeping,   0 stopped,   0 zombie Cpu(s): 85.3%us, 14.2%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi, 
0.5%si,  0.0%st Mem:  32865348k total, 13868888k used, 18996460k free,    28104k buffers Swap:  8388604k total,   727164k used,  7661440k free, 
394328k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 
7962 mysql     20   0 3082m 321m 4072 S 114.1  1.0  82:07.36 mysqld 
9405 php       20   0 1387m 924m 8156 R  4.9  2.9   1:33.70 php-fpm 
9505 php       20   0  714m 252m 8116 R  4.9  0.8   1:29.31 php-fpm 
9448 php       20   0  475m  21m  14m S  4.6  0.1   1:34.66 php-fpm 
9533 php       20   0  472m  19m  12m D  4.6  0.1   1:34.74 php-fpm 
9628 php       20   0  516m  62m  14m R  4.6  0.2   1:40.33 php-fpm 
9685 php       20   0  670m 210m 8156 R  4.6  0.7   1:31.48 php-fpm 
9380 php       20   0  712m 253m 7896 R  4.3  0.8   1:28.64 php-fpm 
9387 php       20   0  507m  48m 7564 R  4.3  0.2   1:31.58 php-fpm 
9395 php       20   0  473m  20m  12m D  4.3  0.1   1:36.94 php-fpm 
9435 php       20   0 1597m 1.1g  12m R  4.3  3.6   1:34.38 php-fpm 
9461 php       20   0  500m  47m  13m R  4.3  0.1   1:36.33 php-fpm 
9493 php       20   0  491m  32m 7560 R  4.3  0.1   1:41.10 php-fpm 
9514 php       20   0  477m  25m  14m R  4.3  0.1   1:32.29 php-fpm 
9554 php       20   0  513m  59m  13m R  4.3  0.2   1:33.10 php-fpm 
9556 php       20   0  512m  57m  12m R  4.3  0.2   1:24.28 php-fpm 
9566 php       20   0  473m  15m 7524 D  4.3  0.0   1:33.42 php-fpm 
9598 php       20   0  531m  71m 7764 R  4.3  0.2   1:32.87 php-fpm 
9620 php       20   0  473m  21m  14m S  4.3  0.1   1:20.62 php-fpm 
9653 php       20   0  509m  54m  11m R  4.3  0.2   1:36.90 php-fpm 
9677 php       20   0  513m  23m 7392 R  4.3  0.1   1:34.66 php-fpm 
9689 php       20   0  714m 258m  14m R  4.3  0.8   1:30.03 php-fpm 
9695 php       20   0 1431m 973m  12m R  4.3  3.0   1:31.20 php-fpm 
9408 php       20   0  472m  13m 7456 S  4.1  0.0   1:35.02 php-fpm 
9463 php       20   0  514m  60m  12m R  4.1  0.2   1:33.01 php-fpm 
9504 php       20   0  505m  48m  21m R  4.1  0.1   1:36.00 php-fpm 
9507 php       20   0  491m  38m  12m R  4.1  0.1   1:37.42 php-fpm 
9516 php       20   0  493m  32m 8108 R  4.1  0.1   1:35.84 php-fpm 
9522 php       20   0  627m 166m 7816 R  4.1  0.5   1:30.21 php-fpm 
9524 php       20   0  936m 480m  13m R  4.1  1.5   1:33.89 php-fpm 
9534 php       20   0  475m  15m 8060 D  4.1  0.0   1:38.95 php-fpm 
9638 php       20   0  504m  45m 7652 S  4.1  0.1   1:35.04 php-fpm 
9641 php       20   0  777m 314m 8508 R  4.1  1.0   1:37.97 php-fpm 
9655 php       20   0  507m  56m  14m R  4.1  0.2   1:28.89 php-fpm 
9671 php       20   0  513m  54m 7544 R  4.1  0.2   1:34.73 php-fpm 
9684 php       20   0  514m  61m  14m R  4.1  0.2   1:36.00 php-fpm 
9793 php       20   0  472m  19m  13m R  4.1  0.1   1:28.19 php-fpm 
9806 php       20   0  515m  55m 8172 R  4.1  0.2   1:35.00 php-fpm 
9818 php       20   0  771m 308m 8068 R  4.1  1.0   1:31.61 php-fpm 
9825 php       20   0  473m  13m 8740 S  4.1  0.0   1:29.41 php-fpm 
9850 php       20   0  475m  18m  11m S  4.1  0.1   1:35.34 php-fpm 
9861 php       20   0  473m  14m 7556 S  4.1  0.0   1:32.32 php-fpm 
9425 php       20   0  475m  21m  13m S  3.8  0.1   1:34.69 php-fpm 
9446 php       20   0  471m  12m 7640 S  3.8  0.0   1:36.17 php-fpm 
9579 php       20   0  471m  12m 7592 D  3.8  0.0   1:37.13 php-fpm 
9588 php       20   0  491m  32m 7468 R  3.8  0.1   1:36.44 php-fpm

[root@davidsm01 ~]# vmstat

procs -----------memory---------- ---swap-- -----io---- --system-------cpu-----  
r  b   swpd   free   buff  cache   si   so    bi    bo   in  cs us sy id wa st 
273  0 669112 13713228  76480 541424   21   30  69    50  567  165 83 15  1  1  0

tcpdump -i eth0

22:55:11.703006 IP 141.101.70.49.36344 >
expressvpn16.clients.netelligent.ca.http: Flags [.], ack 2921, win 77,
length 0 22:55:11.703017 IP 141.101.70.49.36344 >
expressvpn16.clients.netelligent.ca.http: Flags [.], ack 3718, win 80,
length 0 22:55:11.717115 IP 108.162.214.215.23885 >
expressvpn16.clients.netelligent.ca.http: Flags [.], ack 3878216429,
win 29, length 0 22:55:11.717173 IP 108.162.214.215.23885 >
expressvpn16.clients.netelligent.ca.http: Flags [P.], seq 0:485, ack
1, win 29, length 485 22:55:11.717192 IP
expressvpn16.clients.netelligent.ca.http > 108.162.214.215.23885:
Flags [.], ack 485, win 123, length 0

15493 packets captured 
43347 packets received by filter 
27820 packets dropped by kernel

mpstat -P 全部

Linux 2.6.32-642.1.1.el6.x86_64 (davidsm01.localdomain)         06/01/2016      _x86_64_        (12 CPU)

11:14:56 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
11:14:56 PM  all   83.76    0.00   14.16    1.08    0.00    0.56    0.00    0.00    0.43
11:14:56 PM    0   84.53    0.00   13.89    1.18    0.00    0.12    0.00    0.00    0.28
11:14:56 PM    1   83.44    0.00   14.19    1.14    0.00    0.85    0.00    0.00    0.38
11:14:56 PM    2   83.38    0.00   14.23    1.16    0.00    0.85    0.00    0.00    0.38
11:14:56 PM    3   84.59    0.00   13.76    1.15    0.00    0.10    0.00    0.00    0.39
11:14:56 PM    4   84.30    0.00   14.04    1.14    0.00    0.13    0.00    0.00    0.39
11:14:56 PM    5   84.66    0.00   13.73    1.10    0.00    0.11    0.00    0.00    0.40
11:14:56 PM    6   83.24    0.00   14.39    1.16    0.00    0.84    0.00    0.00    0.37
11:14:56 PM    7   84.44    0.00   13.94    0.55    0.00    0.15    0.00    0.00    0.92
11:14:56 PM    8   83.07    0.00   14.54    1.17    0.00    0.87    0.00    0.00    0.35
11:14:56 PM    9   83.07    0.00   14.43    1.09    0.00    0.97    0.00    0.00    0.45
11:14:56 PM   10   83.11    0.00   14.50    1.05    0.00    0.84    0.00    0.00    0.50
11:14:56 PM   11   83.31    0.00   14.33    1.13    0.00    0.84    0.00    0.00    0.39

ps -eo pcpu,pid,用户,参数|sort -k 1 -r|head -10

%CPU   PID USER     COMMAND
 2.3 10772 php      php-fpm: pool www
 2.3 10490 php      php-fpm: pool www
 2.2  9869 php      php-fpm: pool www
 2.2  9868 php      php-fpm: pool www
 2.2  9864 php      php-fpm: pool www
 2.2  9854 php      php-fpm: pool www
 2.2  9850 php      php-fpm: pool www
 2.2  9827 php      php-fpm: pool www
 2.2  9824 php      php-fpm: pool www

SS-S

 Total: 14878 (kernel 15172)
 TCP:   30203 (estab 11194, closed 16290, orphaned 89, synrecv 0, timewait >      16281/0), ports 4092

 Transport Total     IP        IPv6
 *         15172     -         -
 RAW       0         0         0
 UDP       0         0         0
 TCP       13913     13908     5
 INET      13913     13908     5
 FRAG      0         0         0

设备 eth0 [] (1/5):

 Incoming:

 Curr: 1.61 MBit/s
 Avg: 1.78 MBit/s
 Min: 886.95 kBit/s
 Max: 5.11 MBit/s
 Ttl: 2.06 GByte

 Outgoing

 Curr: 4.82 MBit/s
 Avg: 5.08 MBit/s
 Min: 2.10 MBit/s
 Max: 14.50 MBit/s
 Ttl: 5.59 GByte

答案1

您的服务器正在遭受损失,因为数据库无法足够快地向您的 Web 应用程序提供数据。

出现这种情况的原因可能有多种:

1)数据库中没有适当的索引,这意味着 MySQL 必须进行全表扫描来检索数据,这需要很长时间。

2) Web 应用程序中的缓存未正确实现。

3) 您的网站正遭受 DDoS 攻击,根据上述事实,攻击可能会成功。

从您的 Web 服务器日志注释来看,原因可能是选项 3。您应该使用一些分析软件分析日志,看看它与“正常”日子有何不同,从而推断是否存在一些不必要的流量以及它的来源。

你可以阅读我遭受了 DDoS 攻击。我该怎么办?了解您有哪些缓解 DDoS 攻击的选项。

答案2

这里有很多错误。而且您的分析/问题中有很多缺失。

您在此处发布的内容中最相关的部分是 top 的输出。只有一个快照很难准确判断发生了什么,但系统无法满足需求。也就是说,这是一个容量问题。

您没有提到明显的提高性能的候选方案,这表明通过提高系统性能来提高系统容量的空间很大。

根据单个 top 快照(这意味着很多猜测),mysql 仅使用了 10%(112/12)的可用 CPU,这意味着 php-fpm 正在使用其余部分。这意味着是您的 PHP 代码和抢先式上下文切换正在降低您的性能。

但是您的断言“启用 Nginx 缓存会使加载时间重新回到 100 秒”(我认为这意味着将 NGinx 缓存从 RAM 移动到文件系统会导致页面加载时间增加到 100 秒)表明问题出在 Nginx 缓存或存储 I/O 上。您的 nginx 磁盘缓存是否与数据库位于同一块设备上?

除非你赠送比特币,否则你的用户不太可能真正忍受这样的加载时间并且仍然托管你的网站。这表明您正遭受 DOS 攻击 - 从一些简单的日志分析中应该很容易看出这一点。但是 DOS 攻击很少会持续超过几个小时。如果您遭受攻击超过一年,那么您一定做了一些真正惹恼 NSA/CIA/FSB 的事情。

您的软件堆栈、Linux、nginx、PHP 和 MySQL 无法很好地利用大型计算机 - 而且这种架构只有在主机托管设施中运行多台服务器时才具有成本效益。

相关内容