![瓶颈在哪里——nginx 文件服务速度慢](https://linux22.com/image/654179/%E7%93%B6%E9%A2%88%E5%9C%A8%E5%93%AA%E9%87%8C%E2%80%94%E2%80%94nginx%20%E6%96%87%E4%BB%B6%E6%9C%8D%E5%8A%A1%E9%80%9F%E5%BA%A6%E6%85%A2.png)
我有nginx 1.6.2
,在 Ubuntu 14+ 上。
我托管了几个“大”文件 - 从 20mb 到 60mb - 这些文件经常被访问(在晚高峰时,同时访问的连接可能超过 1000 个)。这些文件很少更改,一两周一次。
服务器还托管大量用户上传的图像,这些图像也会不断被访问,但可以根据用户的意愿进行更改。图像通过 php 上传。
所以,问题是 - 这些大文件的下载速度真的很慢。有时甚至会断线。以下是一些日志和配置:
Nginx 配置
user www-data;
worker_processes auto;
pid /run/nginx.pid;
worker_rlimit_nofile 65000;
events {
worker_connections 4096;
multi_accept on;
use epoll; }
http {
client_body_buffer_size 50m;
sendfile off;
output_buffers 2 64k;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 120s;
types_hash_max_size 2048;
directio 10m;
include /etc/nginx/mime.types;
default_type application/octet-stream;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
access_log off; #/var/log/nginx/access.log;
error_log /var/log/nginx/error.log crit;
gzip on;
gzip_disable "msie6";
gzip_static on;
gzip_comp_level 5;
gzip_min_length 1024;
gzip_proxied any;
gzip_types text/plain application/xml application/x-javascript text/javascript text/css text$
open_file_cache max=1000 inactive=5m;
open_file_cache_valid 2m;
open_file_cache_min_uses 1;
open_file_cache_errors on;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
这是站点配置
server {
listen 80;
server_name _;
root /var/www/;
location / {
try_files $uri $uri/ =404;
}
location ~* \.php$ {
try_files $uri =404;
fastcgi_pass unix:/var/run/php5-fpm.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
location ~* .(unity3d|html)$ { #yep, that's the ones 20mb-50mb file types
expires 7d;
}
location /uploads/customimages {
location ~\.php$ {return 403;}
}
location /uploads/temp {
location ~\.php$ {return 403;}
}
location ~\.php$ {
try_files $uri = 404;
fastcgi_pass unix:/var/run/php5-fpm.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
}
与 https 协议几乎相同。
我检查的内容:处理器,内存,ssd i/o 和网络。
这是其中的一部分dstat
,几乎完全一样。
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
3 0 96 0 0 0|2066k 728k| 0 0 | 0 0 |4251 11k
3 0 96 1 0 0|6080k 0 | 219k 6722k| 0 0 |5633 6856
2 0 96 1 0 0|8128k 0 | 255k 8089k| 0 0 |5988 6895
3 0 96 1 0 0|5888k 72k| 232k 6811k| 0 0 |5818 7074
2 0 97 1 0 0|6592k 48k| 231k 7228k| 0 0 |5745 7721
3 1 95 1 0 0|7104k 0 | 243k 7903k| 0 0 |6024 7439
3 1 96 1 0 0|6272k 1440k| 220k 6759k| 0 0 |6232 9463
4 1 94 1 0 0|6784k 0 | 216k 6984k| 0 0 |5893 9412
4 1 94 1 0 0|7040k 0 | 718k 7723k| 0 0 |6551 13k
5 1 94 1 0 1|8000k 108k| 478k 8926k| 0 0 |6647 13k
4 1 94 1 0 0| 10M 192k| 402k 10M| 0 0 |6665 14k
4 1 95 1 0 0| 11M 2772k|1080k 12M| 0 0 |7188 12k
4 1 94 1 0 0|9536k 0 |1012k 10M| 0 0 |6758 14k
5 1 93 1 0 0|9472k 504k|1734k 10M| 0 0 |7298 13k
3 1 95 1 0 0|9984k 0 |1027k 10M| 0 0 |6776 11k
3 0 95 1 0 0| 10M 0 | 355k 11M| 0 0 |6550 8152
3 1 95 1 0 0| 11M 2784k| 498k 14M| 0 0 |7403 10k
8 2 89 1 0 0|6592k 228k| 311k 7482k| 0 0 |7394 9242
3 0 96 1 0 0|5760k 0 | 211k 6076k| 0 0 |5807 8031
5 1 94 1 0 0|6720k 180k| 201k 6378k| 0 0 |5949 9432
3 0 96 1 0 0|6400k 0 | 234k 7113k| 0 0 |5866 11k
4 1 95 1 0 0|5696k 672k| 231k 6988k| 0 0 |6128 10k
5 1 94 1 0 0|6336k 108k| 189k 5477k| 0 0 |6002 12k
8 1 90 1 0 0|5120k 0 | 181k 5832k| 0 0 |6445 18k
4 1 94 1 0 0|6272k 84k| 212k 7521k| 0 0 |6656 21k
6 1 92 1 0 0|7488k 0 | 237k 8334k| 0 0 |7109 23k
5 1 94 1 0 0|8960k 3264k| 278k 9362k| 0 0 |7360 22k
4 1 94 1 0 0|8896k 12k| 326k 9736k| 0 0 |7190 15k
3 0 96 1 0 0|4864k 168k| 191k 5179k| 0 0 |6123 13k
内存检查通过以下方式进行free
:
total used free shared buffers cached
Mem: 32840320 9392976 23447344 6652 309472 4688688
-/+ buffers/cache: 4394816 28445504
Swap: 1569780 0 1569780
网络lshw -C network
*-network
description: Ethernet interface
product: 82579V Gigabit Network Connection
vendor: Intel Corporation
physical id: 19
bus info: pci@0000:00:19.0
logical name: eth0
version: 05
serial: e0:69:95:72:8e:25
size: 1Gbit/s
capacity: 1Gbit/s
width: 32 bits
clock: 33MHz
capabilities: pm msi bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=e1000e driverversion=2.3.2-k duplex=full firmware=0.13-4 ip=94.23.42.228 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
resources: irq:43 memory:fe500000-fe51ffff memory:fe524000-fe524fff ioport:f080(size=32)
*-network:0 DISABLED
description: Ethernet interface
physical id: 3
logical name: dummy0
serial: 56:14:e3:9e:36:59
capabilities: ethernet physical
configuration: broadcast=yes
*-network:1 DISABLED
description: Ethernet interface
physical id: 4
logical name: bond0
serial: ce:5f:11:d3:5b:01
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=bonding driverversion=3.7.1 firmware=2 link=no master=yes multicast=yes
我很确定硬件没有问题,问题出在 Ubuntu 或 nginx 上。
ulimit -n
65536
抱歉,格式不正确。
答案1
这肯定和directio有关。
您可能不想使用它,因为它绕过了缓存机制(特别是如果您的大文件不会发生很大变化)并且大多是同步的。
从man 2 open
:
O_DIRECT (since Linux 2.4.10)
Try to minimize cache effects of the I/O to and from this
file. In general this will degrade performance, but it is
useful in special situations, such as when applications do
their own caching. File I/O is done directly to/from user-
space buffers. The O_DIRECT flag on its own makes an effort
to transfer data synchronously, but does not give the
guarantees of the O_SYNC flag that data and necessary metadata
are transferred. To guarantee synchronous I/O, O_SYNC must be
used in addition to O_DIRECT.
A semantically similar (but deprecated) interface for block
devices is described in raw(8).
如果您认为同时服务的大文件太多,而且由于缓存效率低下而需要 directio,则请检查 iowait,如果是,则应启用 aio。
PS:aio 和 sendfile 可以一起使用:sendfile 用于较小的文件,aio 用于较大的文件。