☰

编辑：

tag-icon • tag-icon ubuntu nginx postgresql django gunicorn
编辑：

我正在尝试使用基准测试工具实现 2000 个并发用户。我使用 locust 来模拟它们。

我的服务器有 24vCPU、128GB RAM、25SSD。

我希望能够无错误地为 2000 个并发用户提供服务，但是当用户数量达到 700 个之后我就遇到了麻烦。

古尼康

我安装了 gevent 以便能够提供异步服务，但这并没有改变我的负载测试中的任何内容（gevent 无法工作吗？）。我的 systemd 文件如下：

mysite-生产环境.conf

    [Unit]
    Description=mysite production daemon
    After=network.target

    [Service]
    User=www-data
    Group=www-data
    WorkingDirectory=/var/www/mysite/production/src
    ExecStart=/var/www/mysite/production/venv/bin/gunicorn --worker-class=gevent --worker-connections=1000  --workers=49 --bind unix:/var/www/mysite/production/sock/gunicorn --log-level DEBUG --log-file '/var/www/mysite/production/log/gunicorn.log' mysite.wsgi:application
    ExecReload=/bin/kill -s HUP $MAINPID
    ExecStop=/bin/kill -s TERM $MAINPID

    [Install]
    WantedBy=multi-user.target

根据我的计算：每秒我可以处理 49 x 1000 = 49000 个请求。

相反，当用户数量达到约 700 人时，我的 Locust 故障选项卡中出现了以下错误：

# fails Method  Name    Type
1227    GET     //  HTTPError('500 Server Error: Internal Server Error for url: http://my.site.com//')

它基本上是说我有一个内部服务器错误。

当我打开我的 gunicorn.log 时我看到忽略 EPIPE：

gunicorn日志

[2020-01-27 20:22:30 +0000] [13121] [DEBUG] Ignoring EPIPE
[2020-01-27 20:22:31 +0000] [13121] [DEBUG] Ignoring EPIPE
[2020-01-27 20:22:31 +0000] [13121] [DEBUG] Ignoring EPIPE
[2020-01-27 20:22:31 +0000] [13121] [DEBUG] Ignoring EPIPE
[2020-01-27 20:22:31 +0000] [13121] [DEBUG] Ignoring EPIPE

Nginx

我的 nginx access.log 显示一些 499 和 500 错误：

访问日志

185.159.126.246 - - [27/Jan/2020:19:22:25 +0000] "GET // HTTP/1.1" 200 30727 "-" "python-requests/2.22.0"
185.159.126.246 - - [27/Jan/2020:19:22:25 +0000] "GET // HTTP/1.1" 499 0 "-" "python-requests/2.22.0"
185.159.126.246 - - [27/Jan/2020:19:22:25 +0000] "GET // HTTP/1.1" 499 0 "-" "python-requests/2.22.0"
185.159.126.246 - - [27/Jan/2020:19:22:25 +0000] "GET // HTTP/1.1" 200 30727 "-" "python-requests/2.22.0"
185.159.126.246 - - [27/Jan/2020:19:22:25 +0000] "GET // HTTP/1.1" 500 2453 "-" "python-requests/2.22.0"
185.159.126.246 - - [27/Jan/2020:19:22:25 +0000] "GET // HTTP/1.1" 500 2453 "-" "python-requests/2.22.0"
185.159.126.246 - - [27/Jan/2020:19:22:25 +0000] "GET // HTTP/1.1" 499 0 "-" "python-requests/2.22.0"
185.159.126.246 - - [27/Jan/2020:19:22:25 +0000] "GET // HTTP/1.1" 499 0 "-" "python-requests/2.22.0"

nginx error.log 没有显示任何内容，但在之前的测试中它显示了资源暂时不可用：

错误日志


2020/01/27 19:15:32 [error] 1514#1514: *57151 connect() to unix:/var/www/mysite/production/sock/gunicorn failed (11: Resource temporarily unavailable) while connecting to upstream, client: 185.159.126.246, server: my.site.com, request: "GET // HTTP/1.1", upstream: "http://unix:/var/www/mysite/production/sock/gunicorn://", host: "my.site.com"
2020/01/27 19:15:32 [error] 1514#1514: *56133 connect() to unix:/var/www/mysite/production/sock/gunicorn failed (11: Resource temporarily unavailable) while connecting to upstream, client: 185.159.126.246, server: my.site.com, request: "GET // HTTP/1.1", upstream: "http://unix:/var/www/mysite/production/sock/gunicorn://", host: "my.site.com"

这是我的 nginx.conf：

nginx.conf

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
worker_rlimit_nofile 40000;

events {
        worker_connections 4096;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
   }

这是我的服务器块：

my.site.com (服务器块)


upstream mysite-production {
    server unix:/var/www/mysite/production/sock/gunicorn;
}
server {
    listen [::]:80;
    listen 80;
    server_name my.site.com;

    # set client body size to 100M #
    client_max_body_size 100M;

    location / {
      include proxy_params;
      proxy_pass http://unix:/var/www/mysite/production/sock/gunicorn;
    }

    location /static/ {
        root /var/www/mysite/production/;
        expires 30d;
        add_header Vary Accept-Encoding;
        access_log off;
        gzip on;
        gzip_comp_level 6;
        gzip_vary on;
        gzip_types text/plain text/css application/json application/x-javascript application/javascript text/xml application/xml application/rss+xml text/javascript image/svg+xml application/vnd.ms-fontobject application/x-font-ttf font/opentype;
    }

    location /media/ {
        root /var/www/mysite/production/;
        expires 30d;
        add_header Vary Accept-Encoding;
        access_log off;
    }
}

问题

当我看到这一切时，我想到的是：

502 错误从何而来？它没有显示足够的信息来让我了解发生了什么... 是不是 700 个用户之后我的所有工作进程都处于忙碌状态，导致了错误？

499错误从何而来？

这与我的 postgresql 数据库有关系吗？如果有，我该如何查找？

我怎么知道 gevent 是否起作用了？使用 ab 和 locust 进行的负载测试的结果并没有真正改变。

什么忽略 EPIPE这是什么意思？为什么会这样？

感谢您花时间阅读此内容。提前感谢您的所有评论和回答！

如果您需要更多信息，请告诉我，我会将其添加到我的问题中。

编辑：

根据@ServerFaults的评论，我检查了负载测试发生时消耗了多少内存。显然没有什么特别重要的：

MemTotal:       132027088 kB
MemFree:        127013484 kB
MemAvailable:   126458072 kB
Buffers:           57788 kB
Cached:           401088 kB

运行后netstat -s（我没有足够的知识去理解这一切意味着什么）：

Ip:
    Forwarding: 2
    6956023 total packets received
    8 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    6953503 incoming packets delivered
    6722961 requests sent out
    20 outgoing packets dropped
Icmp:
    107 ICMP messages received
    0 input ICMP message failed
    ICMP input histogram:
        destination unreachable: 45
        echo requests: 62
    968 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 906
        echo replies: 62
IcmpMsg:
        InType3: 45
        InType8: 62
        OutType0: 62
        OutType3: 906
Tcp:
    222160 active connection openings
    136492 passive connection openings
    78293 failed connection attempts
    161239 connection resets received
    2 connections established
    6245053 segments received
    6487229 segments sent out
    11978 segments retransmitted
    0 bad segments received
    185530 resets sent
Udp:
    667101 packets received
    41158 packets to unknown port received
    0 packet receive errors
    727164 packets sent
    0 receive buffer errors
    0 send buffer errors
UdpLite:
TcpExt:
    2939 SYN cookies sent
    28 SYN cookies received
    215 invalid SYN cookies received
    113 resets received for embryonic SYN_RECV sockets
    18751 TCP sockets finished time wait in fast timer
    175 packetes rejected in established connections because of timestamp
    61997 delayed acks sent
    54 delayed acks further delayed because of locked socket
    Quick ack mode was activated 3475 times
    2687504 packet headers predicted
    1171485 acknowledgments not containing data payload received
    2270054 predicted acknowledgments
    TCPSackRecovery: 921
    Detected reordering 22500 times using SACK
    Detected reordering 596 times using time stamp
    139 congestion windows fully recovered without slow start
    490 congestion windows partially recovered using Hoe heuristic
    TCPDSACKUndo: 14
    413 congestion windows recovered without slow start after partial ack
    TCPLostRetransmit: 47
    TCPSackFailures: 5
    7 timeouts in loss state
    2416 fast retransmits
    123 retransmits in slow start
    TCPTimeouts: 5279
    TCPLossProbes: 4284
    TCPLossProbeRecovery: 22
    TCPSackRecoveryFail: 1
    TCPDSACKOldSent: 3482
    TCPDSACKRecv: 1996
    TCPDSACKOfoRecv: 4
    52968 connections reset due to unexpected data
    54079 connections reset due to early user close
    51 connections aborted due to timeout
    TCPDSACKIgnoredOld: 25
    TCPDSACKIgnoredNoUndo: 822
    TCPSpuriousRTOs: 81
    TCPSackShifted: 4787
    TCPSackMerged: 5128
    TCPSackShiftFallback: 33199
    TCPReqQFullDoCookies: 2939
    TCPRcvCoalesce: 84434
    TCPOFOQueue: 26
    TCPChallengeACK: 3
    TCPAutoCorking: 7
    TCPFromZeroWindowAdv: 125
    TCPToZeroWindowAdv: 125
    TCPWantZeroWindowAdv: 1376
    TCPSynRetrans: 4695
    TCPOrigDataSent: 5023492
    TCPHystartTrainDetect: 2403
    TCPHystartTrainCwnd: 53596
    TCPHystartDelayDetect: 346
    TCPHystartDelayCwnd: 9883
    TCPACKSkippedSynRecv: 6
    TCPACKSkippedPAWS: 10
    TCPACKSkippedSeq: 15
    TCPWinProbe: 15
IpExt:
    InOctets: 6090026491
    OutOctets: 7098177391
    InNoECTPkts: 6966279
    InECT0Pkts: 11
    InCEPkts: 1

编辑2：

在创建静态 html 页面而不生成 Gunicorn 之后，它能够轻松地为 4000 名用户提供服务而不会出现任何延迟。传递 worker_connections 后，我收到了：

/loadtest/  ConnectionError(MaxRetryError("HTTPConnectionPool(host='ll-my.site.com', port=80): Max retries exceeded with url: /loadtest/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x....>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))"))

据我了解，NGINX 在这里不是问题。

编辑3：

我检查了我的 postgresql 日志，发现它充满了这个错误：

FATAL:  remaining connection slots are reserved for non-replication superuser connections

相关内容