提高千兆网络上大量连接和高流量的存储和流媒体服务的 TCP 性能

提高千兆网络上大量连接和高流量的存储和流媒体服务的 TCP 性能

我有两台服务器,两台服务器的硬件规格都是

Processor : Dual Processor
RAM : over 128 G.B
Hard disk : SSD Hard disk
Outging Traffic bandwidth : 3 Gbps
network cards speed : 10 Gbps

服务器 A : 用于编码视频

服务器 B :用于存储视频并通过 Web 界面播放视频就像 YouTube

两台服务器之间的入站带宽为10Gbps,出站带宽互联网带宽是500Mpbs

两台服务器均在公网和私网中使用公有 IP 地址

两台服务器都在 nginx 端口上传输和连接,服务器 B 用于流媒体,如 youtube 流视频

两台服务器在同一个网络中,当我从服务器 A ping 到服务器 B 时,我得到了高于 1.0 毫秒的较高时间延迟,时间范围为时间=52.7 毫秒至时间=215.7 毫秒 -

这是 iftop 实用程序的输出

                            353Mb                       707Mb                       1.04Gb                      1.38Gb                1.73Gb
mqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqq
server.example.com                                        => ip.address                                              6.36Mb  4.31Mb  1.66Mb
                                                         <=                                                           158Kb  94.8Kb  35.1Kb
server.example.com                                        => ip.address                                             1.23Mb  4.28Mb  1.12Mb
                                                         <=                                                          17.1Kb  83.5Kb  21.9Kb
server.example.com                                        => ip.address                                            395Kb  3.89Mb  1.07Mb
                                                         <=                                                          6.09Kb   109Kb  28.6Kb
server.example.com                                        => ip.address                                           4.55Mb  3.83Mb  1.04Mb
                                                         <=                                                          55.6Kb  45.4Kb  13.0Kb
server.example.com                                        => ip.address                                             649Kb  3.38Mb  1.47Mb
                                                         <=                                                          9.00Kb  38.7Kb  16.7Kb
server.example.com                                        => ip.address                                             5.00Mb  3.32Mb  1.80Mb
                                                         <=                                                          65.7Kb  55.1Kb  29.4Kb
server.example.com                                        => ip.address                                              387Kb  3.13Mb  1.06Mb
                                                         <=                                                          18.4Kb  39.9Kb  15.0Kb
server.example.com                                        => ip.address                                              3.27Mb  3.11Mb  1.01Mb
                                                         <=                                                          81.2Kb  64.5Kb  20.9Kb
server.example.com                                        => ip.address                                             1.75Mb  3.08Mb  2.72Mb
                                                         <=                                                          16.6Kb  35.6Kb  32.5Kb
server.example.com                                        => ip.address                                               1.75Mb  2.90Mb  2.79Mb
                                                         <=                                                          22.4Kb  32.6Kb  35.6Kb
server.example.com                                        => ip.address                                             3.03Mb  2.78Mb  1.82Mb
                                                         <=                                                          26.6Kb  27.4Kb  20.2Kb
server.example.com                                        => ip.address                                              2.26Mb  2.66Mb  1.36Mb
                                                         <=                                                          51.7Kb  49.1Kb  24.4Kb
server.example.com                                        => ip.address                                              586Kb  2.50Mb  1.03Mb
                                                         <=                                                          4.17Kb  26.1Kb  10.7Kb
server.example.com                                        => ip.address                                               2.42Mb  2.49Mb  2.44Mb
                                                         <=                                                          31.6Kb  29.7Kb  29.9Kb
server.example.com                                        => ip.address                                            2.41Mb  2.46Mb  2.41Mb
                                                         <=                                                          26.4Kb  24.5Kb  23.8Kb
server.example.com                                        => ip.address                                             2.37Mb  2.39Mb  2.40Mb
                                                         <=                                                          28.9Kb  27.0Kb  28.5Kb
server.example.com                                        => ip.address                                             525Kb  2.20Mb  1.05Mb
                                                         <=                                                          7.03Kb  26.0Kb  12.8Kb
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
TX:             cum:    102GB   peak:   1.65Gb                                                              rates:   1.46Gb  1.44Gb  1.48Gb
RX:                    1.31GB           24.3Mb                                                                       19.5Mb  18.9Mb  20.0Mb
TOTAL:                  103GB           1.67Gb                                                                       1.48Gb  1.46Gb  1.50Gb

我使用 iperf 实用程序检查传输速度服务器 A服务器 B

# iperf -c 0.0.0.2 -p 8777
------------------------------------------------------------
Client connecting to 0.0.0.2, TCP port 8777
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  3] local 0.0.0.1 port 38895 connected with 0.0.0.2 port 8777
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.8 sec    528 KBytes    399 Kbits/sec

我当前在服务器 B 中的连接

# netstat -an|grep ":8777"|awk '/tcp/ {print $6}'|sort -nr| uniq -c
   2072 TIME_WAIT
     28 SYN_RECV
      1 LISTEN
    189 LAST_ACK
    139 FIN_WAIT2
    373 FIN_WAIT1
   3381 ESTABLISHED
     34 CLOSING

服务器 A网卡信息

Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 10000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

服务器 B网卡信息

Settings for eth2:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: No
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

是否配置服务器A

eth0      Link encap:Ethernet  HWaddr 00:25:90:ED:9E:AA
          inet addr:0.0.0.1  Bcast:0.0.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1202795665 errors:0 dropped:64334 overruns:0 frame:0
          TX packets:2313161968 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:893413096188 (832.0 GiB)  TX bytes:3360949570454 (3.0 TiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2207544 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2207544 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:247769175 (236.2 MiB)  TX bytes:247769175 (236.2 MiB)

是否配置服务器 B

eth2      Link encap:Ethernet  HWaddr 00:25:90:82:C4:FE
          inet addr:0.0.0.2  Bcast:0.0.0.2  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:39973046980 errors:0 dropped:1828387600 overruns:0 frame:0
          TX packets:69618752480 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3013976063688 (2.7 TiB)  TX bytes:102250230803933 (92.9 TiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:1049495 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1049495 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:129012422 (123.0 MiB)  TX bytes:129012422 (123.0 MiB)

Netstat -i 开启服务器 B

# netstat -i
Kernel Interface table
Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth2       9000   0 42098629968      0 2131223717      0 73698797854      0      0      0 BMRU
lo        65536   0  1077908      0      0      0  1077908      0      0      0 LRU

我将网卡上的发送/接收缓冲区调至 2048,问题仍然存在

我增加了服务器 A 的 MTU,问题仍然存在,我增加了服务器 B 的 MTU,以获得更好的连接和传输速度,但根本无法传输

问题是:从 iperf 实用程序中可以看到,当我重新启动服务器 B 中的网络服务时,从服务器 A 到服务器 B 的传输速度很慢,服务器 A 中的传输以全速进行,2 分钟后,速度变慢

我该如何排除服务器 B 速度慢的问题并修复它?

注意:如果我需要在服务器中执行其他命令以获取更多信息,以便可能有助于解决问题,请在评论中告诉我

相关内容