nfs 服务器导出到 300-400 个客户端。假设客户端 A 和 B 正在同时从 nfs 服务器复制同一个文件。客户端 A 的速度为 100mb/s，客户端 B 的速度为 1-2mb/s。然后一天后，速度问题会逆转。客户端 A 下降到 1-2mb/s。我确信客户端带宽没有饱和。当它以 1-2/mbs 的速度运行时，我甚至很难做一个简单的ls或者光盘在传输速率缓慢期间，dmesg 或 /var/log/messages 中没有任何内容。
服务器上 eth0 RX 数据包丢失率持续增加。
NFS 服务器：
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet x.x.x.x netmask 255.255.255.0 broadcast x.x.x.x
inet6 fe80::225:b5ff:feff:b000 prefixlen 64 scopeid 0x20<link>
ether 00:25:b5:ff:b0:00 txqueuelen 10000 (Ethernet)
RX packets 943628558144 bytes 616933252471494 (561.0 TiB)
RX errors 0 dropped 760517201 overruns 0 frame 0
TX packets 294979961515 bytes 920493526252565 (837.1 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ethtool -S eth0
NIC statistics:
tx_frames_ok: 295630824296
tx_unicast_frames_ok: 295630824219
tx_multicast_frames_ok: 10
tx_broadcast_frames_ok: 67
tx_bytes_ok: 921153721030047
tx_unicast_bytes_ok: 921153721024891
tx_multicast_bytes_ok: 868
tx_broadcast_bytes_ok: 4288
tx_drops: 0
tx_errors: 0
tx_tso: 147060374896
rx_frames_ok: 945863920948
rx_frames_total: 946625249253
rx_unicast_frames_ok: 946623896815
rx_multicast_frames_ok: 11
rx_broadcast_frames_ok: 1352427
rx_bytes_ok: 617220927332403
rx_unicast_bytes_ok: 617453019308584
rx_multicast_bytes_ok: 1006
rx_broadcast_bytes_ok: 86648192
rx_drop: 0
rx_no_bufs: 761328305
rx_errors: 0
rx_rss: 0
rx_crc_errors: 0
rx_frames_64: 7771228440
rx_frames_127: 609137402623
rx_frames_255: 153476479280
rx_frames_511: 56641135859
rx_frames_1023: 64839335794
rx_frames_1518: 490685265
rx_frames_to_max: 54268981993
nfsstat -o all
Server packet stats:
packets udp tcp tcpconn
2740643409 1180 2740494779 1113812
Server rpc stats:
calls badcalls badclnt badauth xdrcall
2740451952 1701 1141 560 0
Server reply cache:
hits misses nocache
185594 1806160969 934275416
Server file handle cache:
lookup anon ncachedir ncachedir stale
0 0 0 0 321859
Server nfs v3:
null getattr setattr lookup access readlink
8802 0% 18074261 0% 787773 0% 1255878 0% 10635785 0% 10426 0%
read write create mkdir symlink mknod
866194719 31% 1802944793 66% 189292 0% 60196 0% 2035 0% 0 0%
remove rmdir rename link readdir readdirplus
201351 0% 73802 0% 5602 0% 7 0% 238 0% 2409689 0%
fsstat fsinfo pathconf commit
16534863 0% 2446 0% 1221 0% 1928896 0%
Server nfs v4:
null compound
25 0% 18886392 99%
Server nfs v4 operations:
op0-unused op1-unused op2-future access close commit
0 0% 0 0% 0 0% 108199 0% 32860 0% 15836 0%
create delegpurge delegreturn getattr getfh link
1063 0% 0 0% 0 0% 17273601 43% 51634 0% 0 0%
lock lockt locku lookup lookup_root nverify
26 0% 0 0% 26 0% 44202 0% 0 0% 0 0%
open openattr open_conf open_dgrd putfh putpubfh
38101 0% 0 0% 30 0% 27 0% 18572115 47% 0 0%
putrootfh read readdir readlink remove rename
23 0% 1018545 2% 48615 0% 15 0% 140807 0% 21397 0%
renew restorefh savefh secinfo setattr setcltid
334450 0% 0 0% 21399 0% 11 0% 43468 0% 31 0%
setcltidconf verify write rellockowner bc_ctl bind_conn
31 0% 0 0% 1739003 4% 25 0% 0 0% 0 0%
exchange_id create_ses destroy_ses free_stateid getdirdeleg getdevinfo
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
getdevlist layoutcommit layoutget layoutreturn secinfononam sequence
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
set_ssv test_stateid want_deleg destroy_clid reclaim_comp
0 0% 0 0% 0 0% 0 0% 0 0%
NFS 客户端
nfsstat
Client rpc stats:
calls retrans authrefrsh
1187938914 59869 1188488980