我们的生产服务器有 4 张网卡,以 2 个绑定的方式 2 2 地链接在一起。
外部网络:bond0:eth0 已启动并正在运行,eth1 处于活动备份状态 内部网络:bond1:eth2 已启动并正在运行,eth3 处于活动备份状态
我们同时发现 eth0 和 eth2 出现故障:
Mar 3 10:38:16 localhost kernel: [93739227.917537] tg3 0000:02:00.0 eth0: 0x000068b0: 0xe0011514, 0x00000000, 0x00000000, 0x00000000
Mar 3 10:38:16 localhost kernel: [93739227.930035] tg3 0000:02:00.0 eth0: 0x000068e0: 0x00000000, 0x00000000, 0x00000000, 0x0001c2cc
Mar 3 10:38:16 localhost kernel: [93739227.942529] tg3 0000:02:00.0 eth0: 0x000068f0: 0x00ff000e, 0x00ff0000, 0x00000000, 0x04444444
...
Mar 3 10:38:17 localhost kernel: [93739228.141585] tg3 0000:02:00.0 eth0: 4: NAPI info [0000000a:0000000a:(0000:0000:01ff):04dc:(04dc:04dc:0000:0000)]
Mar 3 10:38:17 localhost kernel: [93739228.201559] bonding: bond0: link status definitely down for interface eth0, disabling it
Mar 3 10:38:17 localhost kernel: [93739228.216343] tg3 0000:02:00.0 eth0: Link is down
Mar 3 10:38:18 localhost kernel: [93739229.253266] bonding: bond0: now running without any active interface !
Mar 3 10:38:18 localhost kernel: [93739229.253331] tg3 0000:08:00.0 eth2: transmit timed out, resetting
Mar 3 10:38:19 localhost kernel: [93739230.509553] tg3 0000:08:00.0 eth2: 0x00000000: 0x165f14e4, 0x00100406, 0x02000000, 0x00800010
Mar 3 10:38:19 localhost kernel: [93739230.521603] tg3 0000:08:00.0 eth2: 0x00000010: 0xd90a000c, 0x00000000, 0xd90b000c, 0x00000000
Mar 3 10:38:19 localhost kernel: [93739230.533658] tg3 0000:08:00.0 eth2: 0x00000020: 0xd90c000c, 0x00000000, 0x00000000, 0x200314e4
Mar 3 10:38:19 localhost kernel: [93739230.545704] tg3 0000:08:00.0 eth2: 0x00000030: 0xdd000000, 0x00000048, 0x00000000, 0x0000010f
Mar 3 10:38:19 localhost kernel: [93739230.557755] tg3 0000:08:00.0 eth2: 0x00000040: 0x00000000, 0xa5000000, 0xc8035001, 0x64002008
Mar 3 10:38:19 localhost kernel: [93739230.569808] tg3 0000:08:00.0 eth2: 0x00000050: 0x818c5803, 0x78000000, 0x0086a005, 0x00000000
...
Mar 3 10:38:23 localhost kernel: [93739234.611688] tg3 0000:08:00.0 eth2: 4: Host status block [00000001:000000df:(0000:0000:0a0f):(0000:0000)]
Mar 3 10:38:23 localhost kernel: [93739234.624030] tg3 0000:08:00.0 eth2: 4: NAPI info [000000c4:000000c4:(0000:0000:01ff):09d4:(01d4:01d4:0000:0000)]
Mar 3 10:38:23 localhost kernel: [93739234.699205] bonding: bond1: link status definitely down for interface eth2, disabling it
Mar 3 10:38:23 localhost kernel: [93739234.738410] tg3 0000:08:00.0: tg3_stop_block timed out, ofs=1400 enable_bit=2
Mar 3 10:38:23 localhost kernel: [93739234.850735] tg3 0000:08:00.0: tg3_stop_block timed out, ofs=c00 enable_bit=2
Mar 3 10:38:23 localhost kernel: [93739234.977285] tg3 0000:08:00.0 eth2: Link is down
Mar 3 10:38:25 localhost kernel: [93739236.081087] bonding: bond1: now running without any active interface !
1) 由于它同时发生在两个不同的网络上,我们怀疑存在硬件问题(主板或电源中的微切口,即电源装置故障)请随时告诉我您是否同意我的诊断;)
2) 配置为 active-backup 的绑定在发生故障时保留热备份网卡。正如您在此处看到的,它似乎没有运行备份,甚至没有考虑任何事情。我ifconfig
在事件发生时检查了,eth1 和 eth3(备份)已正确连接到各自的绑定。
债券无法切换到热备份卡的问题可能是什么?
编辑:完整的网络配置:
bond0 Link encap:Ethernet HWaddr 90:b1:1c:xxxxx
inet addr:195.178.186.222 Bcast:195.178.xxxxxxx Mask:255.255.255.224
inet6 addr: fe80::92xxxxa:4b1e/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:11806289 errors:0 dropped:563346 overruns:0 frame:0
TX packets:15209428 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2314496738 (2.3 GB) TX bytes:17247449206 (17.2 GB)
bond1 Link encap:Ethernet HWaddr 00:10:1xxxx:ce
inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::210:18ff:fed3:b1ce/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:161091053340 errors:0 dropped:1071 overruns:0 frame:13821
TX packets:112926434041 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:99357307904176 (99.3 TB) TX bytes:45744253012472 (45.7 TB)
eth0 Link encap:Ethernet HWaddr 90:b1:xxxxxx4b:1e
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:11806289 errors:0 dropped:563346 overruns:0 frame:0
TX packets:15209428 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2314496738 (2.3 GB) TX bytes:17247449206 (17.2 GB)
Interrupt:16
eth1 Link encap:Ethernet HWaddr 90:b1:1xxxxxx:1e
UP BROADCAST SLAVE MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:17
eth2 Link encap:Ethernet HWaddr 00:10:xxxxx1:ce
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:161091053340 errors:0 dropped:1070 overruns:0 frame:13821
TX packets:112926434041 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:99357307904176 (99.3 TB) TX bytes:45744253012472 (45.7 TB)
Interrupt:48
eth3 Link encap:Ethernet HWaddr 00:10xxxb1:ce
UP BROADCAST SLAVE MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:52
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:6935638599 errors:0 dropped:0 overruns:0 frame:0
TX packets:6935638599 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:18028725295176 (18.0 TB) TX bytes:18028725295176 (18.0 TB)
这是 /proc/net/bonding/bond0(bond1 类似)
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: eth0 (primary_reselect always)
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 90:b1:1c:4a:4b:1e
Slave queue ID: 0
Slave Interface: eth1
MII Status: down
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: 90:b1:1c:4a:4b:1f
Slave queue ID: 0
答案1
终于弄清楚问题所在时,又出现了另一个问题。当发生这种情况时,我们打电话给数据中心,让技术人员来检查端口和电缆。我们得到的答案是一切正常,端口在闪烁。
当我亲自去数据中心处理另一个问题时,我查看了机器后面的电缆...eth0 和 eth2 连接正确,但 eth1 和 eth3 甚至没有插入!!!!他们怎么会错过这个!
这个故事的寓意是,如果热备份已启动但无法处理故障转移,并且日志中没有任何内容,则可能是电缆或端口问题。此外,一定要自己检查,不要相信别人为你做这件事,他们大多不会在意