我们有一台 DELL R610 服务器连接到 DELL PowerConnect 5424 交换机。此交换机连接到 DELL Equallogic SAN。DELL R610 用作 MySQL 数据库服务器,SAN 提供数据目录,作为 iSCSI 驱动器安装。
根据我之前提出的一个问题这里,我们观察到重启后,MySQL 无法自行启动。这也是一种间歇性行为。调查后我们发现 iSCSI 启动器服务在启动时无法执行命令。检查日志后,我们发现网络接口启动时会出现一个小故障。使用 grep 命令查看 bnx2 日志:
bnx2: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.2.4 (Aug 05, 2013)
bnx2 0000:01:00.0 eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem d6000000, IRQ 36, node addr 5c:f9:dd:f1:8a:ea
bnx2 0000:01:00.1 eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem d8000000, IRQ 48, node addr 5c:f9:dd:f1:8a:ec
bnx2 0000:02:00.0 eth2: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem da000000, IRQ 32, node addr 5c:f9:dd:f1:8a:ee
bnx2 0000:02:00.1 eth3: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem dc000000, IRQ 42, node addr 5c:f9:dd:f1:8a:f0
bnx2 0000:02:00.0: irq 78 for MSI/MSI-X
bnx2 0000:02:00.0: irq 79 for MSI/MSI-X
bnx2 0000:02:00.0: irq 80 for MSI/MSI-X
bnx2 0000:02:00.0: irq 81 for MSI/MSI-X
bnx2 0000:02:00.0: irq 82 for MSI/MSI-X
bnx2 0000:02:00.0: irq 83 for MSI/MSI-X
bnx2 0000:02:00.0: irq 84 for MSI/MSI-X
bnx2 0000:02:00.0: irq 85 for MSI/MSI-X
bnx2 0000:02:00.0: irq 86 for MSI/MSI-X
bnx2 0000:02:00.0 em3: using MSIX
bnx2 0000:01:00.0: irq 87 for MSI/MSI-X
bnx2 0000:01:00.0: irq 88 for MSI/MSI-X
bnx2 0000:01:00.0: irq 89 for MSI/MSI-X
bnx2 0000:01:00.0: irq 90 for MSI/MSI-X
bnx2 0000:01:00.0: irq 91 for MSI/MSI-X
bnx2 0000:01:00.0: irq 92 for MSI/MSI-X
bnx2 0000:01:00.0: irq 93 for MSI/MSI-X
bnx2 0000:01:00.0: irq 94 for MSI/MSI-X
bnx2 0000:01:00.0: irq 95 for MSI/MSI-X
bnx2 0000:01:00.0 em1: using MSIX
bnx2 0000:01:00.1: irq 96 for MSI/MSI-X
bnx2 0000:01:00.1: irq 97 for MSI/MSI-X
bnx2 0000:01:00.1: irq 98 for MSI/MSI-X
bnx2 0000:01:00.1: irq 99 for MSI/MSI-X
bnx2 0000:01:00.1: irq 100 for MSI/MSI-X
bnx2 0000:01:00.1: irq 101 for MSI/MSI-X
bnx2 0000:01:00.1: irq 102 for MSI/MSI-X
bnx2 0000:01:00.1: irq 103 for MSI/MSI-X
bnx2 0000:01:00.1: irq 104 for MSI/MSI-X
bnx2 0000:01:00.1 em2: using MSIX
bnx2 0000:01:00.0 em1: NIC Copper Link is Up, 100 Mbps full duplex
bnx2 0000:02:00.0 em3: NIC Copper Link is Up, 1000 Mbps full duplex
bnx2 0000:01:00.1 em2: NIC Copper Link is Up, 1000 Mbps full duplex
**bnx2 0000:01:00.1 em2: NIC Copper Link is Down**
bnx2 0000:01:00.1 em2: NIC Copper Link is Up, 1000 Mbps full duplex
我们目前的解决方法是重启服务器。到目前为止,重启服务器后,一切都顺利进行,没有观察到上面日志中显示的故障。
有人能帮忙解决故障吗?我已经提到这里但这很可能不是我的情况,因为我们的问题只发生在重启时。除此之外,ifconfig 中没有 NIC 错误,NIC 中也没有数据包丢失。服务器启动后,我们从未遇到过任何网络问题。
DELL R610 运行的是 Ubuntu 14.04。
@Dom 建议的更多日志:
$ cat logfile.mysql.withoutdate |grep -B 15 -A 15 "NIC Copper Link is Down"
Loading iSCSI transport class v2.0-870.
bnx2 0000:01:00.1 em2: using MSIX
IPv6: ADDRCONF(NETDEV_UP): em2: link is not ready
iscsi: registered transport (tcp)
iscsi: registered transport (iser)
multipathd (2470): /proc/2470/oom_adj is deprecated, please use /proc/2470/oom_score_adj instead.
bnx2 0000:01:00.0 em1: NIC Copper Link is Up, 100 Mbps full duplex
IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready
bnx2 0000:02:00.0 em3: NIC Copper Link is Up, 1000 Mbps full duplex
, receive & transmit flow control ON
IPv6: ADDRCONF(NETDEV_CHANGE): em3: link becomes ready
bnx2 0000:01:00.1 em2: NIC Copper Link is Up, 1000 Mbps full duplex
, receive & transmit flow control ON
IPv6: ADDRCONF(NETDEV_CHANGE): em2: link becomes ready
bnx2 0000:01:00.1 em2: NIC Copper Link is Down
bnx2 0000:01:00.1 em2: NIC Copper Link is Up, 1000 Mbps full duplex
, receive & transmit flow control ON