Docker 导致虚拟机关闭

Docker 导致虚拟机关闭

我正在运行 Nutanix 托管的 Ubuntu 18.04 虚拟机。时不时地,我会看到一个问题,即使用 创建 docker 网络docker-compose,然后机器会完全无响应。我只见过三次这种情况,中间间隔了好几个月,所以很难找到规律。

我运行的导致此问题的最后一条命令是:

docker-compose -f /path/to/compose.yml up

它开始创建网络然后失败:

Creating network "compose_kong-ee" with the default driver
packet_write_wait: Connection to 10.120.160.100 port 22: Broken pipe

现在,如果我在不停止 docker 守护进程的情况下重新启动机器,系统就会崩溃。查看 kern.log,我看到 br-a249 接口在发生这种情况时(~12:30 utc)被禁用:

Jun  3 12:26:28 USDALXKADV01 kernel: [1397332.561962] br-a249e56f08c5: port 1(vethe00f673) entered disabled state
Jun  3 12:26:28 USDALXKADV01 kernel: [1397332.565381] veth6a978dc: renamed from eth0
Jun  3 12:26:28 USDALXKADV01 kernel: [1397332.583094] br-a249e56f08c5: port 1(vethe00f673) entered disabled state
Jun  3 12:26:28 USDALXKADV01 kernel: [1397332.590855] device vethe00f673 left promiscuous mode
Jun  3 12:26:28 USDALXKADV01 kernel: [1397332.590860] br-a249e56f08c5: port 1(vethe00f673) entered disabled state
Jun  3 12:29:15 USDALXKADV01 kernel: [1397500.520269] IPv6: ADDRCONF(NETDEV_UP): br-342fcad19ff7: link is not ready
Jun  3 12:48:52 USDALXKADV01 kernel: [1398677.266687] vmxnet3 0000:03:00.0 ens160: intr type 3, mode 0, 2 vectors allocated
Jun  3 12:48:52 USDALXKADV01 kernel: [1398677.268869] vmxnet3 0000:03:00.0 ens160: NIC Link is Up 10000 Mbps
Jun  3 12:48:52 USDALXKADV01 kernel: [1398677.271911] IPv6: ADDRCONF(NETDEV_UP): ens160: link is not ready
Jun  3 12:48:52 USDALXKADV01 kernel: [1398677.271929] IPv6: ADDRCONF(NETDEV_CHANGE): ens160: link becomes ready
Jun  3 12:50:57 USDALXKADV01 kernel: [1398801.703190] br-a249e56f08c5: port 2(vetha6450a1) entered disabled state
Jun  3 12:50:57 USDALXKADV01 kernel: [1398801.705254] veth1c11b8c: renamed from eth0
Jun  3 12:50:57 USDALXKADV01 kernel: [1398801.718831] br-a249e56f08c5: port 2(vetha6450a1) entered disabled state
Jun  3 12:50:57 USDALXKADV01 kernel: [1398801.726647] device vetha6450a1 left promiscuous mode
Jun  3 12:50:57 USDALXKADV01 kernel: [1398801.726652] br-a249e56f08c5: port 2(vetha6450a1) entered disabled state

当我进入 shell 时,这个接口已经启动了,它对应于我当时正在运行的两个容器之一。我docker stop向其中一个容器发出了命令,以便进行升级:

  IP address for docker0:         172.17.0.1
  IP address for br-a14bcb10b447: 172.18.0.1
  IP address for br-a249e56f08c5: 172.22.0.1

查看系统日志条目,我可以看到整个 ssh 会话的时间跨度:

Jun  3 12:22:33 USDALXKADV01 systemd[1]: Started Session 468 of user kong.
Jun  3 12:26:27 USDALXKADV01 containerd[1468]: time="2020-06-03T12:26:27.921166420Z" level=info msg="shim reaped" id=f3a678f2747a3398a15dd605299d1b18b9d173e6c68d4bb8c8c44e7a56c2ed2a
Jun  3 12:26:27 USDALXKADV01 dockerd[12217]: time="2020-06-03T12:26:27.933328211Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun  3 12:26:28 USDALXKADV01 kernel: [1397332.561962] br-a249e56f08c5: port 1(vethe00f673) entered disabled state
Jun  3 12:26:28 USDALXKADV01 systemd-networkd[1398]: vethe00f673: Lost carrier
Jun  3 12:26:28 USDALXKADV01 systemd-timesyncd[499]: Network configuration changed, trying to establish connection.
Jun  3 12:26:28 USDALXKADV01 kernel: [1397332.565381] veth6a978dc: renamed from eth0
Jun  3 12:26:28 USDALXKADV01 systemd-udevd[14938]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun  3 12:26:28 USDALXKADV01 kernel: [1397332.583094] br-a249e56f08c5: port 1(vethe00f673) entered disabled state
Jun  3 12:26:28 USDALXKADV01 systemd-networkd[1398]: vethe00f673: Link DOWN
Jun  3 12:26:28 USDALXKADV01 networkd-dispatcher[763]: WARNING:Unknown index 173 seen, reloading interface list
Jun  3 12:26:28 USDALXKADV01 kernel: [1397332.590855] device vethe00f673 left promiscuous mode
Jun  3 12:26:28 USDALXKADV01 kernel: [1397332.590860] br-a249e56f08c5: port 1(vethe00f673) entered disabled state
Jun  3 12:26:28 USDALXKADV01 networkd-dispatcher[763]: ERROR:Unknown interface index 173 seen even after reload
Jun  3 12:26:28 USDALXKADV01 systemd-timesyncd[499]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Jun  3 12:26:28 USDALXKADV01 systemd[1]: Starting OpenNebula delayed reconfiguration script...
Jun  3 12:26:28 USDALXKADV01 systemd[1]: Started OpenNebula delayed reconfiguration script.
Jun  3 12:27:28 USDALXKADV01 one-contextd[15036]: Started for type all to reconfigure
Jun  3 12:27:28 USDALXKADV01 one-contextd[15040]: Acquiring lock /var/run/one-context/one-context.lock
Jun  3 12:27:28 USDALXKADV01 one-contextd[15042]: Acquired lock /var/run/one-context/one-context.lock
Jun  3 12:27:29 USDALXKADV01 one-contextd[15055]: Reading context via vmtoolsd
Jun  3 12:27:29 USDALXKADV01 one-contextd[15064]: Comparing /var/run/one-context/context.sh.0KXIk7 and /var/run/one-context/context.sh.local for changes
Jun  3 12:27:29 USDALXKADV01 one-contextd[15066]: No changes in context, skipping
Jun  3 12:27:29 USDALXKADV01 one-contextd[15067]: Comparing /var/run/one-context/context.sh.0KXIk7 and /var/run/one-context/context.sh.network for changes
Jun  3 12:27:29 USDALXKADV01 one-contextd[15069]: No changes in context, skipping
Jun  3 12:27:29 USDALXKADV01 one-contextd[15070]: Done
Jun  3 12:27:29 USDALXKADV01 one-contextd[15071]: Unmounting /var/run/one-context/mount.VEaypB
Jun  3 12:27:29 USDALXKADV01 one-contextd[15075]: Releasing lock /var/run/one-context/one-context.lock
Jun  3 12:29:15 USDALXKADV01 systemd-udevd[15148]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun  3 12:29:15 USDALXKADV01 systemd-networkd[1398]: br-342fcad19ff7: Link UP
Jun  3 12:29:15 USDALXKADV01 systemd-timesyncd[499]: Network configuration changed, trying to establish connection.
Jun  3 12:29:15 USDALXKADV01 networkd-dispatcher[763]: WARNING:Unknown index 177 seen, reloading interface list
Jun  3 12:29:15 USDALXKADV01 kernel: [1397500.520269] IPv6: ADDRCONF(NETDEV_UP): br-342fcad19ff7: link is not ready
Jun  3 12:29:16 USDALXKADV01 systemd-timesyncd[499]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Jun  3 12:29:16 USDALXKADV01 systemd[1]: Starting OpenNebula delayed reconfiguration script...
Jun  3 12:29:16 USDALXKADV01 systemd[1]: Started OpenNebula delayed reconfiguration script.
Jun  3 12:29:16 USDALXKADV01 systemd-timesyncd[499]: Network configuration changed, trying to establish connection.
Jun  3 12:29:16 USDALXKADV01 systemd-timesyncd[499]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Jun  3 12:30:16 USDALXKADV01 one-contextd[15249]: Started for type all to reconfigure
Jun  3 12:30:16 USDALXKADV01 one-contextd[15253]: Acquiring lock /var/run/one-context/one-context.lock
Jun  3 12:30:16 USDALXKADV01 one-contextd[15255]: Acquired lock /var/run/one-context/one-context.lock
Jun  3 12:30:16 USDALXKADV01 one-contextd[15268]: Reading context via vmtoolsd
Jun  3 12:30:16 USDALXKADV01 one-contextd[15277]: Comparing /var/run/one-context/context.sh.JXXPUJ and /var/run/one-context/context.sh.local for changes
Jun  3 12:30:16 USDALXKADV01 one-contextd[15279]: No changes in context, skipping
Jun  3 12:30:16 USDALXKADV01 one-contextd[15280]: Comparing /var/run/one-context/context.sh.JXXPUJ and /var/run/one-context/context.sh.network for changes
Jun  3 12:30:16 USDALXKADV01 one-contextd[15282]: No changes in context, skipping
Jun  3 12:30:16 USDALXKADV01 one-contextd[15283]: Done
Jun  3 12:30:16 USDALXKADV01 one-contextd[15284]: Unmounting /var/run/one-context/mount.arNJ1s
Jun  3 12:30:16 USDALXKADV01 one-contextd[15288]: Releasing lock /var/run/one-context/one-context.lock

什么会导致这个问题?我宁愿不必重新安装 docker,因为这似乎只是权宜之计,并没有触及问题的关键。这似乎与 docker 如何设置网络有关,但我不能 100% 肯定这就是问题所在。

我正在使用 Docker 版本 19.03.8,构建 afacb8b7f0

更新

我做了一件非常糟糕的事情,并编辑了 local-kv.db 文件,从而删除了其中一个网络/var/lib/docker/network/files。删除网络后,docker 就可以顺利启动了。

虽然这解决了问题,但它仍然没有给我一个明确的答案,让我知道这里发生了什么。似乎在这台特定机器上的默认docker配置中,你在任何时候可以拥有的网络数量是有上限的

答案1

或许:

实际上你应该调整:

相关内容