我正在运行 Nutanix 托管的 Ubuntu 18.04 虚拟机。时不时地,我会看到一个问题,即使用 创建 docker 网络docker-compose
,然后机器会完全无响应。我只见过三次这种情况,中间间隔了好几个月,所以很难找到规律。
我运行的导致此问题的最后一条命令是:
docker-compose -f /path/to/compose.yml up
它开始创建网络然后失败:
Creating network "compose_kong-ee" with the default driver
packet_write_wait: Connection to 10.120.160.100 port 22: Broken pipe
现在,如果我在不停止 docker 守护进程的情况下重新启动机器,系统就会崩溃。查看 kern.log,我看到 br-a249 接口在发生这种情况时(~12:30 utc)被禁用:
Jun 3 12:26:28 USDALXKADV01 kernel: [1397332.561962] br-a249e56f08c5: port 1(vethe00f673) entered disabled state
Jun 3 12:26:28 USDALXKADV01 kernel: [1397332.565381] veth6a978dc: renamed from eth0
Jun 3 12:26:28 USDALXKADV01 kernel: [1397332.583094] br-a249e56f08c5: port 1(vethe00f673) entered disabled state
Jun 3 12:26:28 USDALXKADV01 kernel: [1397332.590855] device vethe00f673 left promiscuous mode
Jun 3 12:26:28 USDALXKADV01 kernel: [1397332.590860] br-a249e56f08c5: port 1(vethe00f673) entered disabled state
Jun 3 12:29:15 USDALXKADV01 kernel: [1397500.520269] IPv6: ADDRCONF(NETDEV_UP): br-342fcad19ff7: link is not ready
Jun 3 12:48:52 USDALXKADV01 kernel: [1398677.266687] vmxnet3 0000:03:00.0 ens160: intr type 3, mode 0, 2 vectors allocated
Jun 3 12:48:52 USDALXKADV01 kernel: [1398677.268869] vmxnet3 0000:03:00.0 ens160: NIC Link is Up 10000 Mbps
Jun 3 12:48:52 USDALXKADV01 kernel: [1398677.271911] IPv6: ADDRCONF(NETDEV_UP): ens160: link is not ready
Jun 3 12:48:52 USDALXKADV01 kernel: [1398677.271929] IPv6: ADDRCONF(NETDEV_CHANGE): ens160: link becomes ready
Jun 3 12:50:57 USDALXKADV01 kernel: [1398801.703190] br-a249e56f08c5: port 2(vetha6450a1) entered disabled state
Jun 3 12:50:57 USDALXKADV01 kernel: [1398801.705254] veth1c11b8c: renamed from eth0
Jun 3 12:50:57 USDALXKADV01 kernel: [1398801.718831] br-a249e56f08c5: port 2(vetha6450a1) entered disabled state
Jun 3 12:50:57 USDALXKADV01 kernel: [1398801.726647] device vetha6450a1 left promiscuous mode
Jun 3 12:50:57 USDALXKADV01 kernel: [1398801.726652] br-a249e56f08c5: port 2(vetha6450a1) entered disabled state
当我进入 shell 时,这个接口已经启动了,它对应于我当时正在运行的两个容器之一。我docker stop
向其中一个容器发出了命令,以便进行升级:
IP address for docker0: 172.17.0.1
IP address for br-a14bcb10b447: 172.18.0.1
IP address for br-a249e56f08c5: 172.22.0.1
查看系统日志条目,我可以看到整个 ssh 会话的时间跨度:
Jun 3 12:22:33 USDALXKADV01 systemd[1]: Started Session 468 of user kong.
Jun 3 12:26:27 USDALXKADV01 containerd[1468]: time="2020-06-03T12:26:27.921166420Z" level=info msg="shim reaped" id=f3a678f2747a3398a15dd605299d1b18b9d173e6c68d4bb8c8c44e7a56c2ed2a
Jun 3 12:26:27 USDALXKADV01 dockerd[12217]: time="2020-06-03T12:26:27.933328211Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 3 12:26:28 USDALXKADV01 kernel: [1397332.561962] br-a249e56f08c5: port 1(vethe00f673) entered disabled state
Jun 3 12:26:28 USDALXKADV01 systemd-networkd[1398]: vethe00f673: Lost carrier
Jun 3 12:26:28 USDALXKADV01 systemd-timesyncd[499]: Network configuration changed, trying to establish connection.
Jun 3 12:26:28 USDALXKADV01 kernel: [1397332.565381] veth6a978dc: renamed from eth0
Jun 3 12:26:28 USDALXKADV01 systemd-udevd[14938]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 3 12:26:28 USDALXKADV01 kernel: [1397332.583094] br-a249e56f08c5: port 1(vethe00f673) entered disabled state
Jun 3 12:26:28 USDALXKADV01 systemd-networkd[1398]: vethe00f673: Link DOWN
Jun 3 12:26:28 USDALXKADV01 networkd-dispatcher[763]: WARNING:Unknown index 173 seen, reloading interface list
Jun 3 12:26:28 USDALXKADV01 kernel: [1397332.590855] device vethe00f673 left promiscuous mode
Jun 3 12:26:28 USDALXKADV01 kernel: [1397332.590860] br-a249e56f08c5: port 1(vethe00f673) entered disabled state
Jun 3 12:26:28 USDALXKADV01 networkd-dispatcher[763]: ERROR:Unknown interface index 173 seen even after reload
Jun 3 12:26:28 USDALXKADV01 systemd-timesyncd[499]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Jun 3 12:26:28 USDALXKADV01 systemd[1]: Starting OpenNebula delayed reconfiguration script...
Jun 3 12:26:28 USDALXKADV01 systemd[1]: Started OpenNebula delayed reconfiguration script.
Jun 3 12:27:28 USDALXKADV01 one-contextd[15036]: Started for type all to reconfigure
Jun 3 12:27:28 USDALXKADV01 one-contextd[15040]: Acquiring lock /var/run/one-context/one-context.lock
Jun 3 12:27:28 USDALXKADV01 one-contextd[15042]: Acquired lock /var/run/one-context/one-context.lock
Jun 3 12:27:29 USDALXKADV01 one-contextd[15055]: Reading context via vmtoolsd
Jun 3 12:27:29 USDALXKADV01 one-contextd[15064]: Comparing /var/run/one-context/context.sh.0KXIk7 and /var/run/one-context/context.sh.local for changes
Jun 3 12:27:29 USDALXKADV01 one-contextd[15066]: No changes in context, skipping
Jun 3 12:27:29 USDALXKADV01 one-contextd[15067]: Comparing /var/run/one-context/context.sh.0KXIk7 and /var/run/one-context/context.sh.network for changes
Jun 3 12:27:29 USDALXKADV01 one-contextd[15069]: No changes in context, skipping
Jun 3 12:27:29 USDALXKADV01 one-contextd[15070]: Done
Jun 3 12:27:29 USDALXKADV01 one-contextd[15071]: Unmounting /var/run/one-context/mount.VEaypB
Jun 3 12:27:29 USDALXKADV01 one-contextd[15075]: Releasing lock /var/run/one-context/one-context.lock
Jun 3 12:29:15 USDALXKADV01 systemd-udevd[15148]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 3 12:29:15 USDALXKADV01 systemd-networkd[1398]: br-342fcad19ff7: Link UP
Jun 3 12:29:15 USDALXKADV01 systemd-timesyncd[499]: Network configuration changed, trying to establish connection.
Jun 3 12:29:15 USDALXKADV01 networkd-dispatcher[763]: WARNING:Unknown index 177 seen, reloading interface list
Jun 3 12:29:15 USDALXKADV01 kernel: [1397500.520269] IPv6: ADDRCONF(NETDEV_UP): br-342fcad19ff7: link is not ready
Jun 3 12:29:16 USDALXKADV01 systemd-timesyncd[499]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Jun 3 12:29:16 USDALXKADV01 systemd[1]: Starting OpenNebula delayed reconfiguration script...
Jun 3 12:29:16 USDALXKADV01 systemd[1]: Started OpenNebula delayed reconfiguration script.
Jun 3 12:29:16 USDALXKADV01 systemd-timesyncd[499]: Network configuration changed, trying to establish connection.
Jun 3 12:29:16 USDALXKADV01 systemd-timesyncd[499]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Jun 3 12:30:16 USDALXKADV01 one-contextd[15249]: Started for type all to reconfigure
Jun 3 12:30:16 USDALXKADV01 one-contextd[15253]: Acquiring lock /var/run/one-context/one-context.lock
Jun 3 12:30:16 USDALXKADV01 one-contextd[15255]: Acquired lock /var/run/one-context/one-context.lock
Jun 3 12:30:16 USDALXKADV01 one-contextd[15268]: Reading context via vmtoolsd
Jun 3 12:30:16 USDALXKADV01 one-contextd[15277]: Comparing /var/run/one-context/context.sh.JXXPUJ and /var/run/one-context/context.sh.local for changes
Jun 3 12:30:16 USDALXKADV01 one-contextd[15279]: No changes in context, skipping
Jun 3 12:30:16 USDALXKADV01 one-contextd[15280]: Comparing /var/run/one-context/context.sh.JXXPUJ and /var/run/one-context/context.sh.network for changes
Jun 3 12:30:16 USDALXKADV01 one-contextd[15282]: No changes in context, skipping
Jun 3 12:30:16 USDALXKADV01 one-contextd[15283]: Done
Jun 3 12:30:16 USDALXKADV01 one-contextd[15284]: Unmounting /var/run/one-context/mount.arNJ1s
Jun 3 12:30:16 USDALXKADV01 one-contextd[15288]: Releasing lock /var/run/one-context/one-context.lock
什么会导致这个问题?我宁愿不必重新安装 docker,因为这似乎只是权宜之计,并没有触及问题的关键。这似乎与 docker 如何设置网络有关,但我不能 100% 肯定这就是问题所在。
我正在使用 Docker 版本 19.03.8,构建 afacb8b7f0
更新
我做了一件非常糟糕的事情,并编辑了 local-kv.db 文件,从而删除了其中一个网络/var/lib/docker/network/files
。删除网络后,docker 就可以顺利启动了。
虽然这解决了问题,但它仍然没有给我一个明确的答案,让我知道这里发生了什么。似乎在这台特定机器上的默认docker配置中,你在任何时候可以拥有的网络数量是有上限的
答案1
或许:
您尝试使用已在另一台设备上设置的 IP 范围,这会中断您的路由。
默认网络范围在 /etc/docker/daemon.json 中设置:
{ "default-address-pools": [ {"base":"10.10.0.0/16","size":24} ] }
也可能是你达到了软/硬限制
或者只是破坏网络,因为您的虚拟机没有为网络堆栈分配足够的 RAM(shm)cat /etc/sysctl.conf |grep vm。
实际上你应该调整:
系统可以处理的打开文件的数量http://www.dba-oracle.com/t_increase_number_of_open_file_descriptors.htm
- 确保禁用交换
调查您的 sysctl 值,特别是以“vm”开头的值。
(例如:调整
sysctl-w vm.swappiness=10;sysctl -w vm.max_map_count=262144
)Kubernetes、Docker 和 vm.max_map_count