我正在使用 docker 容器运行一些自动化测试,其中包括创建大量veth
接口(每个容器大约 2 个,约 1000 个容器)。
每当我这样做时,系统就会关闭。读取日志/var/log/messages
告诉我检查/var/crash/timestamp
目录中的日志。打开 dmesg.timestamp 文件,我在文件末尾看到此日志消息(在容器相关消息之后):
[ 1094.309055] BUG: unable to handle kernel NULL pointer dereference at 0000000000000078
[ 1094.309142] IP: [<ffffffff810a45f8>] pick_next_task_fair+0x6b8/0x820
[ 1094.309197] PGD 0
[ 1094.309216] Oops: 0000 [#1] SMP
[ 1094.309245] Modules linked in: veth xfrm_user xfrm_algo xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables tun bridge stp llc nvidia_uvm(PO) aufs(C) nvidia(PO) bnep bluetooth 6lowpan_iphc rfkill binfmt_misc nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc nls_utf8 nls_cp437 vfat fat fuse ecryptfs joydev x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crc32_pclmul snd_hda_codec_hdmi aesni_intel aes_x86_64 lrw gf128mul snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep glue_helper tilegxpci(O) mgag200 ttm evdev snd_pcm drm_kms_helper dcdbas ipmi_devintf ablk_helper iTCO_wdt iTCO_vendor_support
[ 1094.309954] snd_timer drm acpi_power_meter cryptd button pcspkr processor acpi_pad snd soundcore lpc_ich efi_pstore efivars ipmi_si ipmi_msghandler shpchp tpm_tis sb_edac tpm wmi mfd_core mei_me ioatdma thermal_sys edac_core mei ext4 crc16 mbcache jbd2 dm_mod hid_generic usbhid hid sr_mod cdrom sg sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci ehci_pci ehci_hcd igb i40e(O) i2c_algo_bit libata megaraid_sas i2c_core usbcore dca usb_common vxlan scsi_mod ptp pps_core
[ 1094.310428] CPU: 4 PID: 30 Comm: ksoftirqd/4 Tainted: P C O 3.16.0-4-amd64 #1 Debian 3.16.39-1
[ 1094.310503] Hardware name: Dell Inc. PowerEdge T620/03GCPM, BIOS 2.1.3 11/20/2013
[ 1094.310589] task: ffff88041b306150 ti: ffff88041b314000 task.ti: ffff88041b314000
[ 1094.310664] RIP: 0010:[<ffffffff810a45f8>] [<ffffffff810a45f8>] pick_next_task_fair+0x6b8/0x820
[ 1094.310756] RSP: 0018:ffff88041b317de0 EFLAGS: 00010046
[ 1094.310809] RAX: 000000000001264c RBX: ffff8801fb46e4c0 RCX: 0000000000000000
[ 1094.310867] RDX: 0000000000000001 RSI: ffff8801fb47ba28 RDI: ffff88059d8a7668
[ 1094.310920] RBP: ffff8801fb47ba00 R08: 0000000000000000 R09: 000000000000b73a
[ 1094.310975] R10: 0000000000000000 R11: ffffffffffffffa0 R12: 0000000000000000
[ 1094.311025] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88042fa52f40
[ 1094.311093] FS: 0000000000000000(0000) GS:ffff88042fa40000(0000) knlGS:0000000000000000
[ 1094.311157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1094.311203] CR2: 0000000000000078 CR3: 0000000005813000 CR4: 00000000001407e0
[ 1094.311267] Stack:
[ 1094.311292] ffff88059d8a75f0 00000001810a0964 ffff88041b306150 0000000000012f40
[ 1094.311378] ffff88042fa52fb8 ffffffff8101ca75 ffff88041b3065a8 ffff88041b306150
[ 1094.311449] ffff88042fa52f40 0000000000000004 0000000000000000 0000000000000000
[ 1094.311535] Call Trace:
[ 1094.311571] [<ffffffff8101ca75>] ? sched_clock+0x5/0x10
[ 1094.311642] [<ffffffff81517006>] ? __schedule+0x106/0x6f0
[ 1094.311691] [<ffffffff81090036>] ? smpboot_thread_fn+0xc6/0x190
[ 1094.311744] [<ffffffff8108ff70>] ? SyS_setgroups+0x170/0x170
[ 1094.311786] [<ffffffff810894fd>] ? kthread+0xbd/0xe0
[ 1094.311822] [<ffffffff81089440>] ? kthread_create_on_node+0x180/0x180
[ 1094.311876] [<ffffffff8151ad18>] ? ret_from_fork+0x58/0x90
[ 1094.311916] [<ffffffff81089440>] ? kthread_create_on_node+0x180/0x180
[ 1094.311976] Code: 49 8b 7c 24 78 48 39 fd 74 2f 44 8b 73 68 45 8b 6c 24 68 45 39 ee 0f 8e c7 00 00 00 48 89 ef 48 89 de e8 ac 91 ff ff 48 8b 5b 70 <49> 8b 7c 24 78 48 8b 6b 78 48 39 fd 75 d1 48 85 ed 74 cc 4c 89
[ 1094.312277] RIP [<ffffffff810a45f8>] pick_next_task_fair+0x6b8/0x820
[ 1094.314275] RSP <ffff88041b317de0>
[ 1094.316245] CR2: 0000000000000078
使用略有不同的容器拓扑多次运行测试,上面显示的日志的第一行始终相同:被取消引用的指针始终位于地址0000000000000078
。
我非常确定容器的数量不是发生这种情况的原因,因为我尝试在不创建虚拟接口的情况下运行容器,并且一切正常。
因此,我假设一次可以拥有的虚拟接口的数量存在某种限制。这是硬件定义的限制,还是我可以更改一些选项以便能够使用更多接口?