我有 2 个节点，它们通过 FC 连接到共享存储 (3Par)。我有 32 个虚拟卷导出到节点。我使用卷（格式化为 ocfs2）作为 minio 后端。因此，许多小文件将存储在其中。当我尝试将数据从另一个 minio 迁移到使用 FC 卷的这个 minio 时，Ubuntu 崩溃了。当我检查结果时，dmesg我看到了以下几行：
[Tue Apr 5 06:57:36 2022] TCP: request_sock_TCP: Possible SYN flooding on port 9000. Sending cookies. Check SNMP counters.
[Tue Apr 5 07:09:49 2022] perf: interrupt took too long (2517 > 2500), lowering kernel.perf_event_max_sample_rate to 79250
[Tue Apr 5 07:19:29 2022] perf: interrupt took too long (3151 > 3146), lowering kernel.perf_event_max_sample_rate to 63250
[Tue Apr 5 07:37:04 2022] perf: interrupt took too long (3940 > 3938), lowering kernel.perf_event_max_sample_rate to 50750
[Tue Apr 5 08:19:42 2022] perf: interrupt took too long (4926 > 4925), lowering kernel.perf_event_max_sample_rate to 40500
[Tue Apr 5 08:28:03 2022] ------------[ cut here ]------------
[Tue Apr 5 08:28:03 2022] rq->tmp_alone_branch != &rq->leaf_cfs_rq_list
[Tue Apr 5 08:28:03 2022] WARNING: CPU: 28 PID: 0 at /build/linux-se2LdG/linux-4.15.0/kernel/sched/fair.c:393 unthrottle_cfs_rq+0x16f/0x200
[Tue Apr 5 08:28:03 2022] Modules linked in: binfmt_misc xt_esp esp4 xfrm4_mode_transport xt_policy ip_set_hash_net veth dummy vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch nsh xt_NFLOG xt_physdev xt_state xt_recent xt_nat xt_statistic ipt_REJECT nf_reject_ipv4 xt_tcpudp nfnetlink_log ip_vs_sh ip_vs_wrr ip_vs_rr ip_set_hash_ip ip_vs xt_set ip_set nf_tables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables iptable_mangle xt_comment xt_mark xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc dlm aufs ocfs2_dlmfs overlay ocfs2_stack_o2cb ocfs2_dlm ocfs2 ocfs2_nodemanager ocfs2_stackglue quota_tree bonding
[Tue Apr 5 08:28:03 2022] nls_iso8859_1 dm_service_time dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua nd_pmem dax_pmem nd_btt device_dax intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass intel_cstate intel_rapl_perf ipmi_ssif input_leds ipmi_si ipmi_devintf hpilo ipmi_msghandler mei_me mac_hid ioatdma mei nfit acpi_power_meter dca lpc_ich shpchp sch_fq_codel ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx5_ib ib_core ses enclosure crct10dif_pclmul crc32_pclmul mgag200 ghash_clmulni_intel i2c_algo_bit ttm pcbc qla2xxx hid_generic drm_kms_helper mlx5_core aesni_intel syscopyarea
[Tue Apr 5 08:28:03 2022] aes_x86_64 nvme_fc sysfillrect mlxfw sysimgblt crypto_simd nvme_fabrics tg3 glue_helper fb_sys_fops devlink uas usbhid nvme_core smartpqi cryptd ptp scsi_transport_sas drm usb_storage scsi_transport_fc hid ahci pps_core libahci wmi
[Tue Apr 5 08:28:03 2022] CPU: 28 PID: 0 Comm: swapper/28 Not tainted 4.15.0-175-generic #184-Ubuntu
[Tue Apr 5 08:28:03 2022] Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 Gen10, BIOS U30 09/05/2019
[Tue Apr 5 08:28:03 2022] RIP: 0010:unthrottle_cfs_rq+0x16f/0x200
[Tue Apr 5 08:28:03 2022] RSP: 0018:ffff8d79a0003e80 EFLAGS: 00010082
[Tue Apr 5 08:28:03 2022] RAX: 0000000000000000 RBX: ffff8d797c974200 RCX: 0000000000000000
[Tue Apr 5 08:28:03 2022] RDX: 000000000000002d RSI: ffffffffa3168c4d RDI: 0000000000000046
[Tue Apr 5 08:28:03 2022] RBP: ffff8d79a0003ea8 R08: 0000000000000004 R09: ffffffffa3168c20
[Tue Apr 5 08:28:03 2022] R10: ffff8d79a0003e38 R11: 00045cf705aa287a R12: ffff8d7980808200
[Tue Apr 5 08:28:03 2022] R13: ffff8d799fc66940 R14: 0000000000000001 R15: 0000000000000001
[Tue Apr 5 08:28:03 2022] FS: 0000000000000000(0000) GS:ffff8d79a0000000(0000) knlGS:0000000000000000
[Tue Apr 5 08:28:03 2022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Tue Apr 5 08:28:03 2022] CR2: 00007fffa7ade1ac CR3: 0000000f2ec0a006 CR4: 00000000007606e0
[Tue Apr 5 08:28:03 2022] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[Tue Apr 5 08:28:03 2022] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[Tue Apr 5 08:28:03 2022] PKRU: 55555554
[Tue Apr 5 08:28:03 2022] Call Trace:
[Tue Apr 5 08:28:03 2022] <IRQ>
[Tue Apr 5 08:28:03 2022] distribute_cfs_runtime+0xc3/0x110
[Tue Apr 5 08:28:03 2022] sched_cfs_period_timer+0xff/0x220
[Tue Apr 5 08:28:03 2022] ? sched_cfs_slack_timer+0xd0/0xd0
[Tue Apr 5 08:28:03 2022] __hrtimer_run_queues+0xdf/0x230
[Tue Apr 5 08:28:03 2022] hrtimer_interrupt+0xa0/0x1d0
[Tue Apr 5 08:28:03 2022] smp_apic_timer_interrupt+0x6f/0x140
[Tue Apr 5 08:28:03 2022] apic_timer_interrupt+0x90/0xa0
[Tue Apr 5 08:28:03 2022] </IRQ>
[Tue Apr 5 08:28:03 2022] RIP: 0010:cpuidle_enter_state+0xa7/0x2f0
[Tue Apr 5 08:28:03 2022] RSP: 0018:ffffac764c607e68 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11
[Tue Apr 5 08:28:03 2022] RAX: ffff8d79a0026940 RBX: 0000067c0f219690 RCX: 000000000000001f
[Tue Apr 5 08:28:03 2022] RDX: 0000067c0f219690 RSI: fffdf158dc91edcc RDI: 0000000000000000
[Tue Apr 5 08:28:03 2022] RBP: ffffac764c607ea8 R08: 0000000000000004 R09: 0000000000026180
[Tue Apr 5 08:28:03 2022] R10: ffffac764c607e38 R11: 00045cf705a93e4e R12: ffffcc59c0802068
[Tue Apr 5 08:28:03 2022] R13: 0000000000000002 R14: ffffffffa2d732b8 R15: 0000000000000000
[Tue Apr 5 08:28:03 2022] ? cpuidle_enter_state+0x97/0x2f0
[Tue Apr 5 08:28:03 2022] cpuidle_enter+0x17/0x20
[Tue Apr 5 08:28:03 2022] call_cpuidle+0x23/0x40
[Tue Apr 5 08:28:03 2022] do_idle+0x18c/0x1f0
[Tue Apr 5 08:28:03 2022] cpu_startup_entry+0x73/0x80
[Tue Apr 5 08:28:03 2022] start_secondary+0x1ab/0x200
[Tue Apr 5 08:28:03 2022] secondary_startup_64+0xa5/0xb0
[Tue Apr 5 08:28:03 2022] Code: 50 09 00 00 49 39 85 60 09 00 00 74 68 80 3d 7a 59 54 01 00 75 5f 31 db 48 c7 c7 f0 48 8d a2 c6 05 68 59 54 01 01 e8 41 30 fc ff <0f> 0b 48 85 db 74 43 49 8b 85 78 09 00 00 49 39 85 70 09 00 00
[Tue Apr 5 08:28:03 2022] ---[ end trace 2c427286bebe289a ]---
是什么原因导致了这个问题？我该如何解决？
答案1
这看起来好像系统连接套接字用完了。如果你正在使用很多小文件，很容易淹没内核的套接字积压。对于全部现代 Linux 系统，不仅仅是 Ubuntu。
幸运的是，可以通过更新限制来缓解这一问题net.core.somaxconn。
首先你需要检查你当前的限制是多少：
sysctl net.core.somaxconn
很多系统都会4096默认设置，一般就足够了。
sysctl您可以通过如下方式临时更新该值：
sysctl -w net.core.somaxconn=8192
笔记：确保以小幅度增加值。如果您131072立即输入一些荒谬的数字，系统可能会因其他原因开始锁定。
如果这个号码有用，您可以将其添加到您的/etc/sysctl.conf文件中，使其在重新启动后仍然存在。
某些系统可能还需要增加同时握手连接的数量。您可以使用以下命令检查当前限制：
sysctl net.ipv4.tcp_max_syn_backlog
此数字不应超过您的net.core.somaxconn限额的 50%。