我希望有人能够帮助我解决这里的问题。我的家庭服务器运行的是 Arch Linux,目前使用 5.4.60-1-lts 内核。我目前在此系统上使用 ZFS 0.8.4-1(由 zfs-dkms 软件包安装)。我的主文件夹实际上驻留在 ZFS 池上,但操作系统的其余部分仅位于单独的 SSD 上。有一天,我打开 Chromium 浏览器,发现它加载网页的速度要么非常慢,要么在大多数情况下根本不加载。在 Chromium 窗口的左下角,它一直显示“等待缓存”,并且不加载页面。我想我应该从我的主目录中删除 .cache/chromium 文件夹。
我执行了命令sudo rm -r /home/justin/.cache/chromium
,一开始我可以听到硬盘驱动器启动的声音,但命令从未完成。它只是挂了。但它并没有锁定整个系统,所以我查看了 htop,发现我执行的命令的进程占用了一个 CPU 核心/线程的 100%。我把它搁置了大概几个小时,它就一直这样。从那时起,我又尝试了很多次,每次都得到相同的结果。 kill -9
也不会杀死挂起的进程,因此唯一的解决办法是重新启动系统。在文件系统中的其他位置创建/删除/移动文件似乎没问题(祈祷其他地方没有问题),但它确实不喜欢那个 chromium 缓存文件夹!
我在尝试删除文件夹时捕获了一些内核日志数据......
Aug 26 22:16:00 IXTREME kernel: watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [rm:1121909]
Aug 26 22:16:00 IXTREME kernel: Modules linked in: vhost_net vhost tap fuse xt_CHECKSUM xt_MASQUERADE ip6table_mangle ip6table_nat iptable_mangle iptable_nat ebtable_filter ebtables tun nfnetlink bridge stp llc nct6775 hwmon_vid eeepc_wmi asus_wmi battery nls_iso8859_1 sparse_keymap nls_cp437 rfkill vfat wmi_bmof mxm_wmi fat zfs(POE) snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_usb_audio edac_mce_amd snd_hda_intel zunicode(POE) snd_intel_nhlt zavl(POE) icp(POE) snd_usbmidi_lib kvm_amd snd_hda_codec snd_rawmidi kvm uvcvideo snd_seq_device videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common zcommon(POE) videodev znvpair(POE) crct10dif_pclmul crc32_pclmul mc ghash_clmulni_intel snd_hda_core spl(OE) snd_hwdep snd_pcm aesni_intel zlua(POE) drm_kms_helper snd_timer crypto_simd ccp cryptd snd glue_helper syscopyarea sysfillrect sp5100_tco k10temp sysimgblt pcspkr soundcore rng_core fb_sys_fops i2c_piix4 igb r8169 realtek libphy ipmi_devintf i2c_algo_bit dca ipmi_msghandler wmi
Aug 26 22:16:00 IXTREME kernel: pinctrl_amd evdev mac_hid acpi_cpufreq nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt nf_log_ipv4 nf_log_common ipt_REJECT nf_reject_ipv4 xt_LOG xt_limit xt_addrtype xt_tcpudp xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nfsd nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_filter auth_rpcgss nfs_acl lockd drm grace sg sunrpc crypto_user agpgart ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 sd_mod ahci mpt3sas libahci raid_class libata crc32c_intel xhci_pci scsi_transport_sas ehci_pci xhci_hcd scsi_mod ehci_hcd vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio
Aug 26 22:16:00 IXTREME kernel: CPU: 6 PID: 1121909 Comm: rm Tainted: P OE 5.4.60-1-lts #1
Aug 26 22:16:00 IXTREME kernel: Hardware name: System manufacturer System Product Name/Pro WS X570-ACE, BIOS 2206 08/13/2020
Aug 26 22:16:00 IXTREME kernel: RIP: 0010:zap_leaf_lookup_closest+0xbc/0x170 [zfs]
Aug 26 22:16:00 IXTREME kernel: Code: 00 00 4c 8b 70 18 8b 87 d0 00 00 00 8d 48 fb 44 89 c8 d3 e0 0f b7 ca 48 8d 0c 49 48 98 48 8d 04 88 49 8d 44 46 30 48 8b 48 10 <4c> 39 c1 76 bc 4c 39 d1 73 5a 8b 68 0c 44 0f b7 50 0a 44 89 db 4c
Aug 26 22:16:00 IXTREME kernel: RSP: 0018:ffffa3c9ed2e7bc0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13
Aug 26 22:16:00 IXTREME kernel: RAX: ffff9598004cc430 RBX: 0000000000000004 RCX: 0001000000000000
Aug 26 22:16:00 IXTREME kernel: RDX: 0000000000000000 RSI: ffffa3c9ed2e7bf0 RDI: ffff95aa9b195a00
Aug 26 22:16:00 IXTREME kernel: RBP: 00000000fcb76346 R08: 1380000000000000 R09: 0000000000000001
Aug 26 22:16:00 IXTREME kernel: R10: 7d77d659f91561db R11: 0000000000000004 R12: ffffa3c9ed2e7c06
Aug 26 22:16:00 IXTREME kernel: R13: 0000000000000000 R14: ffff9598004cc000 R15: ffffa3c9ed2e7d38
Aug 26 22:16:00 IXTREME kernel: FS: 00007fba662735c0(0000) GS:ffff95ac2e980000(0000) knlGS:0000000000000000
Aug 26 22:16:00 IXTREME kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 26 22:16:00 IXTREME kernel: CR2: 0000562deb863088 CR3: 0000000c6668e000 CR4: 0000000000340ee0
Aug 26 22:16:00 IXTREME kernel: Call Trace:
Aug 26 22:16:00 IXTREME kernel: fzap_cursor_retrieve+0x103/0x340 [zfs]
Aug 26 22:16:00 IXTREME kernel: zap_cursor_retrieve+0x16c/0x300 [zfs]
Aug 26 22:16:00 IXTREME kernel: zfs_readdir+0x147/0x420 [zfs]
Aug 26 22:16:00 IXTREME kernel: zpl_iterate+0x4c/0x70 [zfs]
Aug 26 22:16:00 IXTREME kernel: iterate_dir+0x143/0x1a0
Aug 26 22:16:00 IXTREME kernel: ksys_getdents64+0x91/0x130
Aug 26 22:16:00 IXTREME kernel: ? __ia32_compat_sys_getdents+0x130/0x130
Aug 26 22:16:00 IXTREME kernel: __x64_sys_getdents64+0x16/0x20
Aug 26 22:16:00 IXTREME kernel: do_syscall_64+0x49/0x90
Aug 26 22:16:00 IXTREME kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Aug 26 22:16:00 IXTREME kernel: RIP: 0033:0x7fba66171a8b
Aug 26 22:16:00 IXTREME kernel: Code: eb b7 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 f3 0f 1e fa 48 81 fa ff ff ff 7f b8 ff ff ff 7f 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 b1 93 0f 00 f7 d8
Aug 26 22:16:00 IXTREME kernel: RSP: 002b:00007ffc4472b468 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
Aug 26 22:16:00 IXTREME kernel: RAX: ffffffffffffffda RBX: 0000562deb7e1980 RCX: 00007fba66171a8b
Aug 26 22:16:00 IXTREME kernel: RDX: 0000000000008000 RSI: 0000562deb7e19b0 RDI: 0000000000000003
Aug 26 22:16:00 IXTREME kernel: RBP: 0000562deb7e19b0 R08: 0000000000008028 R09: 00007fba6626ba60
Aug 26 22:16:00 IXTREME kernel: R10: 0000000000000150 R11: 0000000000000293 R12: ffffffffffffff88
Aug 26 22:16:00 IXTREME kernel: R13: 0000562deb7e1984 R14: 0000000000000000 R15: 0000562deb7e9980
Aug 26 22:16:28 IXTREME kernel: watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [rm:1121909]
Aug 26 22:16:28 IXTREME kernel: Modules linked in: vhost_net vhost tap fuse xt_CHECKSUM xt_MASQUERADE ip6table_mangle ip6table_nat iptable_mangle iptable_nat ebtable_filter ebtables tun nfnetlink bridge stp llc nct6775 hwmon_vid eeepc_wmi asus_wmi battery nls_iso8859_1 sparse_keymap nls_cp437 rfkill vfat wmi_bmof mxm_wmi fat zfs(POE) snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_usb_audio edac_mce_amd snd_hda_intel zunicode(POE) snd_intel_nhlt zavl(POE) icp(POE) snd_usbmidi_lib kvm_amd snd_hda_codec snd_rawmidi kvm uvcvideo snd_seq_device videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common zcommon(POE) videodev znvpair(POE) crct10dif_pclmul crc32_pclmul mc ghash_clmulni_intel snd_hda_core spl(OE) snd_hwdep snd_pcm aesni_intel zlua(POE) drm_kms_helper snd_timer crypto_simd ccp cryptd snd glue_helper syscopyarea sysfillrect sp5100_tco k10temp sysimgblt pcspkr soundcore rng_core fb_sys_fops i2c_piix4 igb r8169 realtek libphy ipmi_devintf i2c_algo_bit dca ipmi_msghandler wmi
Aug 26 22:16:28 IXTREME kernel: pinctrl_amd evdev mac_hid acpi_cpufreq nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt nf_log_ipv4 nf_log_common ipt_REJECT nf_reject_ipv4 xt_LOG xt_limit xt_addrtype xt_tcpudp xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nfsd nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_filter auth_rpcgss nfs_acl lockd drm grace sg sunrpc crypto_user agpgart ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 sd_mod ahci mpt3sas libahci raid_class libata crc32c_intel xhci_pci scsi_transport_sas ehci_pci xhci_hcd scsi_mod ehci_hcd vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio
Aug 26 22:16:28 IXTREME kernel: CPU: 6 PID: 1121909 Comm: rm Tainted: P OEL 5.4.60-1-lts #1
Aug 26 22:16:28 IXTREME kernel: Hardware name: System manufacturer System Product Name/Pro WS X570-ACE, BIOS 2206 08/13/2020
Aug 26 22:16:28 IXTREME kernel: RIP: 0010:zap_leaf_lookup_closest+0x83/0x170 [zfs]
Aug 26 22:16:28 IXTREME kernel: Code: c2 ff ff ff ff 41 b9 01 00 00 00 4c 8d 66 16 41 0f b7 c3 41 0f b7 54 46 30 66 83 fa ff 75 20 e9 89 00 00 00 0f 84 b5 00 00 00 <0f> b7 50 02 66 83 fa ff 74 79 48 8b 87 d8 00 00 00 4c 8b 70 18 8b
Aug 26 22:16:28 IXTREME kernel: RSP: 0018:ffffa3c9ed2e7bc0 EFLAGS: 00000287 ORIG_RAX: ffffffffffffff13
Aug 26 22:16:28 IXTREME kernel: RAX: ffff9598004cc430 RBX: 0000000000000004 RCX: 0001000000000000
Aug 26 22:16:28 IXTREME kernel: RDX: 0000000000000000 RSI: ffffa3c9ed2e7bf0 RDI: ffff95aa9b195a00
Aug 26 22:16:28 IXTREME kernel: RBP: 00000000fcb76346 R08: 1380000000000000 R09: 0000000000000001
Aug 26 22:16:28 IXTREME kernel: R10: 7d77d659f91561db R11: 0000000000000004 R12: ffffa3c9ed2e7c06
Aug 26 22:16:28 IXTREME kernel: R13: 0000000000000000 R14: ffff9598004cc000 R15: ffffa3c9ed2e7d38
Aug 26 22:16:28 IXTREME kernel: FS: 00007fba662735c0(0000) GS:ffff95ac2e980000(0000) knlGS:0000000000000000
Aug 26 22:16:28 IXTREME kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 26 22:16:28 IXTREME kernel: CR2: 0000562deb863088 CR3: 0000000c6668e000 CR4: 0000000000340ee0
Aug 26 22:16:28 IXTREME kernel: Call Trace:
Aug 26 22:16:28 IXTREME kernel: fzap_cursor_retrieve+0x103/0x340 [zfs]
Aug 26 22:16:28 IXTREME kernel: zap_cursor_retrieve+0x16c/0x300 [zfs]
Aug 26 22:16:28 IXTREME kernel: zfs_readdir+0x147/0x420 [zfs]
Aug 26 22:16:28 IXTREME kernel: zpl_iterate+0x4c/0x70 [zfs]
Aug 26 22:16:28 IXTREME kernel: iterate_dir+0x143/0x1a0
Aug 26 22:16:28 IXTREME kernel: ksys_getdents64+0x91/0x130
Aug 26 22:16:28 IXTREME kernel: ? __ia32_compat_sys_getdents+0x130/0x130
Aug 26 22:16:28 IXTREME kernel: __x64_sys_getdents64+0x16/0x20
Aug 26 22:16:28 IXTREME kernel: do_syscall_64+0x49/0x90
Aug 26 22:16:28 IXTREME kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Aug 26 22:16:28 IXTREME kernel: RIP: 0033:0x7fba66171a8b
Aug 26 22:16:28 IXTREME kernel: Code: eb b7 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 f3 0f 1e fa 48 81 fa ff ff ff 7f b8 ff ff ff 7f 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 b1 93 0f 00 f7 d8
Aug 26 22:16:28 IXTREME kernel: RSP: 002b:00007ffc4472b468 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
Aug 26 22:16:28 IXTREME kernel: RAX: ffffffffffffffda RBX: 0000562deb7e1980 RCX: 00007fba66171a8b
Aug 26 22:16:28 IXTREME kernel: RDX: 0000000000008000 RSI: 0000562deb7e19b0 RDI: 0000000000000003
Aug 26 22:16:28 IXTREME kernel: RBP: 0000562deb7e19b0 R08: 0000000000008028 R09: 00007fba6626ba60
Aug 26 22:16:28 IXTREME kernel: R10: 0000000000000150 R11: 0000000000000293 R12: ffffffffffffff88
Aug 26 22:16:28 IXTREME kernel: R13: 0000562deb7e1984 R14: 0000000000000000 R15: 0000562deb7e9980
Aug 26 22:16:34 IXTREME kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Aug 26 22:16:34 IXTREME kernel: rcu: 6-....: (6000 ticks this GP) idle=4c2/1/0x4000000000000002 softirq=534040/534040 fqs=2798
Aug 26 22:16:34 IXTREME kernel: (t=6000 jiffies g=1811881 q=109219)
Aug 26 22:16:34 IXTREME kernel: NMI backtrace for cpu 6
Aug 26 22:16:34 IXTREME kernel: CPU: 6 PID: 1121909 Comm: rm Tainted: P OEL 5.4.60-1-lts #1
Aug 26 22:16:34 IXTREME kernel: Hardware name: System manufacturer System Product Name/Pro WS X570-ACE, BIOS 2206 08/13/2020
Aug 26 22:16:34 IXTREME kernel: Call Trace:
Aug 26 22:16:34 IXTREME kernel: <IRQ>
Aug 26 22:16:34 IXTREME kernel: dump_stack+0x64/0x88
Aug 26 22:16:34 IXTREME kernel: nmi_cpu_backtrace.cold+0x14/0x53
Aug 26 22:16:34 IXTREME kernel: ? lapic_can_unplug_cpu.cold+0x3e/0x3e
Aug 26 22:16:34 IXTREME kernel: nmi_trigger_cpumask_backtrace+0xd5/0xd7
Aug 26 22:16:34 IXTREME kernel: rcu_dump_cpu_stacks+0x90/0xbe
Aug 26 22:16:34 IXTREME kernel: rcu_sched_clock_irq.cold+0x1b3/0x3ad
Aug 26 22:16:34 IXTREME kernel: update_process_times+0x5b/0x90
Aug 26 22:16:34 IXTREME kernel: tick_sched_handle+0x22/0x60
Aug 26 22:16:34 IXTREME kernel: tick_sched_timer+0x51/0xb0
Aug 26 22:16:34 IXTREME kernel: ? can_stop_idle_tick+0x90/0x90
Aug 26 22:16:34 IXTREME kernel: __hrtimer_run_queues+0x128/0x2a0
Aug 26 22:16:34 IXTREME kernel: hrtimer_interrupt+0x10e/0x280
Aug 26 22:16:34 IXTREME kernel: smp_apic_timer_interrupt+0x6e/0x130
Aug 26 22:16:34 IXTREME kernel: apic_timer_interrupt+0xf/0x20
Aug 26 22:16:34 IXTREME kernel: </IRQ>
Aug 26 22:16:34 IXTREME kernel: RIP: 0010:zap_leaf_lookup_closest+0xb8/0x170 [zfs]
Aug 26 22:16:34 IXTREME kernel: Code: 8b 87 d8 00 00 00 4c 8b 70 18 8b 87 d0 00 00 00 8d 48 fb 44 89 c8 d3 e0 0f b7 ca 48 8d 0c 49 48 98 48 8d 04 88 49 8d 44 46 30 <48> 8b 48 10 4c 39 c1 76 bc 4c 39 d1 73 5a 8b 68 0c 44 0f b7 50 0a
Aug 26 22:16:34 IXTREME kernel: RSP: 0018:ffffa3c9ed2e7bc0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13
Aug 26 22:16:34 IXTREME kernel: RAX: ffff9598004cc430 RBX: 0000000000000004 RCX: 0000000000000000
Aug 26 22:16:34 IXTREME kernel: RDX: 0000000000000000 RSI: ffffa3c9ed2e7bf0 RDI: ffff95aa9b195a00
Aug 26 22:16:34 IXTREME kernel: RBP: 00000000fcb76346 R08: 1380000000000000 R09: 0000000000000001
Aug 26 22:16:34 IXTREME kernel: R10: 7d77d659f91561db R11: 0000000000000004 R12: ffffa3c9ed2e7c06
Aug 26 22:16:34 IXTREME kernel: R13: 0000000000000000 R14: ffff9598004cc000 R15: ffffa3c9ed2e7d38
Aug 26 22:16:34 IXTREME kernel: fzap_cursor_retrieve+0x103/0x340 [zfs]
Aug 26 22:16:34 IXTREME kernel: zap_cursor_retrieve+0x16c/0x300 [zfs]
Aug 26 22:16:34 IXTREME kernel: zfs_readdir+0x147/0x420 [zfs]
Aug 26 22:16:34 IXTREME kernel: zpl_iterate+0x4c/0x70 [zfs]
Aug 26 22:16:34 IXTREME kernel: iterate_dir+0x143/0x1a0
Aug 26 22:16:34 IXTREME kernel: ksys_getdents64+0x91/0x130
Aug 26 22:16:34 IXTREME kernel: ? __ia32_compat_sys_getdents+0x130/0x130
Aug 26 22:16:34 IXTREME kernel: __x64_sys_getdents64+0x16/0x20
Aug 26 22:16:34 IXTREME kernel: do_syscall_64+0x49/0x90
Aug 26 22:16:34 IXTREME kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Aug 26 22:16:34 IXTREME kernel: RIP: 0033:0x7fba66171a8b
Aug 26 22:16:34 IXTREME kernel: Code: eb b7 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 f3 0f 1e fa 48 81 fa ff ff ff 7f b8 ff ff ff 7f 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 b1 93 0f 00 f7 d8
Aug 26 22:16:34 IXTREME kernel: RSP: 002b:00007ffc4472b468 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
Aug 26 22:16:34 IXTREME kernel: RAX: ffffffffffffffda RBX: 0000562deb7e1980 RCX: 00007fba66171a8b
Aug 26 22:16:34 IXTREME kernel: RDX: 0000000000008000 RSI: 0000562deb7e19b0 RDI: 0000000000000003
Aug 26 22:16:34 IXTREME kernel: RBP: 0000562deb7e19b0 R08: 0000000000008028 R09: 00007fba6626ba60
Aug 26 22:16:34 IXTREME kernel: R10: 0000000000000150 R11: 0000000000000293 R12: ffffffffffffff88
Aug 26 22:16:34 IXTREME kernel: R13: 0000562deb7e1984 R14: 0000000000000000 R15: 0000562deb7e9980
我看到在一些地方提到了 ZFS,所以我假设 ZFS 因某些事情而感到不安。我安装的ZFS版本已经有一段时间没有改变了。我工作得很好(尽管这个问题可能已经被忽视很长时间了!)。在过去的几个月里,LTS 内核进行了一些更新......
我的硬件是一个相当新的系统,大约是 2 个月前构建的: 主板:ASUS Pro WS x570-ACE CPU:AMD Ryzen 7 3800x 内存:Corsair Vengeance LPX 3200MHz 总共 128GB
我用 MemTest86 测试了 RAM(花了 AGES),并且顺利通过,没有错误。
所以我对可能导致这种情况的原因感到有点茫然。数据损坏是否会导致 rm 进程或 ZFS 模块像这样挂起?
这是真正奇怪的部分:该系统中的 ZFS 池每天复制一次到备份计算机。备份机运行FreeNAS。我今天通过 ssh 连接到该计算机并尝试删除相同的 .cache 文件夹,这导致 FreeNAS(或 ZFS)也挂在那里。不过,我只尝试过一次,而且我没有来自该测试的日志数据可分享。
任何帮助将非常感激。
干杯 - 贾斯汀