关机时卸载逻辑卷失败

关机时卸载逻辑卷失败

/etc/fstab在我的系统中, systemd 启动后会创建、挂载多个逻辑卷并添加条目。

当系统第一次重新启动时,关机期间会观察到许多卸载失败的情况。

有趣的是,在后续的重启中,并没有发现这个问题。

首次重启和后续重启之间的行为差​​异可能是由于仅在首次重启期间,碎片*.mount不是由 生成的systemd-fstab-generator。我特别想修复这个问题,因为/var/log我在这里面临数据丢失。

按照这里,我添加了覆盖配置如下systemd-journal-flush.service

[Unit]
PartOf=var.mount var-log.mount

[Service]
ExecStop=/usr/local/bin/debugger.sh

此外,在 ExecStop 中添加了一个调试脚本来检查是否有任何打开的 fds。

#!/bin/sh
#
date > /root/out
echo "=============" >> /root/out
fuser -m /var/log >> /root/out
echo "=============" >> /root/out
lsof | grep /var/log >> /root/out
echo "===============" >> /root/out
ps aux  >> /root/out

经过此更改后,umount 在第一次重启时会失败,但在后续重启时可以正常使用。

[FAILED] Failed unmounting /var/crash/kernel.
[FAILED] Failed unmounting /var/lib/systemd/coredump.
[FAILED] Failed unmounting /var/log/audit.
[FAILED] Failed unmounting /var/sonic.
[FAILED] Failed unmounting /var/tmp.
[  OK  ] Stopped target Swap.
         Unmounting /mnt/sftp...
         Unmounting /var/crash...
[FAILED] Failed unmounting /var/crash.
[  OK  ] Stopped Flush Journal to Persistent Storage.
         Unmounting /var/log...
[FAILED] Failed unmounting /var/log.

从调试脚本输出来看,lsof和输出都是空的。我在输出中也fuser没有发现任何可疑的东西。(状态中有一个。不确定是否需要检查。)psumountD

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1 27.4  0.2 188048 37548 ?        Ss   15:43   5:26 /usr/lib/systemd/systemd --show-status=1 --log-target=journal-or-kmsg --log-level=notice
root           2  0.0  0.0      0     0 ?        S    15:43   0:00 [kthreadd]
root           3  0.0  0.0      0     0 ?        I<   15:43   0:00 [rcu_gp]
root           4  0.0  0.0      0     0 ?        I<   15:43   0:00 [rcu_par_gp]
root           5  0.0  0.0      0     0 ?        I    15:43   0:00 [kworker/0:0-cgroup_destroy]
root           6  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/0:0H-kblockd]
root           8  0.0  0.0      0     0 ?        I<   15:43   0:00 [mm_percpu_wq]
root           9  0.0  0.0      0     0 ?        S    15:43   0:00 [ksoftirqd/0]
root          10  0.2  0.0      0     0 ?        I    15:43   0:02 [rcu_sched]
root          11  0.0  0.0      0     0 ?        I    15:43   0:00 [rcu_bh]
root          12  0.0  0.0      0     0 ?        S    15:43   0:00 [migration/0]
root          13  0.0  0.0      0     0 ?        S    15:43   0:00 [cpuhp/0]
root          14  0.0  0.0      0     0 ?        S    15:43   0:00 [cpuhp/1]
root          15  0.0  0.0      0     0 ?        S    15:43   0:00 [migration/1]
root          16  0.0  0.0      0     0 ?        S    15:43   0:00 [ksoftirqd/1]
root          18  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/1:0H-kblockd]
root          19  0.0  0.0      0     0 ?        S    15:43   0:00 [cpuhp/2]
root          20  0.0  0.0      0     0 ?        S    15:43   0:00 [migration/2]
root          21  0.0  0.0      0     0 ?        S    15:43   0:00 [ksoftirqd/2]
root          23  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/2:0H-kblockd]
root          24  0.0  0.0      0     0 ?        S    15:43   0:00 [cpuhp/3]
root          25  0.0  0.0      0     0 ?        S    15:43   0:00 [migration/3]
root          26  0.0  0.0      0     0 ?        S    15:43   0:00 [ksoftirqd/3]
root          28  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/3:0H-kblockd]
root          29  0.0  0.0      0     0 ?        S    15:43   0:00 [cpuhp/4]
root          30  0.0  0.0      0     0 ?        S    15:43   0:00 [migration/4]
root          31  0.0  0.0      0     0 ?        S    15:43   0:00 [ksoftirqd/4]
root          33  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/4:0H-kblockd]
root          34  0.0  0.0      0     0 ?        S    15:43   0:00 [cpuhp/5]
root          35  0.0  0.0      0     0 ?        S    15:43   0:00 [migration/5]
root          36  0.0  0.0      0     0 ?        S    15:43   0:00 [ksoftirqd/5]
root          38  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/5:0H-kblockd]
root          39  0.0  0.0      0     0 ?        S    15:43   0:00 [cpuhp/6]
root          40  0.0  0.0      0     0 ?        S    15:43   0:00 [migration/6]
root          41  0.0  0.0      0     0 ?        S    15:43   0:00 [ksoftirqd/6]
root          43  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/6:0H-kblockd]
root          44  0.0  0.0      0     0 ?        S    15:43   0:00 [cpuhp/7]
root          45  0.0  0.0      0     0 ?        S    15:43   0:00 [migration/7]
root          46  0.0  0.0      0     0 ?        S    15:43   0:00 [ksoftirqd/7]
root          48  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/7:0H-kblockd]
root          49  0.0  0.0      0     0 ?        S    15:43   0:00 [kdevtmpfs]
root          50  0.0  0.0      0     0 ?        I<   15:43   0:00 [netns]
root          51  0.0  0.0      0     0 ?        S    15:43   0:00 [kauditd]
root          53  0.0  0.0      0     0 ?        S    15:43   0:00 [khungtaskd]
root          54  0.0  0.0      0     0 ?        S    15:43   0:00 [oom_reaper]
root          55  0.0  0.0      0     0 ?        I<   15:43   0:00 [writeback]
root          56  0.0  0.0      0     0 ?        S    15:43   0:00 [kcompactd0]
root          57  0.0  0.0      0     0 ?        SN   15:43   0:00 [ksmd]
root          58  0.0  0.0      0     0 ?        I<   15:43   0:00 [crypto]
root          59  0.0  0.0      0     0 ?        I<   15:43   0:00 [kblockd]
root          60  0.0  0.0      0     0 ?        I<   15:43   0:00 [ata_sff]
root          61  0.0  0.0      0     0 ?        I<   15:43   0:00 [md]
root          62  0.0  0.0      0     0 ?        I<   15:43   0:00 [devfreq_wq]
root          63  0.0  0.0      0     0 ?        S    15:43   0:00 [watchdogd]
root          65  0.0  0.0      0     0 ?        S    15:43   0:00 [kswapd0]
root          73  0.0  0.0      0     0 ?        I<   15:43   0:00 [kthrotld]
root          74  0.0  0.0      0     0 ?        S    15:43   0:00 [irq/30-pciehp]
root          76  0.0  0.0      0     0 ?        I    15:43   0:00 [kworker/1:1-memcg_kmem_cache]
root          77  0.0  0.0      0     0 ?        I    15:43   0:01 [kworker/3:1-events]
root          78  0.0  0.0      0     0 ?        I    15:43   0:00 [kworker/4:1-cgroup_destroy]
root          79  0.0  0.0      0     0 ?        I    15:43   0:00 [kworker/5:1-events]
root          80  0.0  0.0      0     0 ?        I    15:43   0:01 [kworker/6:1-memcg_kmem_cache]
root          82  0.0  0.0      0     0 ?        I<   15:43   0:00 [acpi_thermal_pm]
root          84  0.0  0.0      0     0 ?        I<   15:43   0:00 [tpm-vtpm]
root          85  0.0  0.0      0     0 ?        S    15:43   0:00 [scsi_eh_0]
root          86  0.0  0.0      0     0 ?        I<   15:43   0:00 [scsi_tmf_0]
root          87  0.0  0.0      0     0 ?        S    15:43   0:00 [scsi_eh_1]
root          88  0.0  0.0      0     0 ?        I<   15:43   0:00 [scsi_tmf_1]
root          89  0.0  0.0      0     0 ?        S    15:43   0:00 [scsi_eh_2]
root          90  0.0  0.0      0     0 ?        I<   15:43   0:00 [scsi_tmf_2]
root          91  0.0  0.0      0     0 ?        S    15:43   0:00 [scsi_eh_3]
root          92  0.0  0.0      0     0 ?        I<   15:43   0:00 [scsi_tmf_3]
root          94  0.0  0.0      0     0 ?        I    15:43   0:00 [kworker/u16:3-flush-252:18]
root          97  0.0  0.0      0     0 ?        I<   15:43   0:00 [ipv6_addrconf]
root          99  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/1:1H-kblockd]
root         118  0.0  0.0      0     0 ?        I<   15:43   0:00 [ixgbe]
root         126  0.0  0.0      0     0 ?        I    15:43   0:00 [kworker/6:2-events]
root         133  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/2:1H-kblockd]
root         135  0.0  0.0      0     0 ?        I<   15:43   0:00 [kdmflush]
root         138  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/4:1H-kblockd]
root         139  0.0  0.0      0     0 ?        S    15:43   0:00 [jbd2/dm-0-8]
root         140  0.0  0.0      0     0 ?        I<   15:43   0:00 [ext4-rsv-conver]
root         142  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/0:1H-kblockd]
root         145  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/6:1H-kblockd]
root         147  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/7:1H-events_highpri]
root         158  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/3:1H-kblockd]
root         166  0.2  0.3  80384 57352 ?        Ss   15:43   0:02 /usr/lib/systemd/systemd-journald
root         171  0.0  0.0      0     0 ?        I<   15:43   0:00 [kworker/5:1H-kblockd]
root         172  0.0  0.0      0     0 ?        I<   15:43   0:00 [ipmi-msghandler]
root         173  0.0  0.0      0     0 ?        I    15:43   0:00 [kworker/2:2-events_power_efficient]
root         183  0.2  0.0   7164  5260 ?        Ss   15:43   0:02 /usr/lib/systemd/systemd-udevd
root         227  0.0  0.0      0     0 ?        S    15:43   0:00 [scsi_eh_4]
root         228  0.0  0.0      0     0 ?        I<   15:43   0:00 [scsi_tmf_4]
root         231  0.0  0.0      0     0 ?        S    15:43   0:00 [usb-storage]
root         237  0.0  0.0      0     0 ?        I<   15:43   0:00 [ttm_swap]
root         272  0.0  0.0      0     0 ?        SN   15:43   0:00 [kipmi0]
root       68631  0.0  0.0      0     0 ?        I    15:48   0:00 [kworker/3:0-cgroup_destroy]
root       68632  0.0  0.0      0     0 ?        I    15:48   0:00 [kworker/1:2-mm_percpu_wq]
root       95828  0.1  0.0      0     0 ?        I    15:51   0:00 [kworker/2:0-cgroup_destroy]
root       98121  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root       98348  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root       98522  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root       98540  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root       98644  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root       98816  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root       98879  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root       98912  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root       99063  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root       99098  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root       99132  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root       99450  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root       99503  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root       99536  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root       99856  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root      100023  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root      100049  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root      100153  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root      100189  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root      100244  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root      100279  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root      100316  0.0  0.0      0     0 ?        I<   15:51   0:00 [kdmflush]
root      101223  0.2  0.0      0     0 ?        I    15:51   0:01 [kworker/u16:1-events_unbound]
root      101857  0.0  0.0      0     0 ?        S    15:51   0:00 [jbd2/dm-2-8]
root      101858  0.0  0.0      0     0 ?        I<   15:51   0:00 [ext4-rsv-conver]
root      101864  0.0  0.0      0     0 ?        S    15:51   0:00 [jbd2/dm-3-8]
root      101865  0.0  0.0      0     0 ?        I<   15:51   0:00 [ext4-rsv-conver]
root      101872  0.0  0.0      0     0 ?        S    15:51   0:00 [jbd2/dm-4-8]
root      101873  0.0  0.0      0     0 ?        I<   15:51   0:00 [ext4-rsv-conver]
root      101946  0.0  0.0      0     0 ?        S    15:51   0:00 [jbd2/dm-14-8]
root      101947  0.0  0.0      0     0 ?        I<   15:51   0:00 [ext4-rsv-conver]
root      101953  0.0  0.0      0     0 ?        S    15:51   0:00 [jbd2/dm-15-8]
root      101954  0.0  0.0      0     0 ?        I<   15:51   0:00 [ext4-rsv-conver]
root      101960  0.0  0.0      0     0 ?        S    15:51   0:00 [jbd2/dm-16-8]
root      101961  0.0  0.0      0     0 ?        I<   15:51   0:00 [ext4-rsv-conver]
root      101981  0.0  0.0      0     0 ?        S    15:51   0:00 [jbd2/dm-20-8]
root      101982  0.0  0.0      0     0 ?        I<   15:51   0:00 [ext4-rsv-conver]
root      101988  0.0  0.0      0     0 ?        S    15:51   0:00 [jbd2/dm-22-8]
root      101989  0.0  0.0      0     0 ?        I<   15:51   0:00 [ext4-rsv-conver]
root      101995  0.0  0.0      0     0 ?        S    15:51   0:00 [jbd2/sda3-8]
root      101996  0.0  0.0      0     0 ?        I<   15:51   0:00 [ext4-rsv-conver]
root      102071  0.1  0.0      0     0 ?        I    15:51   0:00 [kworker/7:0-events]
root      102103  0.0  0.0      0     0 ?        I    15:51   0:00 [kworker/5:0-rcu_gp]
root      102283  0.1  0.0      0     0 ?        I    15:51   0:00 [kworker/4:0-memcg_kmem_cache]
root      102829  0.0  0.0      0     0 ?        I    15:52   0:00 [kworker/7:3-cgroup_pidlist_destroy]
root      102941  0.0  0.0      0     0 ?        I    15:52   0:00 [kworker/0:1-rcu_gp]
root      116853  0.0  0.0      0     0 ?        I    15:59   0:00 [kworker/u16:0-events_unbound]
root      116965  0.0  0.0      0     0 ?        I    15:59   0:00 [kworker/1:0-mm_percpu_wq]
root      118574  0.0  0.0      0     0 ?        I    16:02   0:00 [kworker/u16:2-flush-252:22]
root      118585  0.0  0.0   7164  3708 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118586  0.0  0.0   7164  3644 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118587  0.0  0.0   7164  3640 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118588  0.0  0.0   7164  3640 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118589  0.5  0.0   7164  3640 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118590  0.0  0.0   7164  3640 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118591  0.0  0.0   7164  3640 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118592  0.0  0.0   7164  3640 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118593  0.0  0.0   7164  3640 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118594  0.0  0.0   7164  3640 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118595  0.0  0.0   7164  3640 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118596  0.0  0.0   7164  3620 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118597  0.0  0.0   7164  3640 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118598  0.0  0.0   7164  3640 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118599  0.0  0.0   7164  3640 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118600  0.0  0.0   7164  3640 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118601  1.6  0.0   7164  3780 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118603  0.0  0.0   7164  3640 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118604  0.0  0.0   7164  3528 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118605  0.0  0.0   7164  3528 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118606  0.0  0.0   7164  3528 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118607  0.0  0.0   7164  3528 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118608  0.0  0.0   7164  3528 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118609  0.0  0.0   7164  3528 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118610  0.0  0.0   7164  3528 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118611  0.0  0.0   7164  3528 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118612  0.0  0.0   7164  3528 ?        S    16:02   0:00 /usr/lib/systemd/systemd-udevd
root      118640  0.0  0.0      0     0 ?        I    16:02   0:00 [kworker/3:2-cgroup_destroy]
root      118731  0.0  0.0      0     0 ?        I    16:02   0:00 [kworker/3:3-events]
root      118745  0.0  0.0      0     0 ?        I    16:02   0:00 [kworker/4:2-cgroup_destroy]
root      118823  0.0  0.0      0     0 ?        I    16:02   0:00 [kworker/7:1-cgroup_destroy]
root      118898  0.0  0.0      0     0 ?        I    16:02   0:00 [kworker/0:2]
root      118899  0.2  0.0      0     0 ?        I    16:02   0:00 [kworker/0:3-events]
root      119014  0.0  0.0      0     0 ?        I    16:02   0:00 [kworker/6:0-mm_percpu_wq]
root      119052  0.0  0.0      0     0 ?        I    16:02   0:00 [kworker/2:1-events]
root      119111  0.0  0.0   6136   932 ?        R    16:02   0:00 /bin/umount /var/lib/docker -c
root      119117  0.0  0.0   6808  2936 ?        Ss   16:02   0:00 /bin/sh /usr/local/bin/debugger.sh
root      119120  0.0  0.0   6136   876 ?        D    16:02   0:00 /bin/umount /mnt/sftp -c
root      119124  0.0  0.0   6132  2036 ?        R    16:02   0:00 ps aux

我的问题是:

  1. 我是否遗漏了运行时添加条目时所需的任何优雅处理/etc/fstab。有什么具体方法可以通知 systemd 创建*.mountsystemd-fsck@*分段?
  2. 如果没有进程正在使用挂载点,还有什么原因会导致 umount 失败?如何进一步调试?
  3. 有什么方法可以解决数据丢失问题?添加另一项服务来执行延迟/强制卸载会有帮助吗?

答案1

感谢@Prem 指出卸载期间必须保持反向顺序。

可以通过将责任委托给 systemd 来解决该问题。systemd systemd-fstab-generator能够处理子目录的层次结构并按所需的顺序进行挂载。

systemd-generator 手册页

编辑 /etc/fstab 后,用户应调用 systemctl daemon-reload。这将重新运行所有生成器并导致 systemd 从磁盘重新加载单元。要实际挂载添加到 fstab 的新目录,可以使用 systemctl start /path/to/mountpoint 或 systemctl start local-fs.target。

我做的完全一样,问题解决了!

相关内容