非常奇怪的依赖关系汇合导致了奇怪的数据库崩溃。上下文:
- CentOS 7.9
- LVM2-2.02
- Postgresql 12,数据卷位于 LVM 卷上,XFS 格式
- systemd
- 总线
当系统启动稳定,数据库正常运行时,我执行了yum update
。在更新过程中,卸载了几个卷,包括安装了数据库的卷。这导致了恐慌消息:
2021-08-12 21:12:35.418 CEST PANIC: could not open file ""/var/lib/pgsql/current/data/global/pg_control"": No such file or directory
journald 日志显示,在此次崩溃之前的几分钟内,有不少软件包已更新。没有来自 postgresql 的中间日志消息,因此可能缩小范围。
显然某些/etc/fstab
配置不正确,但在 yum 更新之前很久就可以看到以下消息:
Aug 12 18:27:32 git.uibk.ac.at systemd-fstab-generator[19273]: Checking was requested for "none", but it is not a device.
并且在更新过程中也经常出现。所以我认为这只是一种转移注意力的手段。
Systemd 更新并重启:
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Reexecuting.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
这是事故发生前 45 秒的情况。紧接着,我们看到:
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: [/etc/systemd/system/rsyslog.service.d/partof-journald.conf:2] Assignment outside of section. Ignoring.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Stopped target Remote File Systems.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Stopping Remote File Systems.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Unset automount mnt-git\x2drepodata.automount.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Stopping mnt-git\x2drepodata.automount.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Set up automount mnt-git\x2drepodata.automount.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Reached target Remote File Systems.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Unset automount Arbitrary Executable File Formats File System Automount Point.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Stopping Arbitrary Executable File Formats File System Automount Point.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Stopped target Local File Systems.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Stopping Local File Systems.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Unset automount var-lib-pgsql.automount.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Stopping var-lib-pgsql.automount.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: var-lib-pgsql.automount: Directory /var/lib/pgsql to mount over is not empty, mounting anyway.
为什么....为什么?....为什么!!!无论如何,它似乎没有正常工作。
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Set up automount var-lib-pgsql.automount.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Mounting /var/lib/pgsql/current/pg_stat_tmp...
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Mounted /var/lib/pgsql/current/pg_stat_tmp.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Reached target Local File Systems.
该卷也无法很快被安装,必须在恢复过程中手动重新安装。
在 PANIC 之前还出现了一些其他消息:
Aug 12 21:12:32 git.uibk.ac.at yum[12642]: Installed: kernel-3.10.0-1160.36.2.el7.x86_64
Aug 12 21:12:33 git.uibk.ac.at yum[12642]: Updated: 2:vim-enhanced-7.4.629-8.el7_9.x86_64
Aug 12 21:12:33 git.uibk.ac.at systemd[1]: Reloading.
Aug 12 21:12:34 git.uibk.ac.at yum[12642]: Updated: 7:lvm2-2.02.187-6.el7_9.5.x86_64
Aug 12 21:12:34 git.uibk.ac.at yum[12642]: Updated: 1:nfs-utils-1.3.0-0.68.el7.1.x86_64
然后,奇怪的是,这两个的顺序不正确:
Aug 12 21:12:35 git.uibk.ac.at systemd[1]: postgresql-12.service: main process exited, code=exited, status=1/FAILURE
Aug 12 21:12:35 git.uibk.ac.at kernel: XFS (dm-6): Unmounting Filesystem
dm-6 是现在(重启后)交换卷,那么……?
以下是卷数列表(不包括常见的卷数):
/dev/mapper/vgdata-git--db on /var/lib/pgsql type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vgroot-lvroot on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vgroot-lvtmp on /tmp type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vgroot-lvvar on /var type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/vda1 on /boot type ext4 (rw,relatime,seclabel,data=ordered)
nfsshare:/git/qt-gitlab on /mnt/git-repodata type nfs (rw,relatime,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountport=635,mountproto=udp,local_lock=none)
none on /var/lib/pgsql/12/pg_stat_tmp type tmpfs (rw,relatime,seclabel,size=16384k,mode=770,uid=102,gid=26)
systemd-1 on /mnt/git-repodata type autofs (rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13188)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13146)
systemd-1 on /var/lib/pgsql type autofs (rw,relatime,fd=41,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13753)
我从未在任何现代 Linux 系统上见过这种情况。这是 systemd 在更新或重新执行时的“预期行为”吗?