我有一台 Ubuntu 18 服务器。我创建了少量持久的 docker 容器,但自从我创建了 docker 容器后,服务器就开始锁定,并出现错误“无法启动日志服务”。
我不是 Linux 专家,所以我不知道从哪里开始调试和解决这个问题。
重新启动一段时间后即可恢复。
[更新 1] 大约在 UTC 时间 8 月 28 日 00:00,我注意到服务器已被锁定或至少没有响应,我无法登录,并且容器已停止工作。
查看后/var/log/syslog
,/var/log/syslog.1
我看到了与该期刊相关的以下内容。
Aug 27 13:00:42 lba-build7 kernel: [87433.224929] systemd[1]: systemd-journald.service: State 'stop-sigabrt' timed out. Terminating.
Aug 27 13:02:44 lba-build7 kernel: [87553.398991] systemd[1]: systemd-journald.service: State 'stop-sigterm' timed out. Killing.
Aug 27 13:02:59 lba-build7 kernel: [87553.709900] systemd[1]: systemd-journald.service: Killing process 408 (systemd-journal) with signal SIGKILL.
Aug 27 13:03:28 lba-build7 kernel: [87599.008763] systemd[1]: systemd-journald.service: Main process exited, code=killed, status=9/KILL
Aug 27 13:03:32 lba-build7 kernel: [87599.444377] systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
Aug 27 13:03:54 lba-build7 kernel: [87624.084710] systemd[1]: systemd-journald.service: Service has no hold-off time, scheduling restart.
Aug 27 13:04:01 lba-build7 kernel: [87624.909606] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
Aug 27 13:04:13 lba-build7 kernel: [87625.769139] systemd[1]: Stopped Flush Journal to Persistent Storage.
Aug 27 13:04:17 lba-build7 kernel: [87625.826840] systemd[1]: Stopping Flush Journal to Persistent Storage...
Aug 27 13:04:24 lba-build7 kernel: [87625.826866] systemd[1]: Stopped Journal Service.
Aug 27 13:04:24 lba-build7 kernel: [87627.321655] systemd[1]: Starting Journal Service...
Aug 27 13:05:27 lba-build7 kernel: [87717.562938] systemd[1]: systemd-journald.service: Start operation timed out. Terminating.
Aug 27 13:05:51 lba-build7 kernel: [87740.615728] systemd[1]: systemd-journald.service: Failed with result 'timeout'.
Aug 27 13:06:00 lba-build7 kernel: [87748.110133] systemd[1]: Failed to start Journal Service.
Aug 27 13:06:05 lba-build7 kernel: [87749.423828] systemd[1]: Dependency failed for Flush Journal to Persistent Storage.
Aug 27 13:06:13 lba-build7 kernel: [87749.423928] systemd[1]: systemd-journal-flush.service: Job systemd-journal-flush.service/start failed with result 'dependency'.
Aug 27 13:06:41 lba-build7 kernel: [87790.379406] systemd[1]: systemd-journald.service: Service has no hold-off time, scheduling restart.
Aug 27 13:06:50 lba-build7 kernel: [87790.521642] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 2.
Aug 27 13:06:54 lba-build7 kernel: [87791.303309] systemd[1]: Stopped Journal Service.
Aug 27 13:07:00 lba-build7 kernel: [87810.558459] systemd[1]: Starting Journal Service...
Aug 27 13:08:33 lba-build7 kernel: [87900.805094] systemd[1]: systemd-journald.service: Start operation timed out. Terminating.
Aug 27 13:08:50 lba-build7 kernel: [87921.130932] systemd[1]: systemd-journald.service: Failed with result 'timeout'.
Aug 27 13:09:04 lba-build7 kernel: [87934.989109] systemd[1]: Failed to start Journal Service.
Aug 27 13:14:04 lba-build7 kernel: [88233.039283] systemd[1]: systemd-journald.service: Service has no hold-off time, scheduling restart.
Aug 27 13:14:13 lba-build7 kernel: [88237.774972] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 3.
Aug 27 13:24:33 lba-build7 kernel: [88860.911182] systemd[1]: Starting Message of the Day...
Aug 27 13:24:55 lba-build7 kernel: [88865.087182] systemd[1]: Stopped Journal Service.
Aug 27 13:28:26 lba-build7 kernel: [89095.395937] systemd[1]: Starting Journal Service...
Aug 27 21:02:50 lba-build7 kernel: [116350.962791] systemd-journald[37743]: File /var/log/journal/3c460b60a2194f24954d45a0036e3ce7/system.journal corrupted or uncleanly shut down, renaming and replacing.
Aug 28 00:07:05 lba-build7 systemd[1]: Started Create Static Device Nodes in /dev.
Aug 28 00:07:05 lba-build7 kernel: [ 0.000000] Linux version 4.15.0-58-generic (buildd@lcy01-amd64-013) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 (Ubuntu 4.15.0-58.64-generic 4.15.18)
Aug 28 00:07:05 lba-build7 systemd-modules-load[406]: Inserted module 'iscsi_tcp'
Aug 28 00:07:05 lba-build7 kernel: [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-58-generic root=UUID=41010336-2673-11e9-bc9b-00155d0a9618 ro
Aug 28 00:07:05 lba-build7 systemd-modules-load[406]: Inserted module 'ib_iser'
Aug 28 00:07:05 lba-build7 kernel: [ 0.000000] KERNEL supported cpus:
...
Aug 28 00:07:05 lba-build7 kernel: [ 18.015549] Loading iSCSI transport class v2.0-870.
Aug 28 00:07:05 lba-build7 kernel: [ 18.207814] iscsi: registered transport (tcp)
Aug 28 00:07:05 lba-build7 kernel: [ 18.336790] iscsi: registered transport (iser)
Aug 28 00:07:05 lba-build7 kernel: [ 18.421551] EXT4-fs (sda2): re-mounted. Opts: (null)
Aug 28 00:07:05 lba-build7 kernel: [ 18.713433] systemd-journald[403]: Received request to flush runtime journal from PID 1
Aug 28 00:07:05 lba-build7 kernel: [ 18.842182] Adding 3993596k swap on /swap.img. Priority:-2 extents:5 across:4280316k FS
Aug 28 00:07:05 lba-build7 kernel: [ 19.178382] systemd-journald[403]: File /var/log/journal/3c460b60a2194f24954d45a0036e3ce7/system.journal corrupted or uncleanly shut down, renaming and replacing.
Aug 28 00:07:05 lba-build7 kernel: [ 19.622637] hv_vmbus: registering driver hv_balloon
Aug 28 00:07:05 lba-build7 kernel: [ 19.623130] hv_balloon: Using Dynamic Memory protocol version 2.0
还有很多其他的东西,但不确定什么是重要的,甚至不确定日记是原因还是症状。
[更新2]此服务器作为 Hyper-V 上的 VM 运行