我们拥有一个由以下机构托管的专用远程服务器赫茨纳。OS 是 Debian 9.4 LAMP(默认安装 Apache、Mysql、PHP)。我们使用服务器来分析大数据,使用 python(一点点)、R(开源 R-studio 服务器)和 Clickhouse(面向列的数据库)。所以,这些是我们在那里唯一使用的东西。
有时候,R 进程占用了太多内存,我们会通过 PID 终止它们。有时我们会使用重启来处理这个问题。但上次启动大型 R 进程后,服务器停止响应(尝试通过 ssh 连接时,它拒绝连接)。我们通过 hetzner robot-panel 重启了它。首先是自动重启。之后我们尝试通过 ssh 或 sftp 连接 - 服务器拒绝连接(或“网络错误。连接被拒绝”是答案)。然后我们命令手动重置硬件 - 但同样没有任何变化。但是,我们可以成功 ping 服务器。
接下来我们激活救援系统这使我们能够在其他操作系统中访问服务器,以便查看日志和其他文件。我们发现最奇怪的事情是 - 第一次重启后,syslog、daemon.log 和 auth.log 尚未更新。只有文件 /var/log/wtmp 在第二次重启后更新。打开时,最后几行是关于两次重启的,其他所有内容都早于第一次重启(在 14:27 25.09 执行)
系统日志的最后几行:
Sep 25 08:48:56 Debian-94-stretch-64-LAMP systemd[17002]: Received SIGRTMIN+24 from PID 28008 (kill).
Sep 25 08:48:56 Debian-94-stretch-64-LAMP systemd[1]: Stopped User Manager for UID 1002.
Sep 25 08:48:56 Debian-94-stretch-64-LAMP systemd[1]: Removed slice User Slice of bor.
Sep 25 08:55:01 Debian-94-stretch-64-LAMP CRON[28072]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 09:05:01 Debian-94-stretch-64-LAMP CRON[28168]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 09:09:01 Debian-94-stretch-64-LAMP CRON[28200]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 25 09:09:20 Debian-94-stretch-64-LAMP systemd[1]: Starting Clean php session files...
Sep 25 09:09:20 Debian-94-stretch-64-LAMP systemd[1]: Started Clean php session files.
Sep 25 09:15:01 Debian-94-stretch-64-LAMP CRON[28320]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 09:17:01 Debian-94-stretch-64-LAMP CRON[28381]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[1]: Created slice User Slice of bor.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[1]: Starting User Manager for UID 1002...
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[1]: Started Session 2480 of user bor.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Listening on GnuPG cryptographic agent (access for web browsers).
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Listening on GnuPG cryptographic agent and passphrase cache.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Listening on GnuPG network certificate management daemon.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Reached target Paths.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Reached target Timers.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Reached target Sockets.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Reached target Basic System.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Reached target Default.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Startup finished in 172ms.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[1]: Started User Manager for UID 1002.
Sep 25 09:25:01 Debian-94-stretch-64-LAMP CRON[28472]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 09:35:01 Debian-94-stretch-64-LAMP CRON[28567]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 09:39:01 Debian-94-stretch-64-LAMP CRON[28610]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 25 09:39:17 Debian-94-stretch-64-LAMP systemd[1]: Starting Clean php session files...
Sep 25 09:39:17 Debian-94-stretch-64-LAMP systemd[1]: Started Clean php session files.
Sep 25 09:40:58 Debian-94-stretch-64-LAMP systemd[1]: Started Session 2485 of user bor.
Sep 25 09:45:01 Debian-94-stretch-64-LAMP CRON[28728]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 09:55:01 Debian-94-stretch-64-LAMP CRON[28844]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 10:01:31 Debian-94-stretch-64-LAMP systemd[1]: Started Session 2488 of user bor.
Sep 25 10:05:03 Debian-94-stretch-64-LAMP CRON[28950]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 10:09:03 Debian-94-stretch-64-LAMP CRON[28986]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 25 10:09:05 Debian-94-stretch-64-LAMP systemd[1]: Starting Clean php session files...
Sep 25 10:09:27 Debian-94-stretch-64-LAMP systemd[1]: Started Clean php session files.
Sep 25 10:15:04 Debian-94-stretch-64-LAMP CRON[29096]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 10:17:04 Debian-94-stretch-64-LAMP CRON[29116]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 25 10:25:01 Debian-94-stretch-64-LAMP CRON[29195]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 10:35:01 Debian-94-stretch-64-LAMP CRON[29286]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 10:39:01 Debian-94-stretch-64-LAMP CRON[29328]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 25 10:39:02 Debian-94-stretch-64-LAMP systemd[1]: Starting Clean php session files...
Sep 25 10:39:03 Debian-94-stretch-64-LAMP systemd[1]: Started Clean php session files.
Sep 25 10:45:01 Debian-94-stretch-64-LAMP CRON[29450]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 10:55:02 Debian-94-stretch-64-LAMP CRON[29549]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 11:01:15 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libcrypto.so.1.0.2: cannot map zero-fill pages
Sep 25 11:04:24 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libcap-ng.so.0: failed to map segment from shared object
Sep 25 11:05:21 Debian-94-stretch-64-LAMP CRON[29619]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 11:05:22 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: /lib/x86_64-linux-gnu/libselinux.so.1: cannot allocate version reference table: Cannot allocate memory
Sep 25 11:06:17 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libgssapi_krb5.so.2: failed to map segment from shared object
Sep 25 11:07:14 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libdl.so.2: failed to map segment from shared object
Sep 25 11:08:11 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libaudit.so.1: failed to map segment from shared object
Sep 25 11:08:17 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libcrypto.so.1.0.2: cannot create shared object descriptor: Cannot allocate memory
Sep 25 11:08:45 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libcrypt.so.1: failed to map segment from shared object
Sep 25 11:09:07 Debian-94-stretch-64-LAMP cron[936]: Authentication failure
Sep 25 11:09:07 Debian-94-stretch-64-LAMP CRON[29635]: Authentication failure
Sep 25 11:09:12 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libpcre.so.3: failed to map segment from shared object
Sep 25 11:09:37 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libcrypt.so.1: cannot map zero-fill pages
Sep 25 11:09:49 Debian-94-stretch-64-LAMP systemd[1]: phpsessionclean.service: Failed to fork: Cannot allocate memory
Sep 25 11:09:50 Debian-94-stretch-64-LAMP systemd[1]: phpsessionclean.service: Failed to run 'start' task: Cannot allocate memory
Sep 25 11:09:50 Debian-94-stretch-64-LAMP systemd[1]: Failed to start Clean php session files.
Sep 25 11:09:54 Debian-94-stretch-64-LAMP systemd[1]: phpsessionclean.service: Unit entered failed state.
Sep 25 11:10:00 Debian-94-stretch-64-LAMP systemd[1]: phpsessionclean.service: Failed with result 'resources'.
Sep 25 11:11:53 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libcrypto.so.1.0.2: failed to map segment from shared object
Sep 25 11:15:03 Debian-94-stretch-64-LAMP cron[936]: (CRON) error (can't fork)
Sep 25 11:17:02 Debian-94-stretch-64-LAMP cron[936]: (CRON) error (can't fork)
Sep 25 11:25:03 Debian-94-stretch-64-LAMP cron[936]: (CRON) error (can't fork)
Sep 25 11:57:02 Debian-94-stretch-64-LAMP systemd[28426]: Time has been changed
Sep 25 11:59:34 Debian-94-stretch-64-LAMP cron[936]: (CRON) error (can't fork)
Sep 25 12:03:49 Debian-94-stretch-64-LAMP systemd[1253]: Time has been changed
Sep 25 12:11:51 Debian-94-stretch-64-LAMP systemd[1253]: Time has been changed
Sep 25 12:18:00 Debian-94-stretch-64-LAMP systemd[28426]: Time has been changed
Sep 25 12:25:13 Debian-94-stretch-64-LAMP systemd[28426]: Time has been changed
Sep 25 12:31:17 Debian-94-stretch-64-LAMP systemd[1253]: Time has been changed
Sep 25 12:35:57 Debian-94-stretch-64-LAMP systemd[28426]: Time has been changed
Sep 25 12:41:15 Debian-94-stretch-64-LAMP systemd[1253]: Time has been changed
Sep 25 12:47:28 Debian-94-stretch-64-LAMP systemd[28426]: Time has been changed
Sep 25 12:57:01 Debian-94-stretch-64-LAMP systemd[1253]: Time has been changed
Sep 25 13:04:57 Debian-94-stretch-64-LAMP systemd[28426]: Time has been changed
Sep 25 13:15:07 Debian-94-stretch-64-LAMP systemd[28426]: Time has been changed
Sep 25 13:27:15 Debian-94-stretch-64-LAMP systemd[1253]: Time has been changed
和 auth.log:
Sep 25 11:02:41 Debian-94-stretch-64-LAMP sshd[29602]: Disconnected from xx.xxx.xxx.xx port 29479 [preauth]
Sep 25 11:02:41 Debian-94-stretch-64-LAMP sshd[29595]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=xx.xxx.xxx.xx user=root
Sep 25 11:02:41 Debian-94-stretch-64-LAMP sshd[29599]: Connection closed by xx.xxx.xxx.xx port 31275 [preauth]
Sep 25 11:02:45 Debian-94-stretch-64-LAMP sshd[29595]: Failed password for root from xx.xxx.xxx.xx port 39659 ssh2
Sep 25 11:02:45 Debian-94-stretch-64-LAMP sshd[29595]: Received disconnect from xx.xxx.xxx.xx port 39659:11: [preauth]
Sep 25 11:02:45 Debian-94-stretch-64-LAMP sshd[29595]: Disconnected from xx.xxx.xxx.xx port 39659 [preauth]
Sep 25 11:04:33 Debian-94-stretch-64-LAMP sshd[29608]: ssh_dispatch_run_fatal: Connection from xx.xxx.xxx.xx port 15191: Broken pipe [preauth]
Sep 25 11:04:33 Debian-94-stretch-64-LAMP sshd[29609]: ssh_dispatch_run_fatal: Connection from xxx.xx.xxx.xxx port 48834: Broken pipe [preauth]
Sep 25 11:05:21 Debian-94-stretch-64-LAMP CRON[29615]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 25 11:05:22 Debian-94-stretch-64-LAMP sshd[29606]: Connection closed by xx.xxx.xxx.xx port 22703 [preauth]
Sep 25 11:05:23 Debian-94-stretch-64-LAMP CRON[29615]: pam_unix(cron:session): session closed for user root
Sep 25 11:07:04 Debian-94-stretch-64-LAMP sshd[29614]: Did not receive identification string from xx.xx.xxx.xx port 36140
Sep 25 11:07:22 Debian-94-stretch-64-LAMP sshd[29622]: Connection closed by xx.xxx.xxx.xx port 34280 [preauth]
Sep 25 11:07:45 Debian-94-stretch-64-LAMP sshd[29629]: error: rexec of /usr/sbin/sshd failed: Cannot allocate memory
Sep 25 11:08:22 Debian-94-stretch-64-LAMP sshd[29629]: Connection closed by xx.xxx.xxx.xx port 19048 [preauth]
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM unable to dlopen(pam_unix.so): /lib/security/pam_unix.so: cannot open shared object file: No such file or directory
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM adding faulty module: pam_unix.so
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM unable to dlopen(pam_permit.so): /lib/security/pam_permit.so: cannot open shared object file: No such file or directory
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM adding faulty module: pam_permit.so
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM unable to dlopen(pam_loginuid.so): /lib/security/pam_loginuid.so: cannot open shared object file: No such file or directory
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM adding faulty module: pam_loginuid.so
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM unable to dlopen(pam_env.so): /lib/security/pam_env.so: cannot open shared object file: No such file or directory
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM adding faulty module: pam_env.so
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM unable to dlopen(pam_limits.so): /lib/security/pam_limits.so: cannot open shared object file: No such file or directory
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM adding faulty module: pam_limits.so
Sep 25 11:10:13 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:10:58 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:11:03 Debian-94-stretch-64-LAMP sshd[29642]: ssh_dispatch_run_fatal: Connection from xx.xxx.xxx.xx port 19637: Broken pipe [preauth]
Sep 25 11:11:31 Debian-94-stretch-64-LAMP sshd[29641]: Connection closed by xx.xxx.xxx.xx port 6478 [preauth]
Sep 25 11:12:53 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:13:03 Debian-94-stretch-64-LAMP sshd[29652]: error: rexec of /usr/sbin/sshd failed: Cannot allocate memory
Sep 25 11:13:30 Debian-94-stretch-64-LAMP sshd[29652]: Connection closed by xx.xxx.xxx.xx port 64145 [preauth]
Sep 25 11:13:51 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:13:52 Debian-94-stretch-64-LAMP sshd[29654]: error: rexec of /usr/sbin/sshd failed: Cannot allocate memory
Sep 25 11:13:55 Debian-94-stretch-64-LAMP sshd[29654]: fatal: fork of unprivileged child failed
Sep 25 11:14:18 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:14:42 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:14:53 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:15:05 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:15:28 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:15:51 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:15:51 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:16:13 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:16:36 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:16:49 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:17:00 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:17:22 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:17:45 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:17:47 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:18:08 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:18:32 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:18:47 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:18:56 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:19:19 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:19:44 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:19:45 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:19:45 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:00 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:07 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:13 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:14 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:29 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:29 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:43 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:45 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:52 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:21:15 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:21:39 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:21:47 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:22:03 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:22:27 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:22:48 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:22:50 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:23:13 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:23:36 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:23:48 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:24:00 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:24:24 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:24:47 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:24:48 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:25:10 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:25:24 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:25:33 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:25:48 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:25:56 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:25:57 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:26:19 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:26:41 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:26:46 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:27:03 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:27:26 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:27:40 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:27:50 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:28:14 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:28:36 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:28:37 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:29:01 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:29:24 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:29:32 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:29:47 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:30:02 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:30:10 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:30:23 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:30:33 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:30:55 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:32:49 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:35:11 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:37:47 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:40:18 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:43:13 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:45:53 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:48:33 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:51:21 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:52:43 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:55:22 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:08:23 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:15:50 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:22:35 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:28:55 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:33:58 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:38:48 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:44:35 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:52:51 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 13:03:08 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 13:10:35 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 13:21:40 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
看来 R 进程过度使用内存,而 sshd 和其他进程无法获得任何内存。但我们无法弄清楚重启后发生了什么,为什么服务器仍然拒绝连接并且不更新日志。
我们执行了快速的 smartctl 磁盘检查(一切正常),并更换了服务器(通过 Hetzner 支持),同时保留了驱动器。因此,这似乎不是硬件问题。
答案1
类似以下的错误在我看来意味着您没有足够的可用 RAM(或可用交换空间)来让进程正常运行。这可能会导致各种意外问题。
error: rexec of /usr/sbin/sshd failed: Cannot allocate memory
...
libcrypto.so.1.0.2: failed to map segment from shared object
...
/lib/x86_64-linux-gnu/libselinux.so.1: cannot allocate version reference table: Cannot allocate memory
尝试运行cat /proc/meminfo
并free
查看有多少可用的 RAM/交换空间。
答案2
经过对问题的长期调查,我们发现文件系统已损坏 - 这就是日志无法更新且我们无法通过 ssh 连接到服务器的原因。