以 root 身份手动运行 logrotate 时成功,但通过 logrotate.service 运行时失败并显示“只读文件系统”

以 root 身份手动运行 logrotate 时成功,但通过 logrotate.service 运行时失败并显示“只读文件系统”

我有一个(以前可以正常工作的)OpenResty logrotate 设置,它是从以前的 Ubuntu 18.04 安装中获取的。然而,logrotate.service 现在失败并出现此错误...

error: error renaming /usr/local/openresty/nginx/logs/access.log.60.zst
to /usr/local/openresty/nginx/logs/access.log.61.zst: Read-only file system

... 我很难理解为什么。新机器运行的是 Ubuntu 20.04,但我不明白为什么在这种情况下这会有所不同。

首先,这是配置:

$ cat /etc/logrotate.d/custom-openresty

/usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
{
  daily
  rotate 60
  maxsize 1G
  missingok
  notifempty
  compress
  compresscmd /usr/bin/zstd
  uncompresscmd /usr/bin/unzstd
  compressoptions -9 --long -T1
  compressext .zst
  delaycompress
  sharedscripts
  postrotate
    test ! -f /usr/local/openresty/nginx/logs/nginx.pid || kill -USR1 `cat /usr/local/openresty/nginx/logs/nginx.pid`
 endscript
}

/etc/logrotate.conf保持不变,如下所示:


# see "man logrotate" for details
# rotate log files weekly
weekly

# use the adm group by default, since this is the owning group
# of /var/log/syslog.
su root adm

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
#dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may be also be configured here.

以下是我的文件的状态(我希望access.log.1在 logrotate 运行后获得):

$ ls -alhg /usr/local/openresty/nginx/logs/
total 10G
drwxr-xr-x  2 root 4.0K Sep 16 20:04 .
drwxr-xr-x 18 root 4.0K Sep 16 19:42 ..
-rw-r--r--  1 root  10G Sep 16 19:56 access.log
-rw-r--r--  1 root 5.5K Sep 16 19:56 error.log

但是,logrotate.service 失败并出现以下错误:

~$ systemctl status logrotate.service 
● logrotate.service - Rotate log files
     Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2020-09-16 20:11:32 UTC; 4min 32s ago
TriggeredBy: ● logrotate.timer
       Docs: man:logrotate(8)
             man:logrotate.conf(5)
    Process: 27403 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
   Main PID: 27403 (code=exited, status=1/FAILURE)

$ sudo journalctl --unit logrotate.service

Sep 16 20:11:32 fetcher-scheduler systemd[1]: Starting Rotate log files...
Sep 16 20:11:32 fetcher-scheduler logrotate[27403]: error: error renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst: Read-only file system
Sep 16 20:11:32 fetcher-scheduler systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Sep 16 20:11:32 fetcher-scheduler systemd[1]: logrotate.service: Failed with result 'exit-code'.
Sep 16 20:11:32 fetcher-scheduler systemd[1]: Failed to start Rotate log files.

当我以 root 身份在调试模式下运行它(不使用 systemd)时,我得到以下输出:

# logrotate -v -d /etc/logrotate.d/custom-openresty

reading config file /etc/logrotate.d/custom-openresty
compress_prog is now /usr/bin/zstd
uncompress_prog is now /usr/bin/unzstd
compress_options is now  -9 --long -T1
compress_ext is now .zst
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
...
Creating new state

Handling 1 logs

rotating pattern: /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
 after 1 days (60 rotations)
empty log files are not rotated, log files >= 1073741824 are rotated earlier, old logs are removed
considering log /usr/local/openresty/nginx/logs/access.log
  Now: 2020-09-16 20:24
  Last rotated at 2020-09-16 20:11
  log needs rotating
considering log /usr/local/openresty/nginx/logs/error.log
  Now: 2020-09-16 20:24
  Last rotated at 2020-09-16 19:00
  log does not need rotating (log has been rotated at 2020-9-16 19:0, that is not day ago yet)
rotating log /usr/local/openresty/nginx/logs/access.log, log->rotateCount is 60
dateext suffix '-20200916'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /usr/local/openresty/nginx/logs/access.log.1 does not exist
renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst (rotatecount 60, logstart 1, i 60), 
renaming /usr/local/openresty/nginx/logs/access.log.59.zst to /usr/local/openresty/nginx/logs/access.log.60.zst (rotatecount 60, logstart 1, i 59), 
...
/logs/access.log.3.zst (rotatecount 60, logstart 1, i 2), 
renaming /usr/local/openresty/nginx/logs/access.log.1.zst to /usr/local/openresty/nginx/logs/access.log.2.zst (rotatecount 60, logstart 1, i 1), 
renaming /usr/local/openresty/nginx/logs/access.log.0.zst to /usr/local/openresty/nginx/logs/access.log.1.zst (rotatecount 60, logstart 1, i 0), 
log /usr/local/openresty/nginx/logs/access.log.61.zst doesn't exist -- won't try to dispose of it
renaming /usr/local/openresty/nginx/logs/access.log to /usr/local/openresty/nginx/logs/access.log.1
running postrotate script
running script with arg /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
: "
    test ! -f /usr/local/openresty/nginx/logs/nginx.pid || kill -USR1 `cat /usr/local/openresty/nginx/logs/nginx.pid`
"

对我来说,一切看起来都很正常。如果我运行它,它也会起作用:

# logrotate -v /etc/logrotate.d/custom-openresty
reading config file /etc/logrotate.d/custom-openresty
compress_prog is now /usr/bin/zstd
uncompress_prog is now /usr/bin/unzstd
compress_options is now  -9 --long -T1
compress_ext is now .zst
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state
...
Creating new state

Handling 1 logs

rotating pattern: /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
 after 1 days (60 rotations)
empty log files are not rotated, log files >= 1073741824 are rotated earlier, old logs are removed
considering log /usr/local/openresty/nginx/logs/access.log
  Now: 2020-09-16 20:26
  Last rotated at 2020-09-16 20:11
  log needs rotating
considering log /usr/local/openresty/nginx/logs/error.log
  Now: 2020-09-16 20:26
  Last rotated at 2020-09-16 19:00
  log does not need rotating (log has been rotated at 2020-9-16 19:0, that is not day ago yet)
rotating log /usr/local/openresty/nginx/logs/access.log, log->rotateCount is 60
dateext suffix '-20200916'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /usr/local/openresty/nginx/logs/access.log.1 does not exist
renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst (rotatecount 60, logstart 1, i 60), 
old log /usr/local/openresty/nginx/logs/access.log.60.zst does not exist
...
old log /usr/local/openresty/nginx/logs/access.log.2.zst does not exist
renaming /usr/local/openresty/nginx/logs/access.log.1.zst to /usr/local/openresty/nginx/logs/access.log.2.zst (rotatecount 60, logstart 1, i 1), 
old log /usr/local/openresty/nginx/logs/access.log.1.zst does not exist
renaming /usr/local/openresty/nginx/logs/access.log.0.zst to /usr/local/openresty/nginx/logs/access.log.1.zst (rotatecount 60, logstart 1, i 0), 
old log /usr/local/openresty/nginx/logs/access.log.0.zst does not exist
log /usr/local/openresty/nginx/logs/access.log.61.zst doesn't exist -- won't try to dispose of it
renaming /usr/local/openresty/nginx/logs/access.log to /usr/local/openresty/nginx/logs/access.log.1
running postrotate script

没有错误,最终access.log.1按预期创建:

# ls -algh /usr/local/openresty/nginx/logs/
total 10G
drwxr-xr-x  2 root 4.0K Sep 16 20:26 .
drwxr-xr-x 18 root 4.0K Sep 16 19:42 ..
-rw-r--r--  1 root  10G Sep 16 19:56 access.log.1
-rw-r--r--  1 root 5.5K Sep 16 19:56 error.log

请注意,它说的是...

log /usr/local/openresty/nginx/logs/access.log.61.zst
doesn't exist -- won't try to dispose of it

... 而不是error renaming ... Read-only file system(正如logrotate.service

为什么以root身份手动运行时可以运行,但通过logrotate.service执行时会失败?

我没有对 做任何更改logrotate.service。为了完整起见,下面是单元文件:

$ systemctl cat logrotate.service
# /lib/systemd/system/logrotate.service
[Unit]
Description=Rotate log files
Documentation=man:logrotate(8) man:logrotate.conf(5)
ConditionACPower=true

[Service]
Type=oneshot
ExecStart=/usr/sbin/logrotate /etc/logrotate.conf

# performance options
Nice=19
IOSchedulingClass=best-effort
IOSchedulingPriority=7

# hardening options
#  details: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
#  no ProtectHome for userdir logs
#  no PrivateNetwork for mail deliviery
#  no ProtectKernelTunables for working SELinux with systemd older than 235
#  no MemoryDenyWriteExecute for gzip on i686
PrivateDevices=true
PrivateTmp=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectSystem=full
RestrictRealtime=true

现在我已经没有选择了。任何帮助解决问题的帮助都将不胜感激。

答案1

我觉得我找到了。logrotate 文件不是问题所在。

相反,这是由 systemd 单元文件中的强化功能引起的。在我禁用ProtectSystem=full强化选项后,它就起作用了。原因是 logrotate 必须对/usr目录进行操作,如果启用该选项,则该目录为只读。

来自文档ProtectSystem=

采用布尔参数或特殊值“full”或“strict”。如果为真,则将 /usr 和引导加载程序目录(/boot 和 /efi)挂载为该单元调用的进程的只读形式。如果设置为“full”,/etc 目录也会以只读方式挂载。如果设置为“strict”,则整个文件系统层次结构都会以只读方式挂载,但 API 文件系统子树 /dev、/proc 和 /sys 除外(使用 PrivateDevices=、ProtectKernelTunables=、ProtectControlGroups= 保护这些目录)。此设置可确保禁止对服务进行任何供应商提供的操作系统(以及可选的配置和本地挂载)的修改。建议为所有长期运行的服务启用此设置,除非它们涉及系统更新或需要以其他方式修改操作系统。如果使用此选项,则可以使用 ReadWritePaths= 排除特定目录的只读设置。如果设置了 DynamicUser=,则隐含此设置。此设置无法确保在所有情况下都受到保护。通常,它具有与 ReadOnlyPaths= 相同的限制,请参见下文。默认为关闭。

为了正确修复它,我现在添加了以下行/lib/systemd/system/logrotate.service

ReadWritePaths=/usr/local/openresty/nginx/logs

然后ProtectSystem=full将除日志所在目录之外的所有内容都挂载为只读。现在,只读错误消失了:

# systemctl daemon-reload && systemctl start logrotate
(completed successfully without output)

我猜想,我从未在 Ubuntu 18.04 上遇到过这个问题,因为强化选项还不是默认选项。Ubuntu 20.04 现在有了这个功能,但我不知道它们是在哪个 Ubuntu 版本中首次引入的。

答案2

很高兴找到我遇到的相同问题。编辑 nginx.conf 以记录到不同的目录(例如公共的 /var/log/nginx)也是一种可行的替代方法。

相关内容