我有一个 nginx 实例,设置为将访问记录到 /var/log/nginx/access.log,将错误记录到 /var/log/nginx/errors.log,但是每周运行 logrotate 时,文件就会移动到 *.log.1,并创建新的 *.log 文件,但 nginx 仍会将日志记录到 log.1 文件而不是新的 .log 文件(并且不会对任何内容进行 gzip 压缩)。我第一次注意到这个问题时,距离日志轮换已经过去了 3 周,日志变得越来越大。运行后,kill -HUP `cat /run/nginx.pid`
nginx 又开始将日志记录到正确的位置,但问题在下一周再次出现。
令人沮丧的更重要的原因是,我已将日志设置为通过 rsyslog 上传到 Loggly,并且当 nginx 停止记录到文件时,我进行了 rsyslog 轮询,然后内容停止上传并且我没有收到任何警报。
我怀疑这与重新启动 nginx 或重新加载配置有关,因为直到我更改配置并以我认为正常的方式重新加载配置后,它才启动。我尝试运行,kill -USR1 `cat /run/nginx.pid`
但文件继续记录到错误的位置,直到我运行kill -HUP `cat /run/nginx.pid`
,我已经知道这并不能解决问题。
知道发生了什么吗?我承认我不是 logrotate 或 nginx 管理方面的专家,但我的 Google 搜索结果却让我失望了。
这是我的 nginx logrotate 脚本,如果您还想看其他内容,请告诉我。除了定义输出位置外,nginx.conf 在日志记录方面没有什么特别之处。
/var/log/nginx/*.log {
weekly
missingok
rotate 52
compress
delaycompress
notifempty
create 0640 www-data adm
sharedscripts
prerotate
if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
run-parts /etc/logrotate.d/httpd-prerotate; \
fi \
endscript
postrotate
[ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
endscript
}
编辑:我想我找到了问题所在。以下是在调试模式下运行 logrotate 的输出:
$ sudo logrotate --force -d /etc/logrotate.d/nginx
reading config file /etc/logrotate.d/nginx
Handling 1 logs
rotating pattern: /var/log/nginx/*.log forced from command line (52 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/nginx/access.log
log needs rotating
considering log /var/log/nginx/error.log
log needs rotating
rotating log /var/log/nginx/access.log, log->rotateCount is 52
dateext suffix '-20141023'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /var/log/nginx/access.log.1 does not exist
renaming /var/log/nginx/access.log.52.gz to /var/log/nginx/access.log.53.gz (rotatecount 52, logstart 1, i 52),
renaming /var/log/nginx/access.log.51.gz to /var/log/nginx/access.log.52.gz (rotatecount 52, logstart 1, i 51),
renaming /var/log/nginx/access.log.50.gz to /var/log/nginx/access.log.51.gz (rotatecount 52, logstart 1, i 50),
renaming /var/log/nginx/access.log.49.gz to /var/log/nginx/access.log.50.gz (rotatecount 52, logstart 1, i 49),
renaming /var/log/nginx/access.log.48.gz to /var/log/nginx/access.log.49.gz (rotatecount 52, logstart 1, i 48),
renaming /var/log/nginx/access.log.47.gz to /var/log/nginx/access.log.48.gz (rotatecount 52, logstart 1, i 47),
renaming /var/log/nginx/access.log.46.gz to /var/log/nginx/access.log.47.gz (rotatecount 52, logstart 1, i 46),
renaming /var/log/nginx/access.log.45.gz to /var/log/nginx/access.log.46.gz (rotatecount 52, logstart 1, i 45),
renaming /var/log/nginx/access.log.44.gz to /var/log/nginx/access.log.45.gz (rotatecount 52, logstart 1, i 44),
renaming /var/log/nginx/access.log.43.gz to /var/log/nginx/access.log.44.gz (rotatecount 52, logstart 1, i 43),
renaming /var/log/nginx/access.log.42.gz to /var/log/nginx/access.log.43.gz (rotatecount 52, logstart 1, i 42),
renaming /var/log/nginx/access.log.41.gz to /var/log/nginx/access.log.42.gz (rotatecount 52, logstart 1, i 41),
renaming /var/log/nginx/access.log.40.gz to /var/log/nginx/access.log.41.gz (rotatecount 52, logstart 1, i 40),
renaming /var/log/nginx/access.log.39.gz to /var/log/nginx/access.log.40.gz (rotatecount 52, logstart 1, i 39),
renaming /var/log/nginx/access.log.38.gz to /var/log/nginx/access.log.39.gz (rotatecount 52, logstart 1, i 38),
renaming /var/log/nginx/access.log.37.gz to /var/log/nginx/access.log.38.gz (rotatecount 52, logstart 1, i 37),
renaming /var/log/nginx/access.log.36.gz to /var/log/nginx/access.log.37.gz (rotatecount 52, logstart 1, i 36),
renaming /var/log/nginx/access.log.35.gz to /var/log/nginx/access.log.36.gz (rotatecount 52, logstart 1, i 35),
renaming /var/log/nginx/access.log.34.gz to /var/log/nginx/access.log.35.gz (rotatecount 52, logstart 1, i 34),
renaming /var/log/nginx/access.log.33.gz to /var/log/nginx/access.log.34.gz (rotatecount 52, logstart 1, i 33),
renaming /var/log/nginx/access.log.32.gz to /var/log/nginx/access.log.33.gz (rotatecount 52, logstart 1, i 32),
renaming /var/log/nginx/access.log.31.gz to /var/log/nginx/access.log.32.gz (rotatecount 52, logstart 1, i 31),
renaming /var/log/nginx/access.log.30.gz to /var/log/nginx/access.log.31.gz (rotatecount 52, logstart 1, i 30),
renaming /var/log/nginx/access.log.29.gz to /var/log/nginx/access.log.30.gz (rotatecount 52, logstart 1, i 29),
renaming /var/log/nginx/access.log.28.gz to /var/log/nginx/access.log.29.gz (rotatecount 52, logstart 1, i 28),
renaming /var/log/nginx/access.log.27.gz to /var/log/nginx/access.log.28.gz (rotatecount 52, logstart 1, i 27),
renaming /var/log/nginx/access.log.26.gz to /var/log/nginx/access.log.27.gz (rotatecount 52, logstart 1, i 26),
renaming /var/log/nginx/access.log.25.gz to /var/log/nginx/access.log.26.gz (rotatecount 52, logstart 1, i 25),
renaming /var/log/nginx/access.log.24.gz to /var/log/nginx/access.log.25.gz (rotatecount 52, logstart 1, i 24),
renaming /var/log/nginx/access.log.23.gz to /var/log/nginx/access.log.24.gz (rotatecount 52, logstart 1, i 23),
renaming /var/log/nginx/access.log.22.gz to /var/log/nginx/access.log.23.gz (rotatecount 52, logstart 1, i 22),
renaming /var/log/nginx/access.log.21.gz to /var/log/nginx/access.log.22.gz (rotatecount 52, logstart 1, i 21),
renaming /var/log/nginx/access.log.20.gz to /var/log/nginx/access.log.21.gz (rotatecount 52, logstart 1, i 20),
renaming /var/log/nginx/access.log.19.gz to /var/log/nginx/access.log.20.gz (rotatecount 52, logstart 1, i 19),
renaming /var/log/nginx/access.log.18.gz to /var/log/nginx/access.log.19.gz (rotatecount 52, logstart 1, i 18),
renaming /var/log/nginx/access.log.17.gz to /var/log/nginx/access.log.18.gz (rotatecount 52, logstart 1, i 17),
renaming /var/log/nginx/access.log.16.gz to /var/log/nginx/access.log.17.gz (rotatecount 52, logstart 1, i 16),
renaming /var/log/nginx/access.log.15.gz to /var/log/nginx/access.log.16.gz (rotatecount 52, logstart 1, i 15),
renaming /var/log/nginx/access.log.14.gz to /var/log/nginx/access.log.15.gz (rotatecount 52, logstart 1, i 14),
renaming /var/log/nginx/access.log.13.gz to /var/log/nginx/access.log.14.gz (rotatecount 52, logstart 1, i 13),
renaming /var/log/nginx/access.log.12.gz to /var/log/nginx/access.log.13.gz (rotatecount 52, logstart 1, i 12),
renaming /var/log/nginx/access.log.11.gz to /var/log/nginx/access.log.12.gz (rotatecount 52, logstart 1, i 11),
renaming /var/log/nginx/access.log.10.gz to /var/log/nginx/access.log.11.gz (rotatecount 52, logstart 1, i 10),
renaming /var/log/nginx/access.log.9.gz to /var/log/nginx/access.log.10.gz (rotatecount 52, logstart 1, i 9),
renaming /var/log/nginx/access.log.8.gz to /var/log/nginx/access.log.9.gz (rotatecount 52, logstart 1, i 8),
renaming /var/log/nginx/access.log.7.gz to /var/log/nginx/access.log.8.gz (rotatecount 52, logstart 1, i 7),
renaming /var/log/nginx/access.log.6.gz to /var/log/nginx/access.log.7.gz (rotatecount 52, logstart 1, i 6),
renaming /var/log/nginx/access.log.5.gz to /var/log/nginx/access.log.6.gz (rotatecount 52, logstart 1, i 5),
renaming /var/log/nginx/access.log.4.gz to /var/log/nginx/access.log.5.gz (rotatecount 52, logstart 1, i 4),
renaming /var/log/nginx/access.log.3.gz to /var/log/nginx/access.log.4.gz (rotatecount 52, logstart 1, i 3),
renaming /var/log/nginx/access.log.2.gz to /var/log/nginx/access.log.3.gz (rotatecount 52, logstart 1, i 2),
renaming /var/log/nginx/access.log.1.gz to /var/log/nginx/access.log.2.gz (rotatecount 52, logstart 1, i 1),
renaming /var/log/nginx/access.log.0.gz to /var/log/nginx/access.log.1.gz (rotatecount 52, logstart 1, i 0),
rotating log /var/log/nginx/error.log, log->rotateCount is 52
dateext suffix '-20141023'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /var/log/nginx/error.log.1 does not exist
renaming /var/log/nginx/error.log.52.gz to /var/log/nginx/error.log.53.gz (rotatecount 52, logstart 1, i 52),
renaming /var/log/nginx/error.log.51.gz to /var/log/nginx/error.log.52.gz (rotatecount 52, logstart 1, i 51),
renaming /var/log/nginx/error.log.50.gz to /var/log/nginx/error.log.51.gz (rotatecount 52, logstart 1, i 50),
renaming /var/log/nginx/error.log.49.gz to /var/log/nginx/error.log.50.gz (rotatecount 52, logstart 1, i 49),
renaming /var/log/nginx/error.log.48.gz to /var/log/nginx/error.log.49.gz (rotatecount 52, logstart 1, i 48),
renaming /var/log/nginx/error.log.47.gz to /var/log/nginx/error.log.48.gz (rotatecount 52, logstart 1, i 47),
renaming /var/log/nginx/error.log.46.gz to /var/log/nginx/error.log.47.gz (rotatecount 52, logstart 1, i 46),
renaming /var/log/nginx/error.log.45.gz to /var/log/nginx/error.log.46.gz (rotatecount 52, logstart 1, i 45),
renaming /var/log/nginx/error.log.44.gz to /var/log/nginx/error.log.45.gz (rotatecount 52, logstart 1, i 44),
renaming /var/log/nginx/error.log.43.gz to /var/log/nginx/error.log.44.gz (rotatecount 52, logstart 1, i 43),
renaming /var/log/nginx/error.log.42.gz to /var/log/nginx/error.log.43.gz (rotatecount 52, logstart 1, i 42),
renaming /var/log/nginx/error.log.41.gz to /var/log/nginx/error.log.42.gz (rotatecount 52, logstart 1, i 41),
renaming /var/log/nginx/error.log.40.gz to /var/log/nginx/error.log.41.gz (rotatecount 52, logstart 1, i 40),
renaming /var/log/nginx/error.log.39.gz to /var/log/nginx/error.log.40.gz (rotatecount 52, logstart 1, i 39),
renaming /var/log/nginx/error.log.38.gz to /var/log/nginx/error.log.39.gz (rotatecount 52, logstart 1, i 38),
renaming /var/log/nginx/error.log.37.gz to /var/log/nginx/error.log.38.gz (rotatecount 52, logstart 1, i 37),
renaming /var/log/nginx/error.log.36.gz to /var/log/nginx/error.log.37.gz (rotatecount 52, logstart 1, i 36),
renaming /var/log/nginx/error.log.35.gz to /var/log/nginx/error.log.36.gz (rotatecount 52, logstart 1, i 35),
renaming /var/log/nginx/error.log.34.gz to /var/log/nginx/error.log.35.gz (rotatecount 52, logstart 1, i 34),
renaming /var/log/nginx/error.log.33.gz to /var/log/nginx/error.log.34.gz (rotatecount 52, logstart 1, i 33),
renaming /var/log/nginx/error.log.32.gz to /var/log/nginx/error.log.33.gz (rotatecount 52, logstart 1, i 32),
renaming /var/log/nginx/error.log.31.gz to /var/log/nginx/error.log.32.gz (rotatecount 52, logstart 1, i 31),
renaming /var/log/nginx/error.log.30.gz to /var/log/nginx/error.log.31.gz (rotatecount 52, logstart 1, i 30),
renaming /var/log/nginx/error.log.29.gz to /var/log/nginx/error.log.30.gz (rotatecount 52, logstart 1, i 29),
renaming /var/log/nginx/error.log.28.gz to /var/log/nginx/error.log.29.gz (rotatecount 52, logstart 1, i 28),
renaming /var/log/nginx/error.log.27.gz to /var/log/nginx/error.log.28.gz (rotatecount 52, logstart 1, i 27),
renaming /var/log/nginx/error.log.26.gz to /var/log/nginx/error.log.27.gz (rotatecount 52, logstart 1, i 26),
renaming /var/log/nginx/error.log.25.gz to /var/log/nginx/error.log.26.gz (rotatecount 52, logstart 1, i 25),
renaming /var/log/nginx/error.log.24.gz to /var/log/nginx/error.log.25.gz (rotatecount 52, logstart 1, i 24),
renaming /var/log/nginx/error.log.23.gz to /var/log/nginx/error.log.24.gz (rotatecount 52, logstart 1, i 23),
renaming /var/log/nginx/error.log.22.gz to /var/log/nginx/error.log.23.gz (rotatecount 52, logstart 1, i 22),
renaming /var/log/nginx/error.log.21.gz to /var/log/nginx/error.log.22.gz (rotatecount 52, logstart 1, i 21),
renaming /var/log/nginx/error.log.20.gz to /var/log/nginx/error.log.21.gz (rotatecount 52, logstart 1, i 20),
renaming /var/log/nginx/error.log.19.gz to /var/log/nginx/error.log.20.gz (rotatecount 52, logstart 1, i 19),
renaming /var/log/nginx/error.log.18.gz to /var/log/nginx/error.log.19.gz (rotatecount 52, logstart 1, i 18),
renaming /var/log/nginx/error.log.17.gz to /var/log/nginx/error.log.18.gz (rotatecount 52, logstart 1, i 17),
renaming /var/log/nginx/error.log.16.gz to /var/log/nginx/error.log.17.gz (rotatecount 52, logstart 1, i 16),
renaming /var/log/nginx/error.log.15.gz to /var/log/nginx/error.log.16.gz (rotatecount 52, logstart 1, i 15),
renaming /var/log/nginx/error.log.14.gz to /var/log/nginx/error.log.15.gz (rotatecount 52, logstart 1, i 14),
renaming /var/log/nginx/error.log.13.gz to /var/log/nginx/error.log.14.gz (rotatecount 52, logstart 1, i 13),
renaming /var/log/nginx/error.log.12.gz to /var/log/nginx/error.log.13.gz (rotatecount 52, logstart 1, i 12),
renaming /var/log/nginx/error.log.11.gz to /var/log/nginx/error.log.12.gz (rotatecount 52, logstart 1, i 11),
renaming /var/log/nginx/error.log.10.gz to /var/log/nginx/error.log.11.gz (rotatecount 52, logstart 1, i 10),
renaming /var/log/nginx/error.log.9.gz to /var/log/nginx/error.log.10.gz (rotatecount 52, logstart 1, i 9),
renaming /var/log/nginx/error.log.8.gz to /var/log/nginx/error.log.9.gz (rotatecount 52, logstart 1, i 8),
renaming /var/log/nginx/error.log.7.gz to /var/log/nginx/error.log.8.gz (rotatecount 52, logstart 1, i 7),
renaming /var/log/nginx/error.log.6.gz to /var/log/nginx/error.log.7.gz (rotatecount 52, logstart 1, i 6),
renaming /var/log/nginx/error.log.5.gz to /var/log/nginx/error.log.6.gz (rotatecount 52, logstart 1, i 5),
renaming /var/log/nginx/error.log.4.gz to /var/log/nginx/error.log.5.gz (rotatecount 52, logstart 1, i 4),
renaming /var/log/nginx/error.log.3.gz to /var/log/nginx/error.log.4.gz (rotatecount 52, logstart 1, i 3),
renaming /var/log/nginx/error.log.2.gz to /var/log/nginx/error.log.3.gz (rotatecount 52, logstart 1, i 2),
renaming /var/log/nginx/error.log.1.gz to /var/log/nginx/error.log.2.gz (rotatecount 52, logstart 1, i 1),
renaming /var/log/nginx/error.log.0.gz to /var/log/nginx/error.log.1.gz (rotatecount 52, logstart 1, i 0),
running prerotate script
running script with arg /var/log/nginx/*.log : "
if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
run-parts /etc/logrotate.d/httpd-prerotate; \
fi \
"
renaming /var/log/nginx/access.log to /var/log/nginx/access.log.1
creating new /var/log/nginx/access.log mode = 0640 uid = 33 gid = 4
renaming /var/log/nginx/error.log to /var/log/nginx/error.log.1
creating new /var/log/nginx/error.log mode = 0640 uid = 33 gid = 4
running postrotate script
running script with arg /var/log/nginx/*.log : "
[ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
"
removing old log /var/log/nginx/access.log.53.gz
error: error opening /var/log/nginx/access.log.53.gz: No such file or directory
但是,存档最多只有 *.log.8.gz,因此 logrotate 在尝试与 交互时会失败/var/log/nginx/access.log.53.gz
。它到底为什么要这样做?我想我需要接触假文件来填充它?这似乎有点不对劲。
答案1
哎呀,经过长时间的挖掘,我终于找到了答案。就我而言,问题不在于 logrotate 失败了。该错误消息很好,并且实际上并没有停止 logrotate。问题是 nginx 在收到来自-USR1
的信号后没有释放日志文件的文件句柄kill
。长话短说,它没有重新加载日志文件的原因是因为该/var/log/nginx
文件夹不属于与 nginx 工作进程相同的用户(由 www-data 拥有,在 web 下运行)。我不知道这是如何改变的(可能是因为这个服务器最近重建了),但是将文件夹更改为由与 nginx 工作进程相同的用户拥有(并修复 logrotate 文件以将新日志作为 web 生成)解决了该问题。
答案2
我在 Ubuntu 14.04 中遇到了同样的问题,但阅读了您的回答后,我检查了我的 nginx 是否在拥有该文件夹的 www-data 下运行,因此......
然而我发现这个错误,指出 /etc/logrotate.d/nginx 中的 postrotate 部分存在错误。
要修复它,你可以注释掉
invoke-rc.d nginx rotate >/dev/null 2>&1
并使用以下任一选项(来自该错误的评论):
- 启动停止守护进程 --stop --signal USR1 --quiet --pidfile /run/nginx.pid --name nginx
- nginx -s 重新加载
- 服务 nginx 重新加载 >/dev/null 2>&1
答案3
导致此问题的另一个原因可能是磁盘已满。
笔记:当访问日志突然填满时甚至可能会发生这种情况!可能是 DoS 吗?
要重现这种情况,您的访问日志可能会突然被许多请求填满,一旦logrotate
触发它就会移动access.log
到access.log.1
,但无法压缩下一个文件,因此无法USR1
向 nginx 发送信号。
你如何解决这个问题?
- 如果您关心日志,请通过 ssh 管道传输文件 + 压缩到那里,然后将其带回来。一定要分析是什么原因导致访问或错误日志如此快地填满。
- 如果你不在乎,截短文件和重新加载nginx 或删除和重新开始, 确定不是到删除并重新加载因为这不会释放 inode(请不要说处理,这是一个 Windows 术语)