![Gearman 将“打开的文件过多”写入日志。磁盘空间问题](https://linux22.com/image/669780/Gearman%20%E5%B0%86%E2%80%9C%E6%89%93%E5%BC%80%E7%9A%84%E6%96%87%E4%BB%B6%E8%BF%87%E5%A4%9A%E2%80%9D%E5%86%99%E5%85%A5%E6%97%A5%E5%BF%97%E3%80%82%E7%A3%81%E7%9B%98%E7%A9%BA%E9%97%B4%E9%97%AE%E9%A2%98.png)
我们正在运行一项服务,该服务会截取提供的 URL 的屏幕截图并将其发布到我们的 S3 存储桶中。类似于马奈特,但我们自定义编码的 nodejs 应用程序。我们不将屏幕截图存储在本地硬盘上。我们将它们临时存储以调整大小,然后删除。临时图像文件夹始终为空。
问题是:磁盘空间越来越少,直到服务器重新启动。例如,现在df -h
显示:
ubuntu@ip-10-0-1-94:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 118G 74G 40G 65% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 7.4G 8.0K 7.4G 1% /dev
tmpfs 1.5G 360K 1.5G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 7.4G 0 7.4G 0% /run/shm
none 100M 0 100M 0% /run/user
然而,du -sh /
显示:
root@ip-10-0-1-94:~# du -sh /
du: cannot access ‘/proc/14440’: No such file or directory
du: cannot access ‘/proc/14520/task/14520/fd/4’: No such file or directory
du: cannot access ‘/proc/14520/task/14520/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/14520/fd/4’: No such file or directory
du: cannot access ‘/proc/14520/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/14521’: No such file or directory
7.0G /
如果我对根文件系统中的所有文件夹都执行此操作du
,则总计将为 7 Gb,而不是 74。如果我重新启动服务器,一旦它再次备份,将有 7 Gb,这是应该的,但在 10-12 小时内将再次达到 70+,并且还在增加。
我们将mongodb
其用作存储,因此我假设它可以,但是,我删除了smallfiles
之前放置的配置选项。仍然是同样的事情。
输出如下mount
:
ubuntu@ip-10-0-1-94:~$ mount
/dev/xvda1 on / type ext4 (rw,discard)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/cgroup type tmpfs (rw)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
none on /sys/fs/pstore type pstore (rw)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)
重新启动任何正在运行的服务(例如mongodb
或)supervisor
不会改变任何内容。以下是示例:
root@ip-10-0-1-94:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 118G 74G 40G 65% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 7.4G 8.0K 7.4G 1% /dev
tmpfs 1.5G 360K 1.5G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 7.4G 0 7.4G 0% /run/shm
none 100M 0 100M 0% /run/user
root@ip-10-0-1-94:~# service mongod restart
mongod stop/waiting
mongod start/running, process 31590
root@ip-10-0-1-94:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 118G 74G 40G 65% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 7.4G 8.0K 7.4G 1% /dev
tmpfs 1.5G 360K 1.5G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 7.4G 0 7.4G 0% /run/shm
none 100M 0 100M 0% /run/user
或supervisor
控制node
流程(工作者和应用程序):
root@ip-10-0-1-94:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 118G 74G 40G 65% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 7.4G 8.0K 7.4G 1% /dev
tmpfs 1.5G 360K 1.5G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 7.4G 0 7.4G 0% /run/shm
none 100M 0 100M 0% /run/user
root@ip-10-0-1-94:~# service supervisor restart
Restarting supervisor: supervisord.
root@ip-10-0-1-94:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 118G 74G 40G 65% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 7.4G 8.0K 7.4G 1% /dev
tmpfs 1.5G 360K 1.5G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 7.4G 0 7.4G 0% /run/shm
none 100M 0 100M 0% /run/user
更新:事实证明,这是因为gearman
日志中有大量
accept(Too many open files) -> libgearman-server/gearmand.cc:851
消息。即使文件被删除,它仍然被 gearman 进程打开,因此空间不会被释放。这是证据:
root@ip-10-0-1-94:~# sudo lsof -s | awk '$5 == "REG"' | sort -n -r -k 7,7 | head -n 1
gearmand 4221 gearman 3w REG 202,1 31748949650 143608 /var/log/gearman-job-server/gearman.log.1 (deleted)
(感谢@Andrew Henle)
现在下一个问题是:为什么要将gearman
其写入日志。如上所述这里gearman
这是因为与状态的连接太多TIME_WAIT
但是,它们不在TIME_WAIT
,而是在ESTABLISHED
。这里他们是。
如果我这样做strace -p 4221
,我只会看到这个
write(22, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169649, 568914324}) = 0
gettimeofday({1446109467, 793708}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33010), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 874
write(17, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169659, 749954206}) = 0
gettimeofday({1446109477, 974726}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33060), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 875
write(32, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169659, 754505349}) = 0
gettimeofday({1446109477, 979307}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33062), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 876
write(27, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169664, 300399805}) = 0
gettimeofday({1446109482, 525209}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33134), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 877
write(22, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169666, 161035104}) = 0
gettimeofday({1446109484, 385826}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33165), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 878
write(17, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169668, 308112847}) = 0
gettimeofday({1446109486, 532900}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33186), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 879
write(32, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169671, 251265264}) = 0
gettimeofday({1446109489, 476077}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33218), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 880
write(27, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169672, 320483648}) = 0
gettimeofday({1446109490, 545274}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33232), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 881
write(22, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169676, 186686282}) = 0
gettimeofday({1446109494, 411486}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33303), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 882
write(17, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169684, 699748557}) = 0
gettimeofday({1446109502, 924549}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33320), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 883
write(32, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169687, 906830251}) = 0
gettimeofday({1446109506, 131601}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33348), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 884
write(27, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169701, 112588731}) = 0
gettimeofday({1446109519, 337387}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33386), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 885
write(22, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169707, 686312787}) = 0
gettimeofday({1446109525, 911113}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33420), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 886
write(17, "\3", 1) = 1
每一部分
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169707, 686312787}) = 0
gettimeofday({1446109525, 911113}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33420), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 886
write(17, "\3", 1)
每 3-5 秒添加一次。几分钟内不会再有其他事情发生。
答案1
无论哪个进程创建了这个文件,它都是罪魁祸首:
gearmand 811 gearman 3w REG 202,1 71016771760 143618 /var/log/gearman-job-server/gearman.log.1 (deleted)
鉴于它被称为gearman.log.1
,我怀疑无论是什么执行日志滚动,它都没有正确地执行。
df
当您看到和之间存在严重不匹配的情况时du
,通常是一个已删除的文件,但进程仍打开它。 lsof | grep deleted
在 Linux 上可以很好地找到它们。
deleted
只需在您发布的输出中搜索即可lsof
显示其他几个*.1
似乎具有相同不正确的翻转问题的日志文件。
答案2
关于 CentOS 的另一条信息。在这种情况下,当使用“systemctl”启动进程时。您必须修改系统文件 ==> /usr/lib/systemd/system/processName.service。文件中有以下行:
LimitNOFILE=50000
然后重新加载系统配置:
systemctl daemon-reload