我们正在运行一项服务，该服务会截取提供的 URL 的屏幕截图并将其发布到我们的 S3 存储桶中。类似于马奈特，但我们自定义编码的 nodejs 应用程序。我们不将屏幕截图存储在本地硬盘上。我们将它们临时存储以调整大小，然后删除。临时图像文件夹始终为空。
问题是：磁盘空间越来越少，直到服务器重新启动。例如，现在df -h显示：
ubuntu@ip-10-0-1-94:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 118G 74G 40G 65% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 7.4G 8.0K 7.4G 1% /dev
tmpfs 1.5G 360K 1.5G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 7.4G 0 7.4G 0% /run/shm
none 100M 0 100M 0% /run/user
然而，du -sh /显示：
root@ip-10-0-1-94:~# du -sh /
du: cannot access ‘/proc/14440’: No such file or directory
du: cannot access ‘/proc/14520/task/14520/fd/4’: No such file or directory
du: cannot access ‘/proc/14520/task/14520/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/14520/fd/4’: No such file or directory
du: cannot access ‘/proc/14520/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/14521’: No such file or directory
7.0G /
如果我对根文件系统中的所有文件夹都执行此操作du，则总计将为 7 Gb，而不是 74。如果我重新启动服务器，一旦它再次备份，将有 7 Gb，这是应该的，但在 10-12 小时内将再次达到 70+，并且还在增加。
我们将mongodb其用作存储，因此我假设它可以，但是，我删除了smallfiles之前放置的配置选项。仍然是同样的事情。
输出如下mount：
ubuntu@ip-10-0-1-94:~$ mount
/dev/xvda1 on / type ext4 (rw,discard)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/cgroup type tmpfs (rw)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
none on /sys/fs/pstore type pstore (rw)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)
重新启动任何正在运行的服务（例如mongodb或）supervisor不会改变任何内容。以下是示例：
root@ip-10-0-1-94:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 118G 74G 40G 65% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 7.4G 8.0K 7.4G 1% /dev
tmpfs 1.5G 360K 1.5G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 7.4G 0 7.4G 0% /run/shm
none 100M 0 100M 0% /run/user
root@ip-10-0-1-94:~# service mongod restart
mongod stop/waiting
mongod start/running, process 31590
root@ip-10-0-1-94:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 118G 74G 40G 65% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 7.4G 8.0K 7.4G 1% /dev
tmpfs 1.5G 360K 1.5G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 7.4G 0 7.4G 0% /run/shm
none 100M 0 100M 0% /run/user
或supervisor控制node流程（工作者和应用程序）：
root@ip-10-0-1-94:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 118G 74G 40G 65% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 7.4G 8.0K 7.4G 1% /dev
tmpfs 1.5G 360K 1.5G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 7.4G 0 7.4G 0% /run/shm
none 100M 0 100M 0% /run/user
root@ip-10-0-1-94:~# service supervisor restart
Restarting supervisor: supervisord.
root@ip-10-0-1-94:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 118G 74G 40G 65% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 7.4G 8.0K 7.4G 1% /dev
tmpfs 1.5G 360K 1.5G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 7.4G 0 7.4G 0% /run/shm
none 100M 0 100M 0% /run/user
更新：事实证明，这是因为gearman日志中有大量
accept(Too many open files) -> libgearman-server/gearmand.cc:851
消息。即使文件被删除，它仍然被 gearman 进程打开，因此空间不会被释放。这是证据：
root@ip-10-0-1-94:~# sudo lsof -s | awk '$5 == "REG"' | sort -n -r -k 7,7 | head -n 1
gearmand 4221 gearman 3w REG 202,1 31748949650 143608 /var/log/gearman-job-server/gearman.log.1 (deleted)
（感谢@Andrew Henle）
现在下一个问题是：为什么要将gearman其写入日志。如上所述这里gearman这是因为与状态的连接太多TIME_WAIT但是，它们不在TIME_WAIT，而是在ESTABLISHED。这里他们是。
如果我这样做strace -p 4221，我只会看到这个
write(22, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169649, 568914324}) = 0
gettimeofday({1446109467, 793708}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33010), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 874
write(17, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169659, 749954206}) = 0
gettimeofday({1446109477, 974726}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33060), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 875
write(32, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169659, 754505349}) = 0
gettimeofday({1446109477, 979307}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33062), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 876
write(27, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169664, 300399805}) = 0
gettimeofday({1446109482, 525209}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33134), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 877
write(22, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169666, 161035104}) = 0
gettimeofday({1446109484, 385826}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33165), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 878
write(17, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169668, 308112847}) = 0
gettimeofday({1446109486, 532900}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33186), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 879
write(32, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169671, 251265264}) = 0
gettimeofday({1446109489, 476077}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33218), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 880
write(27, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169672, 320483648}) = 0
gettimeofday({1446109490, 545274}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33232), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 881
write(22, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169676, 186686282}) = 0
gettimeofday({1446109494, 411486}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33303), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 882
write(17, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169684, 699748557}) = 0
gettimeofday({1446109502, 924549}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33320), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 883
write(32, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169687, 906830251}) = 0
gettimeofday({1446109506, 131601}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33348), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 884
write(27, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169701, 112588731}) = 0
gettimeofday({1446109519, 337387}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33386), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 885
write(22, "\3", 1) = 1
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169707, 686312787}) = 0
gettimeofday({1446109525, 911113}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33420), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 886
write(17, "\3", 1) = 1
每一部分
epoll_wait(6, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
clock_gettime(CLOCK_MONOTONIC, {169707, 686312787}) = 0
gettimeofday({1446109525, 911113}, NULL) = 0
accept4(9, {sa_family=AF_INET, sin_port=htons(33420), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 886
write(17, "\3", 1)
每 3-5 秒添加一次。几分钟内不会再有其他事情发生。
答案1
无论哪个进程创建了这个文件，它都是罪魁祸首：
gearmand 811 gearman 3w REG 202,1 71016771760 143618 /var/log/gearman-job-server/gearman.log.1 (deleted)
鉴于它被称为gearman.log.1，我怀疑无论是什么执行日志滚动，它都没有正确地执行。
df当您看到和之间存在严重不匹配的情况时du，通常是一个已删除的文件，但进程仍打开它。 lsof | grep deleted在 Linux 上可以很好地找到它们。
deleted只需在您发布的输出中搜索即可lsof显示其他几个*.1似乎具有相同不正确的翻转问题的日志文件。
答案2
关于 CentOS 的另一条信息。在这种情况下，当使用“systemctl”启动进程时。您必须修改系统文件 ==> /usr/lib/systemd/system/processName.service。文件中有以下行：
LimitNOFILE=50000
然后重新加载系统配置：
systemctl daemon-reload