rsync --link-dest. '建立文件列表' 的时间递增

rsync --link-dest. '建立文件列表' 的时间递增

我已经在 Windows Server 2008 上将 rsync 作为 cygwin 任务运行了大约两个月。我使用 --link-dest 选项每天将包含大约 10TB 数据(约 1300 万个文件)的服务器内容“快照”到基于 Linux 的 NAS 服务器。一切开始都很顺利,但我很快注意到完成所需的时间在慢慢增加。它开始大约需要三个小时,但现在大约需要十五个小时。

命令如下...

rsync -rlptDhPR \
    --password-file=password \
    --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \
    --Stats \
    --delete \
    --log-file=logfilename \
    --link-dest=linkdestpath \
    sourceDirectory \
    [email protected]::destinationDirectory

我没有使用完整的 -a 选项,因为 Windows 和 Linux 所有权之间的差异把事情搞乱了。

第一个日志文件看起来像这样......

2018/10/01 23:00:14 [2164] building file list
...transfer file list here
2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, dir: 1,499,821, link: 1)
2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11)
2018/10/02 02:11:30 [2164] Number of regular files transferred: 310
2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes
2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes
2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes
2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes
2018/10/02 02:11:30 [2164] File list size: 10.42M
2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds
2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds
2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M
2018/10/02 02:11:30 [2164] Total bytes received: 7.51M
2018/10/02 02:11:30 [2164] sent 235.68M bytes  received 7.51M bytes  21.17K bytes/sec
2018/10/02 02:11:30 [2164] total size is 10.40T  speedup is 42,753.79

最近的看起来像这样......

2018/11/24 23:00:15 [2924] building file list
2018/11/24 23:00:17 [2924] cd..t...... /cygdrive/
2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, dir: 1,501,780, link: 1)
2018/11/25 13:21:16 [2924] Number of created files: 0
2018/11/25 13:21:16 [2924] Number of regular files transferred: 0
2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes
2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes
2018/11/25 13:21:16 [2924] Literal data: 0 bytes
2018/11/25 13:21:16 [2924] Matched data: 0 bytes
2018/11/25 13:21:16 [2924] File list size: 10.35M
2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds
2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds
2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M
2018/11/25 13:21:16 [2924] Total bytes received: 7.51M
2018/11/25 13:21:16 [2924] sent 236.55M bytes  received 7.51M bytes  4.72K bytes/sec
2018/11/25 13:21:16 [2924] total size is 10.49T  speedup is 42,996.96

如您所见,两种情况下的开始时间都是晚上 11:00(23:00)。第一个日志显示识别要传输的文件大约需要三个小时(我省略了文件列表 - 它很长),第二个日志需要十四个小时来完成相同的工作(在这种情况下,这是在周末完成的,我包含了整个日志文件,它正确地识别出没有文件发生变化)

文件数量和其他所有东西都符合预期。只是“构建文件列表”所花的时间明显更长。

相关内容