![同时将多个源复制到多个目标](https://linux22.com/image/1431892/%E5%90%8C%E6%97%B6%E5%B0%86%E5%A4%9A%E4%B8%AA%E6%BA%90%E5%A4%8D%E5%88%B6%E5%88%B0%E5%A4%9A%E4%B8%AA%E7%9B%AE%E6%A0%87.png)
假设我需要复制文件
~/path/to/file
到user@remote:/path/to/file
~/path/to/another/file
到user@remote:/path/to/another/file
~/path/alternative
到user@remote:/path/alternative
有没有办法使用rsync
或来做到这一点scp
?
我试过了rsync ~/path/{to/file,to/another/file,alternative} user@remote:/path
但如果我需要复制到另一个目的地,那么这不是一个选择another_path
。
如果我需要复制至少 20 个文件,那么逐个文件复制的时间就太长了。
答案1
如果您想通过 rsync 传输选定的文件(即不是包括内容的整个目录),那么最好的方法是使用 rsync 的--files-from
选项。放置要传输的文件的路径名(相对于您指定的源目录)。有关更多详细信息,请参阅 rsync 的手册页。
答案2
该脚本将复制任意数量的输入目录到任意数量的主机,在平行下通过使用 shell 后台作业。这里我使用了太平洋保险协会因为拆分和管道输入文件很容易,但您rsync --files-from
也可以修改它以供使用。
用法:(multi-cpio.sh <srcfile> <dstfile>
需要 ssh-keys 才能进行非交互式登录)
srcfile
每行包含一个目录,例如:
/usr/share/man/man1
/usr/share/man/man3
/usr/share/man/man5
dstfile
每行包含一个远程目标,例如:
user@host:/tmp/a
user@host:/tmp/b
user@host:/tmp/c
...以及来源multi-cpio.sh
:
#!/bin/bash
SRCLIST=$1
DSTLIST=$2
# create temporary files, safely, in /tmp with mcpio prefix
TEMP=`tempfile -p mcpio`
# loop over input file and create a list of all files to be copied
for I in `cat $SRCLIST` ; do
find $I -depth -print >>$TEMP
done
# split the file into CPU count + factor
# set to '4' here, seems like ~6x CPIO will max my 1Gb Ethernet link to ~800Mbps in atop
SPLITCOUNT= $(( $( wc -l $TEMP | cut -d' ' -f1) / $(( $( grep processor /proc/cpuinfo | wc -l ) + 4 )) ))
split -l $SPLITCOUNT $TEMP $TEMP
# nested loops, for each target and for each chunk, start a background copy
for DEST in `cat $DSTLIST` ; do
# this selects the "user@host" from user@host:/target/path
HOST=$(expr substr $DEST 1 $(($(expr index $DEST :)-1)))
# this selects the "/target/path" from user@host:/target/path
DIR=$(expr substr $DEST $(($(expr index $DEST :)+1)) $(expr length $DEST))
# loop over all the split tempfiles with ${TEMP}??
for I in ${TEMP}?? ; do
# use cpio in copy-out mode to stream the files through ssh
# then ensure the target is in place in the remote host, cd into it,
# and copy-in the files.
# note the '&' at the end backgrounds this command, so the loop
# will continue and generate more concurrent background jobs
cat $I | cpio -o | ssh $HOST \
"mkdir $DIR ; cd $DIR && cpio -idum --no-absolute-filenames" &
# sleep for a second before the next spawn to allow for ssh auth
# and so sshd doesn't start dropping new sessions
sleep 1
done
done
# cleanup the tempfiles
rm ${TEMP}??
rm $TEMP
也可以看看http://rightsock.com/~kjw/Ramblings/tar_v_cpio.html以获得额外的 cpio 参考和此脚本的灵感。
编辑:仅使用一个 src dst 的替代语法,无需创建 srcfile destfile:
multi-cpio.sh <(echo "/path/to/src") <(echo "user@host:/path/to/dest")