如何使用 s3cmd 将文件同步到 Amazon S3，检查是否已发送并在本地删除？

Question 1

顺便说一下，我需要做一些类似的事情，并编写了以下 bash 脚本。它的作用是：

使用以下命令获取目录中早于 $MINUTES 分钟的文件列表find
用于lsof确定文件是否已打开（如果文件是由编辑器打开的，则可能不正确）
用于s3cmd将文件复制到 S3 存储桶中。
比较 S3 中远程文件和本地文件的 MD5 和。如果符合要求，则删除本地文件。

-

#!/bin/bash
MINUTES=60
TARGET_DIR="s3://AWSbucketname/subfolder/`hostname -s`/"
LOCAL_DIR="/path/to/folder"
FILES=()

echo ""
echo "About to upload files in $LOCAL_DIR up to S3 folder:"
echo "    $TARGET_DIR"
echo "Then delete if MD5 sums line up."
echo "Starting in 5 seconds..."
sleep 5

cd $LOCAL_DIR

# Throw the list of files that the find command gets into an array
while IFS= read -d $'\0' -r file ; do
    FILES=("${FILES[@]}" "$file")
done < <(find $LOCAL_DIR -name \*.wav -mmin +$MINUTES -print0)

# echo "${WAV_FILES[@]}"   # DEBUG

for local_file in "${WAV_FILES[@]}"
do
    # Check that the file in question is not open.
    # lsof returns non-zero return value for file not in use
    lsof "$local_file" 2>&1 > /dev/null
    if test $? -ne 0 ; then
        echo ""
        echo "$local_file isn't open. Copying to S3..."
        s3cmd -p put $local_file $TARGET_DIR
        # s3cmd -n put $local_file $TARGET_DIR # DEBUG - dry-run

        ## Now attempt to delete if the MD5 sums check out:

        remote_file=${local_file##*/}
        md5sum_remote=`s3cmd info  "$TARGET_DIR$remote_file" | grep MD5 | awk '{print $3}'`
        md5sum_local=`md5sum $local_file | awk '{print $1}'`
        if [[ "$md5sum_remote" == "$md5sum_local" ]]; then
          echo "$remote_file MD5 sum checks out. Deleting..."
          rm $local_file
        fi
    fi
done

Answer

顺便说一下，我需要做一些类似的事情，并编写了以下 bash 脚本。它的作用是：

使用以下命令获取目录中早于 $MINUTES 分钟的文件列表find
用于lsof确定文件是否已打开（如果文件是由编辑器打开的，则可能不正确）
用于s3cmd将文件复制到 S3 存储桶中。
比较 S3 中远程文件和本地文件的 MD5 和。如果符合要求，则删除本地文件。

-

#!/bin/bash
MINUTES=60
TARGET_DIR="s3://AWSbucketname/subfolder/`hostname -s`/"
LOCAL_DIR="/path/to/folder"
FILES=()

echo ""
echo "About to upload files in $LOCAL_DIR up to S3 folder:"
echo "    $TARGET_DIR"
echo "Then delete if MD5 sums line up."
echo "Starting in 5 seconds..."
sleep 5

cd $LOCAL_DIR

# Throw the list of files that the find command gets into an array
while IFS= read -d $'\0' -r file ; do
    FILES=("${FILES[@]}" "$file")
done < <(find $LOCAL_DIR -name \*.wav -mmin +$MINUTES -print0)

# echo "${WAV_FILES[@]}"   # DEBUG

for local_file in "${WAV_FILES[@]}"
do
    # Check that the file in question is not open.
    # lsof returns non-zero return value for file not in use
    lsof "$local_file" 2>&1 > /dev/null
    if test $? -ne 0 ; then
        echo ""
        echo "$local_file isn't open. Copying to S3..."
        s3cmd -p put $local_file $TARGET_DIR
        # s3cmd -n put $local_file $TARGET_DIR # DEBUG - dry-run

        ## Now attempt to delete if the MD5 sums check out:

        remote_file=${local_file##*/}
        md5sum_remote=`s3cmd info  "$TARGET_DIR$remote_file" | grep MD5 | awk '{print $3}'`
        md5sum_local=`md5sum $local_file | awk '{print $1}'`
        if [[ "$md5sum_remote" == "$md5sum_local" ]]; then
          echo "$remote_file MD5 sum checks out. Deleting..."
          rm $local_file
        fi
    fi
done

Question 2

过了一段时间，我能够开发一个代码来bash检查md5sum两者s3和我的local文件并删除local 已经存在的文件amazon s3：

#!/bin/bash
datacenter="amazon"
hostname=`hostname`;
path="backup/server245"

s3=`s3cmd ls --list-md5 -H s3://company-backup/company/"$datacenter"/"$hostname"/"$path"/`

s3_list=`echo "$s3"|awk {'print $4" "$5'} | sed 's= .*/= ='`

locally=`md5sum /"$path"/*.gz`;
locally_list=$(echo "$locally" | sed 's= .*/= =');
#echo "$locally_list";

IFS=$'\n'
for i in $locally_list
do
  #echo $i
  locally_hash=`echo $i|awk {'print $1'}`
  locally_file=`echo $i|awk {'print $2'}`

  for j in $s3_list
  do
    s3_hash=$(echo $j|awk {'print $1'}); 
    s3_file=$(echo $j|awk {'print $2'});

    #to avoid empty file when have only hash from folder
    if [[ $s3_hash != "" ]] && [[ $s3_file != "" ]]; then 
      if [[ $s3_hash == $locally_hash ]] && [[ $s3_file == $locally_file ]]; then
        echo "### REMOVING ###";
        echo "$locally_file";
        #rm /"$path"/"$locally_file";
      fi
    fi
  done
done
unset IFS

Answer

过了一段时间，我能够开发一个代码来bash检查md5sum两者s3和我的local文件并删除local 已经存在的文件amazon s3：

#!/bin/bash
datacenter="amazon"
hostname=`hostname`;
path="backup/server245"

s3=`s3cmd ls --list-md5 -H s3://company-backup/company/"$datacenter"/"$hostname"/"$path"/`

s3_list=`echo "$s3"|awk {'print $4" "$5'} | sed 's= .*/= ='`

locally=`md5sum /"$path"/*.gz`;
locally_list=$(echo "$locally" | sed 's= .*/= =');
#echo "$locally_list";

IFS=$'\n'
for i in $locally_list
do
  #echo $i
  locally_hash=`echo $i|awk {'print $1'}`
  locally_file=`echo $i|awk {'print $2'}`

  for j in $s3_list
  do
    s3_hash=$(echo $j|awk {'print $1'}); 
    s3_file=$(echo $j|awk {'print $2'});

    #to avoid empty file when have only hash from folder
    if [[ $s3_hash != "" ]] && [[ $s3_file != "" ]]; then 
      if [[ $s3_hash == $locally_hash ]] && [[ $s3_file == $locally_file ]]; then
        echo "### REMOVING ###";
        echo "$locally_file";
        #rm /"$path"/"$locally_file";
      fi
    fi
  done
done
unset IFS

Question 3

来自官方文档：

--delete-after（新上传后执行删除[同步]）

或者

--delete-after-fetch（获取到本地文件后删除远程对象（仅适用于 [get] 和 [sync] 命令）。）

如果你想从远程同步到本地

https://s3tools.org/usage

Answer