☰

如何比较bash中一个目录中的所有文件？

tag-icon • tag-icon bash scripting diff
如何比较bash中一个目录中的所有文件？

给出的例子：

dir1/
 file1
 file2
 file3
 file4
  • file1-file2, file1-file3, file1-file4那么你如何比较file2->file3, file2->file4等等。在本例中，名称为 file_number，但也可以是任何名字。
  • 基本上制作一个diff但不仅仅是一个文件给其他人，而是将所有文件与所有文件进行比较。

我一直在尝试：

for f in $(find ./ -type f | sort | uniq); do

    compare=$(diff -rq "$f" "$1" &>/dev/null)
    if [[ $compare -eq $? ]]; then
        echo "$(basename "$f") yes equal $1"
    else
        echo "$(basename "$f") no equal $1"
    fi
done

回报

./file1 yes equal ./file1
./file2 no equal ./file1
./file3 yes equal ./file1
./script no equal ./file1
./sd no equal ./file1
  • 仅将任意文件号与 file1 进行比较
  • 我想需要另一个循环，但现在我有库存了[就像冒泡排序算法]
  • 如何制作另一个 IF 语句来停止比较 file1 yes equal file1 [same file]

答案1

是的，你确实需要两个循环。但看起来您不需要，diff因为您将 diff 输出丢弃到 /dev/null，您可以使用cmp它。

例如：

#!/bin/bash

# Read the list of files into an array called $files - we're iterating over
# the same list of files twice, but there's no need to run find twice.
#
# This uses NUL as the separator between filenames, so will work with
# filenames containing ANY valid character. Requires GNU find and GNU
# sort or non-GNU versions that support the `-print0` and `-z` options.
#
# The `-maxdepth 1` predicate prevents recursion into subdirs, remove it
# if that's not what you want.
mapfile -d '' -t files < <(find ./ -maxdepth 1 -type f -print0 | sort -z -u)

for a in "${files[@]}" ; do
  for b in "${files[@]}" ; do
    if [ ! "$a" = "$b" ] ; then
      if cmp --quiet "$a" "$b" ; then
        echo "Yes. $a is equal to $b"
      else
        echo "No. $a is not equal to $b"
      fi
    fi
  done
done

顺便说一句，这将产生大量输出（n × (n-1)输出行，其中 n 是文件数）。就我个人而言，我会删除或注释掉else和echo "No...."行，因为与其他文件相同的文件可能比唯一的文件少得多。

另请注意，如果 filesabc和xyz相同，那么它将比较它们两次并打印 Yes 两次：

Yes. ./abc is equal to ./xyz
Yes. ./xyz is equal to ./abc

有多种方法可以防止这种情况发生，最简单的方法可能是使用关联数组来跟踪我们相互比较的文件。例如

#!/bin/bash

# Read the list of files into an array called $files
mapfile -d '' -t files < <(find ./ -maxdepth 1 -type f -print0 | sort -z -u)

# declare that $seen is an associative array
declare -A seen

for a in "${files[@]}" ; do
  for b in "${files[@]}" ; do
    if [ ! "$a" = "$b" ] && [ -z "${seen[$a$b]}" ] && [ -z "${seen[$b$a]}" ] ; then
      seen[$a$b]=1
      seen[$b$a]=1
      if cmp --quiet "$a" "$b" ; then
        echo "Yes. $a is equal to $b"
      #else
      #  echo "No. $a is not equal to $b"
      fi
    fi
  done
done

答案2

成对比较许多文件很快就会变得很麻烦。 10 个文件需要 45 次比较。 100 个文件几乎需要 5000 个文件。我的测试集是 595 个文件（总共 10 GB），这需要超过 175,000 个配对比较。 （这组是 9 个过时的存档目录，包含来自各个分区的完整和部分备份的元数据。）

该方法是计算每个文件的校验和（在笔记本电脑上总共花费两分钟多一点），然后使用 awk 按校验和对文件进行分组（花费不到一秒）。

校验和过程是这个 Bash 片段：

#.. Checksum all the files, and show some statistics.
[ x ] && {
    time ( cd "${BackUpDir}" && cksum */* ) > "${CkSums}"
    du -s -h "${BackUpDir}"
    head -n 3 "${CkSums}"
    awk '{ Bytes += $2; }
        END { printf ("Files %d, %.2f MB\n", FNR, Bytes / (1024 * 1024)); }
        ' "${CkSums}"
}

显示此日志。

$ ./fileGroup
real    2m5.139s
user    1m3.141s
sys 0m24.685s
9.8G    /media/paul/WinData/tarMETADATA
2288228966 156844 20220107_002000/02_History.tld
1812380507 156992 20220107_002000/02_History.toc
3028427874 1000411 20220107_002000/06_TechHist.tld
Files 565, 10001.10 MB

real    0m0.024s  #.. (Runtime of the awk component)
user    0m0.018s
sys 0m0.008s

结果摘录：

Group of 5 files for cksum 1459775330
    20220319_114500/lib64.tld
    20220401_182500/lib64.tld
    20220407_192000/lib64.tld
    20220503_190500/lib64.tld
    20220503_232500/lib64.tld

Group of 3 files for cksum 2937156162
    20220407_192000/sbin.tld
    20220503_190500/sbin.tld
    20220503_232500/sbin.tld

Group of 2 files for cksum 3291901599
    20220503_190500/30_Photos.tld
    20220503_232500/30_Photos.tld

Counted 304 non-grouped files.

Bash 脚本大约有 60 行，其中 30 行是嵌入的 awk 脚本（我不知道需要任何 GNU/特定语法）。

#! /bin/bash --

#.. Determine groups of identical files.

BackUpDir="/media/paul/WinData/tarMETADATA"
CkSums="Cksum.txt"
Groups="Groups.txt"

#.. Group all the files by checksum, and report them.

fileGroup () {

    local Awk='
BEGIN { Db = 0; reCut2 = "^[ ]*[^ ]+[ ]+[^ ]+[ ]+"; }
{ if (Db) printf ("\n%s\n", $0); }

#.. Add a new cksum value.
! (($1,0) in Fname) {
    Cksum[++Cksum[0]] = $1;
    if (Db) printf ("Added Cksum %d value %s.\n", 
        Cksum[0], Cksum[Cksum[0]]);
    Fname[$1,0] = 0;
}
#.. Add a filename.
{
    Fname[$1,++Fname[$1,0]] = $0;
    sub (reCut2, "", Fname[$1,Fname[$1,0]]);
    if (Db) printf ("Fname [%s,%s] is \047%s\047\n",
        $1, Fname[$1,0], Fname[$1, Fname[$1,0]]);
}
#.. Report the identical files, grouped by checksum.
function Report (Local, k, ke, cs, j, je, Single) {
    ke = Cksum[0];
    for (k = 1; k <= ke; ++k) {
        cs = Cksum[k];
        je = Fname[cs,0];
        if (je < 2) { ++Single; continue; }
        printf ("\nGroup of %d files for cksum %s\n", je, cs);
        for (j = 1; j <= je; ++j) printf ("    %s\n", Fname[cs,j]);
    }
    printf ("\nCounted %d non-grouped files.\n", Single);
}
END { Report( ); }
'
    awk -f <( printf '%s' "${Awk}" )
}

#### Script Body Starts Here.

    #.. Checksum all the files, and show some statistics.
    [ x ] && {
        time ( cd "${BackUpDir}" && cksum */* ) > "${CkSums}"
        du -s -h "${BackUpDir}"
        head -n 3 "${CkSums}"
        awk '{ Bytes += $2; }
            END { printf ("Files %d, %.2f MB\n", FNR, Bytes / (1024 * 1024)); }
            ' "${CkSums}"
    }

    #.. Analyse the cksum data.
    time fileGroup < "${CkSums}" > "${Groups}"

相关内容