☰

while read line find – 使用 perl 更快？

tag-icon • tag-icon awk find perl
while read line find – 使用 perl 更快？

我有一个文本文件，其中包含由制表符分隔的三列，我逐行读取第三列以查找目录中名称中包含此名称的所有文件。由于它是一个最多有 1000 个条目的文件，因此我尝试用“查找”来解决它并不合适，因为它花费了太多时间。

while read f; 
do var1=`echo "$f" | cut -f1`; 
var2=`echo "$f" | cut -f2` ; 
var3=`echo "$f" | cut -f3`; 
echo "\n ID1 = $var1 \n ID2 = $var2 \n\n Path:";
find //myDirectory/ -type f -name *$var3* -not -path '*/zz_masters/*' -exec ls -Sd {} + ;
echo "\n----------------------"; 
done >> /SearchList.txt < /ResultList.txt

正如您所看到的，由于某些文件的分辨率不同，因此排除了一个文件夹，并且结果按大小排序。

搜索列表.txt：

a1 a    1 x1    Trappist
b2 b    2 y2    Mars
c3 c    3 z3    Pegasi

结果：

/myDirectory/

 ID1 = a1 a 
 ID2 = 1 x1 
 
 Path:
/myDirectory/xx/Trappist-1.png
/myDirectory/xx/Trappist-2.png

----------------------

 ID1 = b2 b 
 ID2 = 2 y2 
 
 Path:
/myDirectory/yy/Mars-1.jpg

----------------------

 ID1 = c3 c 
 ID2 = 3 z3 
 
 Path:
/myDirectory/xx/51PegasiB.tif

----------------------

为了希望它运行得更快，我用 perl 尝试了它。我是 Perl 新手，但我的结果很糟糕，而且我被困在脚本中。它创建了一个循环。这就是我所在的地方：

perl find.pl /myDirectory/ /SearchList.txt /ResultList.txt

#!/usr/bin/perl -w
use strict; 
use warnings; 
use File::Find;

open (IN, "$ARGV[1]") or die;
open(my $fh_out, '>', "$ARGV[2]");

my @files;

print $fh_out "$ARGV[0]\n";

while (my $line = <IN>) {    
    chomp $line;
my @columns = split(/\t/, $line);

find(sub { 
      push @files,"$File::Find::name" if /$columns[2]/;

### I think print has to be inside sub but each search result  shows separately and is still slow:
#   print $fh_out "\n\n----------------------------\n
#ID1: $columns[0]\nID2: $columns[1]Searchstring: $columns[2]\n
#Path:\n", "$File::Find::name\n" if /$columns[2]/;

    }, $ARGV[0]);

### outside sub: displays the search results together, but also slow and with a loop :(
print $fh_out "\n\n----------------------------\n
ID1: $columns[0]\nID2: $columns[1]
Searchstring: $columns[2]\n\nPath:\n", join "\n", @files;

}

close IN;
close $fh_out;

exit;

Perl 可能无法提供我想要的速度提升吗？如果没有，会有什么替代方案？

答案1

对 bash 代码进行代码审查：

  • read可以为你挑选单词
  • echo "\n" 不会打印换行符
  • 使用$(...)而不是`...`-参考
  • 使用正确的缩进 对重定向符号要更加小心
while read -r var1 var2 var3 rest; do
    printf "\n ID1 = %s \n ID2 = %s \n\n Path:\n" "$var1" "$var2"
    find //myDirectory/ -type f -name "*$var3*" -not -path '*/zz_masters/*' -exec ls -Sd {} +
    # ........................ quoted ^.......^
    printf "\n----------------------\n"; 
done < /SearchList.txt > /ResultList.txt

然而，加快速度的方法是只运行find一次：

id1=()
id2=()
substrings=()
names=( -false )
declare -A paths=()

while read -r var1 var2 var3 rest; do
    id1+=( "$var1" )
    id2+=( "$var2" )
    substrings+=( "*$var3*" )
    names+=( -o -name "*$var3*" )
done < /SearchList.txt 


find /myDirectory/ -type f \( "${names[@]}" \) -not -path '*/zz_masters/*' -prinf "%s %p\0" \
| sort -znr \
| while read -d '' -r size name; do
    for s in "${substrings[@]}"; do
        if [[ $name == *"$s"* ]]; then
            paths[$s]+="$name"$'\n'
            break
        fi
    done
done

fmt="\n ID1 = %s \n ID2 = %s \n\n Path:\n%s\n----------------------\n"

for idx in "${!id1[@]}"; do
    printf "$fmt" "${id1[idx]}" "${id2[idx]}" "${paths[${substrings[idx]}]}"
done > /ResultList.txt

答案2

如果您的文件名不包含制表符或换行符，您可以尝试此操作：

find . -type f -print |
awk '
    NR==FNR {
        name2ids[$3][1] = $1
        name2ids[$3][2] = $2
        next
    }
    {
        for (name in name2ids) {
            if ( index($NF,name) ) {
                matches[name][$0]
            }
        }
    }
    END {
        for (name in name2ids) {
            print "ID1 =", name2ids[name][1]
            print "ID2 =", name2ids[name][2]
            print "\nPath:"
            if (name in matches) {
                for (file in matches[name]) {
                    print file
                }
            }
        }
    }
' FS='\t' SearchList.txt FS='/' -

上面使用 GNU awk 来处理数组的数组，这里是 POSIX 版本（未经测试）：

find . -type f -print |
awk '
    NR==FNR {
        name2ids[$3] = $1 RS $2
        next
    }
    {
        for (name in name2ids) {
            if ( index($NF,name) ) {
                matches[name] = (name in matches ? matches[name] RS : "") $0
            }
        }
    }
    END {
        for (name in name2ids) {
            split(name2ids[name],ids,RS)
            print "ID1 =", ids[1]
            print "ID2 =", ids[2]
            print "\nPath:"
            split(matches[name],files,RS)
            for (idx in files) {
                print files[idx]
            }
        }
    }
' FS='\t' SearchList.txt FS='/' -

相关内容