☰

将文件视为集合并对其执行集合操作的 Linux 工具

tag-icon • tag-icon command-line text-processing utilities
将文件视为集合并对其执行集合操作的 Linux 工具

有谁知道有专门设计用于将文件视为集合并对它们执行集合操作的 Linux 工具吗？比如差异、交集等？

答案1

假设元素是除 NUL 和换行符之外的字符串（但要注意换行符在文件名中是有效的），您可以表示放作为每行一个元素的文本文件，并使用一些标准 Unix 实用程序。

设置会员资格

$ grep -Fxc 'element' set   # outputs 1 if element is in set
                            # outputs >1 if set is a multi-set
                            # outputs 0 if element is not in set

$ grep -Fxq 'element' set   # returns 0 (true)  if element is in set
                            # returns 1 (false) if element is not in set

$ awk '$0 == "element" { s=1; exit }; END { exit !s }' set
# returns 0 if element is in set, 1 otherwise.

$ awk -v e='element' '$0 == e { s=1; exit } END { exit !s }'

设置交集

$ comm -12 <(sort set1) <(sort set2)  # outputs intersect of set1 and set2

$ grep -xF -f set1 set2

$ sort set1 set2 | uniq -d

$ join -t <(sort A) <(sort B)

$ awk '!done { a[$0]; next }; $0 in a' set1 done=1 set2

设置相等

$ cmp -s <(sort set1) <(sort set2) # returns 0 if set1 is equal to set2
                                   # returns 1 if set1 != set2

$ cmp -s <(sort -u set1) <(sort -u set2)
# collapses multi-sets into sets and does the same as previous

$ awk '{ if (!($0 in a)) c++; a[$0] }; END{ exit !(c==NR/2) }' set1 set2
# returns 0 if set1 == set2
# returns 1 if set1 != set2

$ awk '{ a[$0] }; END{ exit !(length(a)==NR/2) }' set1 set2
# same as previous, requires >= gnu awk 3.1.5

设置基数

$ wc -l < set     # outputs number of elements in set

$ awk 'END { print NR }' set

$ sed '$=' set

子集测试

$ comm -23 <(sort -u subset) <(sort -u set) | grep -q '^'
# returns true iff subset is not a subset of set (has elements not in set)

$ awk '!done { a[$0]; next }; { if !($0 in a) exit 1 }' set done=1 subset
# returns 0 if subset is a subset of set
# returns 1 if subset is not a subset of set

集合联盟

$ cat set1 set2     # outputs union of set1 and set2
                    # assumes they are disjoint

$ awk 1 set1 set2   # ditto

$ cat set1 set2 ... setn   # union over n sets

$ sort -u set1 set2  # same, but doesn't assume they are disjoint

$ sort set1 set2 | uniq

$ awk '!a[$0]++' set1 set2       # ditto without sorting

设置补集

$ comm -23 <(sort set1) <(sort set2)
# outputs elements in set1 that are not in set2

$ grep -vxF -f set2 set1           # ditto

$ sort set2 set2 set1 | uniq -u    # ditto

$ awk '!done { a[$0]; next }; !($0 in a)' set2 done=1 set1

设置对称差

$ comm -3 <(sort set1) <(sort set2) | tr -d '\t'  # assumes not tab in sets
# outputs elements that are in set1 or in set2 but not both

$ sort set1 set2 | uniq -u

$ cat <(grep -vxF -f set1 set2) <(grep -vxF -f set2 set1)

$ grep -vxF -f set1 set2; grep -vxF -f set2 set1

$ awk '!done { a[$0]; next }; $0 in a { delete a[$0]; next }; 1;
       END { for (b in a) print b }' set1 done=1 set2

电源组

一组显示的所有可能的子集以空格分隔，每行一个：

$ p() { [ "$#" -eq 0 ] && echo || (shift; p "$@") |
        while read r; do printf '%s %s\n%s\n' "$1" "$r" "$r"; done; }
$ p $(cat set)

（假设元素不包含 SPC、TAB（假设默认值为$IFS）、反斜杠、通配符）。

设置笛卡尔积

$ while IFS= read -r a; do while IFS= read -r b; do echo "$a, $b"; done < set1; done < set2

$ awk '!done { a[$0]; next }; { for (i in a) print i, $0 }' set1 done=1 set2

不相交集测试

$ comm -12 <(sort set1) <(sort set2)  # does not output anything if disjoint

$ awk '++seen[$0] == 2 { exit 1 }' set1 set2 # returns 0 if disjoint
                                             # returns 1 if not

空置测试

$ wc -l < set            # outputs 0  if the set is empty
                         # outputs >0 if the set is not empty

$ grep -q '^' set        # returns true (0 exit status) unless set is empty

$ awk '{ exit 1 }' set   # returns true (0 exit status) if set is empty

最低限度

$ sort set | head -n 1   # outputs the minimum (lexically) element in the set

$ awk 'NR == 1 { min = $0 }; $0 < min { min = $0 }; END { print min }'
# ditto, but does numeric comparison when elements are numerical

最大限度

$ sort test | tail -n 1    # outputs the maximum element in the set

$ sort -r test | head -n 1

$ awk 'NR == 1 || $0 > max { max = $0 }; END { print max }'
# ditto, but does numeric comparison when elements are numerical

全部可在http://www.catonmat.net/blog/set-operations-in-unix-shell-simplified/

答案2

有点。您需要自己处理排序，但comm可以用来做到这一点，将每行视为集合成员： -12用于交集，-13用于差异。 （并-23给你翻转的差异，即，set2 - set1而不是set1 - set2。）联盟就sort -u在这个设置中。

答案3

自 16.10 起，微型控制台工具“setop”现已在 Debian Stretch 和 Ubuntu 中可用。您可以通过以下方式获取它 sudo apt install setop

这里有些例子。要操作的集合作为不同的输入文件给出： setop input # is equal to "sort input --unique" setop file1 file2 --union # option --union is default and can be omitted setop file1 file2 file3 --intersection # more than two inputs are allowed setop file1 - --symmetric-difference # ndash stands for standard input setop file1 -d file2 # all elements contained in 1 but not 2

布尔查询仅EXIT_SUCCESS在 true 的情况下返回，EXIT_FAILURE否则返回消息。这样，setop就可以在shell中使用了。 setop inputfile --contains "value" # is element value contained in input? setop A.txt B.txt --equal C.txt # union of A and B equal to C? setop bigfile --subset smallfile # analogous --superset setop -i file1 file2 --is-empty # intersection of 1 and 2 empty (disjoint)?

还可以准确地描述如何解析输入流，实际上是通过正则表达式：

  • setop input.txt --input-separator "[[:space:]-]"表示空白（即\v \t \n \r \f或空格）或减号被解释为元素之间的分隔符（默认为换行符，即输入文件的每一行都是一个元素）
  • setop input.txt --input-element "[A-Za-z]+"表示元素只是由拉丁字符组成的单词，所有其他字符都被视为元素之间的分隔符

此外，您还可以

  • --count输出集的所有元素，
  • --trim所有输入元素（即删除所有不需要的前后字符，如空格、逗号等），
  • 将空元素视为有效 via --include-empty，
  • --ignore-case,
  • 设置--output-separator输出流的元素之间（默认为\n），
  • 等等。

请参阅man setop或github.com/phisigma/setop了解更多信息。

答案4

如果您将文件视为一组行，并且文件已排序，则有comm。

如果您将文件视为行的（多）集，并且行未排序，则grep可以进行差异和交集（它实现了集合差异和交集，但不考虑多重集的计数）。联盟只是cat。

grep -xF -f small large >intersection
grep -vxF -f small large >difference
cat small large >union

相关内容