如何删除文件中选定字段内的重复(或多个)值,只保留一份副本?
示例
输入文件:
A 1,2,3,45,1,8,2,3
B 5,6,6,6,6,6,2,3,7
预期输出:
A 1,2,3,45,8
B 5,6,2,3,7
答案1
一个sed
一:
sed '
s/[^[:blank:]]\{1,\}/,&,/g;:1
s/\(\(,[^,[:blank:]]*\)\(,[^,[:blank:]]*\)*\)\2,/\1,/;t1
s/,\([^[:blank:]]*\),/\1/g'
(它处理所有包含,
字符并保留间距的字段)
答案2
和perl
:
perl -MList::MoreUtils=uniq -pe 's{\S*,\S*}{join ",", uniq split ",", $&}ge'
(它处理所有包含,
字符并保留间距的字段)
答案3
另一个perl
解决方案:
perl -anle '
print "$F[0] ", join ",", grep {!$seen{$_}++} split ",",$F[1];
%seen=();
' file
A 1,2,3,45,8
B 5,6,2,3,7
答案4
awk '{n=split($2, a, ","); $2=a[1];
for(i=2; i<=n; i++)
{$2 = ($2 ~ "(^|,)" a[i] "($|,)") ? $2 : ($2 "," a[i])}}1' OFS='\t' file