☰

删除两个文件中的重复项并合并唯一的文件

tag-icon • tag-icon command-line text-processing
删除两个文件中的重复项并合并唯一的文件

我有两个大文本文件，checksums_1.txt 和 checksums_2.txt，我想解析这些文件并删除它们之间的重复部分并将唯一的行合并到一个文件中。

每个文件的每一行都有以下结构。

大小、md5、路径

例如：Checksums_1.txt

9565, a4fs614as6fas4fa4s46fsaf1, /mnt/app/1tier/2tier/filename.exe
9565, a4fs614as6fas4fa4s46fsaf1, /mnt/app/1tier/2tier/filename2.exe

例如：Checksums_2.txt

9565, a4fs614as6fas4fa4s46fsaf1, /mnt/temp/1tier/2tier/filename.exe
9565, a4fs614as6fas4fa4s46fsaf1, /mnt/temp/1tier/2tier/filename2.exe
9565, a4fs614as6fas4fa4s46fsaf1, /mnt/temp/1tier/2tier/newfile.exe

checksums_1.txt 和 checksums_2.txt 之间需要检查的部分位于挂载点 /mnt/app/ 和 /mnt/temp/ 之后，换句话说，从每行的开头到挂载点 /mnt/temp/ 或 /mnt/app/ 的结尾都将被忽略。

checksums_1.txt 里面的数据更重要，因此如果发现重复，则必须将 checksums_1.txt 中的行移动到合并文件中。

Checksums_1.txt 的一部分

1058,b8203a236b4f15316e516165a6546666,/mnt/app/Certificados/ca.crt
2694,8a815adefde4fa0c263e74832b15de64,/mnt/app/Certificados/ca.db.certs/01.pem
136,77bf2e5313dbaac4df76a4b72df2e2ad,/mnt/app/Certificados/ca.db.index

Checksums_2.txt 的一部分

1058,b8203a236b4f1531616318284202c9e6,/mnt/temp/Certificados/ca.crt
3,72b2ac90f7f3ff075a937d6be8fc3dc3,/mnt/temp/Certificados/ca.db.serial 
2694,8a815adefde4fa0c263e74832b15de64,/mnt/temp/Certificados/ca.db.certs/01.pem
136,77bf2e5313dbaac4df76a4b72df2e2ad,/mnt/temp/Certificados/ca.db.index

合并文件的示例

1058,b8203a236b4f15316e516165a6546666,/mnt/app/Certificados/ca.crt 
3,72b2ac90f7f3ff075a937d6be8fc3dc3,/mnt/temp/Certificados/ca.db.serial 
2694,8a815adefde4fa0c263e74832b15de64,/mnt/app/Certificados/ca.db.certs/01.pem
136,77bf2e5313dbaac4df76a4b72df2e2ad,/mnt/app/Certificados/ca.db.index

答案1

假设两个文件都不是巨大的，下面的python脚本也可以完成这个工作。

怎么运行的

脚本读取这两个文件。文件_1（优先的文件）由您在头部部分为文件输入的目录进行拆分（在您的示例中/mnt/app/）。

随后，file_1 中的行将写入输出文件（合并文件）。同时，file_2 中的行将从行列表中删除如果标识字符串（挂载点之后的部分）出现在该行中。最后，file_2 的“剩余”行（file_1 中不存在重复行）也写入输出文件。结果：

文件1：

1058,b8203a236b4f15316e516165a6546666,/mnt/app/Certificados/ca.crt
2694,8a815adefde4fa0c263e74832b15de64,/mnt/app/Certificados/ca.db.certs/01.pem
136,77bf2e5313dbaac4df76a4b72df2e2ad,/mnt/app/Certificados/ca.db.index

文件2：

1058,b8203a236b4f15316e516165a6546666,/mnt/app/Certificados/ca.crt
3,72b2ac90f7f3ff075a937d6be8fc3dc3,/mnt/temp/Certificados/ca.db.serial
2694,8a815adefde4fa0c263e74832b15de64,/mnt/app/Certificados/ca.db.certs/01.pem
136,77bf2e5313dbaac4df76a4b72df2e2ad,/mnt/app/Certificados/ca.db.index

合并：

1058,b8203a236b4f15316e516165a6546666,/mnt/app/Certificados/ca.crt
2694,8a815adefde4fa0c263e74832b15de64,/mnt/app/Certificados/ca.db.certs/01.pem
136,77bf2e5313dbaac4df76a4b72df2e2ad,/mnt/app/Certificados/ca.db.index
3,72b2ac90f7f3ff075a937d6be8fc3dc3,/mnt/temp/Certificados/ca.db.serial

剧本

#!/usr/bin/env python3
#---set the path to file1, file2 and the mountpoint used in file1 below
f1 = "/path/to/file_1"; m_point = "/mnt/app"; f2 = "/path/to/file_2"
merged = "/path/to/merged_file"
#---
lines1 = [(l, l.split(m_point)[-1]) for l in open(f1).read().splitlines()]
lines2 = [l for l in open(f2).read().splitlines()]

for l in lines1:
    open(merged, "a+").write(l[0]+"\n")
    for line in [line for line in lines2 if l[1] in line]:
            lines2.remove(line)

for l in lines2:
    open(merged, "a+").write(l+"\n")

如何使用

  1. 将脚本复制到一个空文件中，另存为merge.py
  2. 在脚本的头部部分，将路径设置为f1（file_1）、、f2合并文件的路径以及中提到的挂载点file_1。

  3. 通过命令运行：

    python3 /path/to/merge.py

编辑

或者稍微短一点：

#!/usr/bin/env python3
#---set the path to file1, file2 and the mountpoint used in file1 below
f1 = "/path/to/file_1"; m_point = "/mnt/app"; f2 = "/path/to/file_2"
merged = "/path/to/merged_file"
#---
lines = lambda f: [l for l in open(f).read().splitlines()]
lines1 = lines(f1); lines2 = lines(f2); checks = [l.split(m_point)[-1] for l in lines1]
for item in sum([[l for l in lines2 if c in l] for c in checks], []):
    lines2.remove(item)
for item in lines1+lines2:
    open(merged, "a+").write(item+"\n")

答案2

如果您愿意使用 python（因此如果性能不是问题），那么您可以使用以下脚本实现您想要的功能：

#!/usr/bin/env python3

import sys
import csv
import re

mountpoint1 = "/mnt/app/"
mountpoint2 = "/mnt/temp/"

if (len(sys.argv) != 4):
    print('Usage: {} <input file 1> <input file 2> <output file>'.format(sys.argv[0]))
    exit(1)

inputFileName1 = sys.argv[1]
inputFileName2 = sys.argv[2]
outputFileName = sys.argv[3]

# We place entries from both input files in the same dictionary
# The key will be the filename stripped of the mountpoint
# The value will be the whole line
fileDictionary = dict()

# First we read entries from file2, so that those
# from file2 will later overwrite them when needed
with open(inputFileName2) as inputFile2:
    csvReader = csv.reader(inputFile2)
    for row in csvReader:
        if len(row) == 3:
            # The key will be the filename stripped of the mountpoint
            key = re.sub(mountpoint2, '', row[2])
            # The value will be the whole line
            fileDictionary[key] = ','.join(row)

# Entries from file1 will overwrite those from file2
with open(inputFileName1) as inputFile1:
    csvReader = csv.reader(inputFile1)
    for row in csvReader:
        if len(row) == 3:
            # The key will be the filename stripped of the mountpoint
            key = re.sub(mountpoint1, '', row[2])
            # The value will be the whole line
            fileDictionary[key] = ','.join(row)

# Write all the entries to the output file
with open(outputFileName, 'w') as outputFile:
    for key in fileDictionary:
        outputFile.write(fileDictionary[key])
        outputFile.write('\n')

只需将脚本另存为merge-checksums.py，并赋予其执行权限

chmod u+x merge-checksums.py

并运行如下：

./merge-checksums.py Checksums_1.txt Checksums_2.txt out.txt

答案3

版本bash（带有awk和grep）：

#!/bin/bash

filename1="$1"
filename2="$2"

keys=$(awk -F'/' '{ for(i=4;i<NF;i++) printf "%s",$i "/"; if (NF) printf "%s",$NF; printf "\n"}' "$filename1" "$filename2" | awk '{gsub(/^[ \t]+|[ \t]+$/,"")};1' | sort -u)

while read -r key
do
    match=$(grep "$key" "$filename1")
    if [ "$match" != "" ]
    then
        echo "$match"
    else
        grep "$key" "$filename2"
    fi
done <<< "$keys"

Checksums_1.txt

9565, 1111111111111111111111111, /mnt/app/1tier/2tier/filename.exe
9565, 0000000000000000000000000, /mnt/app/1tier/2tier/filename2.exe

Checksums_2.txt

9565, 2222222222222222222222222, /mnt/temp/1tier/2tier/filename.exe
9565, 0000000000000000000000000, /mnt/temp/1tier/2tier/filename2.exe
9565, 3333333333333333333333333, /mnt/temp/1tier/2tier/newfile.exe

运行

./merge_checksum Checksums_1.txt Checksums_2.txt > Checksums_3.txt

Checksums_3.txt

9565, 1111111111111111111111111, /mnt/app/1tier/2tier/filename.exe
9565, 0000000000000000000000000, /mnt/app/1tier/2tier/filename2.exe
9565, 3333333333333333333333333, /mnt/temp/1tier/2tier/newfile.exe

或者互换输入文件

./merge_checksum Checksums_2.txt Checksums_1.txt > Checksums_3.txt

Checksums_3.txt

9565, 0000000000000000000000000, /mnt/temp/1tier/2tier/filename2.exe
9565, 2222222222222222222222222, /mnt/temp/1tier/2tier/filename.exe
9565, 3333333333333333333333333, /mnt/temp/1tier/2tier/newfile.exe

相关内容