![如何在日志中查找模式列表并从中生成报告?](https://linux22.com/image/58388/%E5%A6%82%E4%BD%95%E5%9C%A8%E6%97%A5%E5%BF%97%E4%B8%AD%E6%9F%A5%E6%89%BE%E6%A8%A1%E5%BC%8F%E5%88%97%E8%A1%A8%E5%B9%B6%E4%BB%8E%E4%B8%AD%E7%94%9F%E6%88%90%E6%8A%A5%E5%91%8A%EF%BC%9F.png)
我有一个 C++ 可执行文件,它扫描我的一些文件,并且这些文件中有一些用户 ID。扫描完成后,它会生成以下日志文件,(abc.log)
如下所示 -
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_32_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_32_200003_5.data"
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 45576752 with value badge_leaf_cat and status -2
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 870206432 with value badge_leaf_cat and status -2
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:120) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_32_200003_5.data is valid
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_13_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_13_200003_5.data"
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 876269533 with value badge_leaf_cat and status -2
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 17256973 with value badge_leaf_cat and status -2
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 830173693 with value badge_leaf_cat and status -2
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:120) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_13_200003_5.data is valid
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_0_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_0_200003_5.data"
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744073135142816 num attributes: 0 seeing 1 bad records from 365 records
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744073698151136 num attributes: 0 seeing 2 bad records from 595 records
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744072929739296 num attributes: 0 seeing 3 bad records from 1214 records
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:117) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data is corrupt
现在我需要 grep 上面的日志并找出已扫描了多少文件、有多少用户 ID 失败以及哪些文件已损坏。
- 对于它扫描了多少个文件,我需要在每一行中查找
checking file
单词,并在此基础上增加它扫描的文件数。 - 对于有多少唯一用户 ID 失败,我需要
failed reading user id
在一行中查找单词并在此基础上增加计数,然后提供失败的用户 ID 列表。 - 如果任何用户 ID 失败,则意味着保存该用户 ID 的文件将被损坏,因此我需要
is corrupt
在每一行中查找单词并找到损坏的文件名。一般来说,该文件database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data
已损坏。
以下是我扫描上述日志后希望看到的响应 -
Total Number of Files Scanned - 1000
Total Number of Unique User ID failed - 10000
Total Number of Files Corrupted - 5
List of Unique User Id's which are corrupt -
UserId-A
UserId-B
Files which are corrupted -
FileName-A
FileName-B
使用 grep 扫描日志后如何继续获得上述结果?
答案1
尝试以下脚本:
#!/bin/bash
logfile="$1"
nfiles=$(grep -c 'checking file' "$logfile")
failed_userid=($(grep -oP 'failed reading user id: \K[^ ]*' "$logfile"))
corrupted_files=($(grep -oP '[^ ]*(?= is corrupt)' "$logfile"))
echo "Total Number of Files Scanned - $nfiles"
echo "Total Number of Unique User ID failed - ${#failed_userid[@]}"
echo "Total Number of Files Corrupted - ${#corrupted_files[@]}"
echo
echo "List of Unique User Id's which are corrupt - "
for uid in "${failed_userid[@]}"; do
echo "$uid"
done
echo
echo "Files which are corrupted - "
for corf in "${corrupted_files[@]}"; do
echo "$corf"
done
运行它
$ ./script file.log
您问题的输入结果如下所示
Total Number of Files Scanned - 3
Total Number of Unique User ID failed - 3
Total Number of Files Corrupted - 1
List of Unique User Id's which are corrupt -
18446744073135142816
18446744073698151136
18446744072929739296
Files which are corrupted -
/database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data
简短说明:
-c
grep 选项计算匹配行数-P
启用 Perl 正则表达式语法-o
仅匹配部分行(?=
构造是所谓的正向前瞻(将其作为模式,但不包含到输出中)\K
是后向断言(采用整个模式,但从结果中丢弃到目前为止的所有内容)
其余的应该是显而易见的。但请注意,我假设文件名中没有空格!