☰

如何在日志中查找模式列表并从中生成报告？

tag-icon • tag-icon bash shell-script text-processing grep
如何在日志中查找模式列表并从中生成报告？

我有一个 C++ 可执行文件，它扫描我的一些文件，并且这些文件中有一些用户 ID。扫描完成后，它会生成以下日志文​​件，(abc.log)如下所示 -

INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_32_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_32_200003_5.data"
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 45576752 with value badge_leaf_cat and status -2
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 870206432 with value badge_leaf_cat and status -2
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:120) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_32_200003_5.data is valid
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_13_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_13_200003_5.data"
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 876269533 with value badge_leaf_cat and status -2
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 17256973 with value badge_leaf_cat and status -2
WARN [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgr.cc:389) - Failed to upsert attribute for uuid 830173693 with value badge_leaf_cat and status -2
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:120) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_13_200003_5.data is valid
INFO [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:110) - checking file: "p1_weekly_1980_0_200003_5.data" with path: "/database/batch/p1_snapshot/p1_weekly_1980_0_200003_5.data"
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744073135142816 num attributes: 0 seeing 1 bad records from 365 records
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744073698151136 num attributes: 0 seeing 2 bad records from 595 records
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/KKLFileMgrUtil.cc:493) - failed reading user id: 18446744072929739296 num attributes: 0 seeing 3 bad records from 1214 records
ERROR [0x7faff2b32a00] (/home/david/abc/golden/mmap/mapper_check/mapper_check.cc:117) - golden_file: /database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data is corrupt

现在我需要 grep 上面的日志并找出已扫描了多少文件、有多少用户 ID 失败以及哪些文件已损坏。

  • 对于它扫描了多少个文件，我需要在每一行中查找checking file单词，并在此基础上增加它扫描的文件数。
  • 对于有多少唯一用户 ID 失败，我需要failed reading user id在一行中查找单词并在此基础上增加计数，然后提供失败的用户 ID 列表。
  • 如果任何用户 ID 失败，则意味着保存该用户 ID 的文件将被损坏，因此我需要is corrupt在每一行中查找单词并找到损坏的文件名。一般来说，该文件database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data已损坏。

以下是我扫描上述日志后希望看到的响应 -

Total Number of Files Scanned - 1000
Total Number of Unique User ID failed - 10000
Total Number of Files Corrupted - 5

List of Unique User Id's which are corrupt - 
UserId-A
UserId-B

Files which are corrupted - 
FileName-A
FileName-B

使用 grep 扫描日志后如何继续获得上述结果？

答案1

尝试以下脚本：

#!/bin/bash

logfile="$1"

nfiles=$(grep -c 'checking file' "$logfile")
failed_userid=($(grep -oP 'failed reading user id: \K[^ ]*' "$logfile"))
corrupted_files=($(grep -oP '[^ ]*(?= is corrupt)' "$logfile"))


echo "Total Number of Files Scanned - $nfiles"
echo "Total Number of Unique User ID failed - ${#failed_userid[@]}"
echo "Total Number of Files Corrupted - ${#corrupted_files[@]}"
echo

echo "List of Unique User Id's which are corrupt - "
for uid in "${failed_userid[@]}"; do
   echo "$uid"
done

echo

echo "Files which are corrupted - "
for corf in "${corrupted_files[@]}"; do
   echo "$corf"
done

运行它

$ ./script file.log

您问题的输入结果如下所示

Total Number of Files Scanned - 3
Total Number of Unique User ID failed - 3
Total Number of Files Corrupted - 1

List of Unique User Id's which are corrupt - 
18446744073135142816
18446744073698151136
18446744072929739296

Files which are corrupted - 
/database/batch/p1_snapshot//p1_weekly_1980_0_200003_5.data

简短说明：

  • -cgrep 选项计算匹配行数
  • -P启用 Perl 正则表达式语法
  • -o仅匹配部分行
  • (?=构造是所谓的正向前瞻（将其作为模式，但不包含到输出中）
  • \K是后向断言（采用整个模式，但从结果中丢弃到目前为止的所有内容）

其余的应该是显而易见的。但请注意，我假设文件名中没有空格！

相关内容