用 Perl 或 Python 重写日志解析脚本（摆脱 awk）

Question 1

这不是一个答案，但它对于注释来说太大并且需要格式化，因此为了解决您的评论“Python 代码更容易阅读和理解它的作用。”，仅供参考，一个具有合理变量名称的 AWK 脚本这就是我所做的思考你的Python脚本它看起来很像你的 python 脚本，但更简短，因为对于操作文本，awk 已经为你完成了你必须在 python 中编写代码才能完成的所有常见操作：

awk -v column=5 '
    { records[$column] = records[$column] $0 ORS }
    END {
        for ( timestamp in records ) {
            if ( gsub(ORS,"&",records[timestamp]) > 2 ) {
                printf "%s", records[timestamp]
            }
        }
    }
' logfile.log

但是在处理之前将整个文件读入内存是解决此问题的一种非常低效的方法。您应该在每次时间变化时进行测试并打印：

awk -v column=5 '
    $column != prev {
        prt()
        records = ""
        prev = $column
    }
    { records = records $0 ORS }
    END { prt() }

    function prt() {
        if ( gsub(ORS,"&",records) > 2 ) {
            printf "%s", records
        }
    }
' logfile.log

Answer

这不是一个答案，但它对于注释来说太大并且需要格式化，因此为了解决您的评论“Python 代码更容易阅读和理解它的作用。”，仅供参考，一个具有合理变量名称的 AWK 脚本这就是我所做的思考你的Python脚本它看起来很像你的 python 脚本，但更简短，因为对于操作文本，awk 已经为你完成了你必须在 python 中编写代码才能完成的所有常见操作：

awk -v column=5 '
    { records[$column] = records[$column] $0 ORS }
    END {
        for ( timestamp in records ) {
            if ( gsub(ORS,"&",records[timestamp]) > 2 ) {
                printf "%s", records[timestamp]
            }
        }
    }
' logfile.log

但是在处理之前将整个文件读入内存是解决此问题的一种非常低效的方法。您应该在每次时间变化时进行测试并打印：

awk -v column=5 '
    $column != prev {
        prt()
        records = ""
        prev = $column
    }
    { records = records $0 ORS }
    END { prt() }

    function prt() {
        if ( gsub(ORS,"&",records) > 2 ) {
            printf "%s", records
        }
    }
' logfile.log

Question 2

该解决方案本身是用 Python 3 编写的。

#!/usr/bin/env python3

import sys
import re
from collections import defaultdict


column_delimiter = sys.argv[1]
column = int(sys.argv[2]) - 1

records = defaultdict(list)

with open(sys.argv[3]) as inputfile:
  for lines in inputfile:
    line = lines.rstrip('\n')
    row_record = line.split(column_delimiter)
    records[row_record[column]].append(line)

for timestamps in records.values():
    if len(timestamps) == 3:
        for i in range(len(timestamps)):
          if (re.search('logged in|changed password|logged off', timestamps[i])):
                  print(timestamps[i])

用法：parse_log.py ' ' 5 logfile.log

Python 代码更容易阅读和理解它的作用。

Answer

该解决方案本身是用 Python 3 编写的。

#!/usr/bin/env python3

import sys
import re
from collections import defaultdict


column_delimiter = sys.argv[1]
column = int(sys.argv[2]) - 1

records = defaultdict(list)

with open(sys.argv[3]) as inputfile:
  for lines in inputfile:
    line = lines.rstrip('\n')
    row_record = line.split(column_delimiter)
    records[row_record[column]].append(line)

for timestamps in records.values():
    if len(timestamps) == 3:
        for i in range(len(timestamps)):
          if (re.search('logged in|changed password|logged off', timestamps[i])):
                  print(timestamps[i])

用法：parse_log.py ' ' 5 logfile.log

Python 代码更容易阅读和理解它的作用。

Question 3

与任何awk：

#!/usr/bin/awk -f
BEGIN { FS = "[|]" }

prvHour == $1 && prvUsr == $3 {
    if ($(NF-1) == "user logged in"        || 
        $(NF-1) == "user changed password" || 
        $(NF-1) == "user logged off" )
         actions[++actionCnt] = $0
    else actionCnt = 0
}

prvHour != $1 && prvUsr != $3 {
    if (prvHour && actionCnt == 3)
        for (i = 1; i <= actionCnt ; i++)
            print actions[i]

    prvHour = $1; prvUsr = $3
    actionCnt = 0 ; actions[++actionCnt] = $0
}

END {
    if (actionCnt == 3)
        for (i = 1; i <= actionCnt; i++)
            print actions[i]
}

不Perl使用外部库：

/bin/perl -e '
    while (1) {
        $uli = $uli // <>;
        $ucp = <> if $uli =~ /^([^|]*)[|][^|]*[|]([^|]*)[|] - [|]user logged in[|] -$/;
        last if tell() < 0 ; 
        if (!defined $ucp) { $uli = undef ; next; }
        $ulo = <> if $ucp =~ /^(\Q$1\E)[|][^|]*[|]($2)[|] - [|]user changed password[|] -$/;
        last if tell() < 0;
        if (!defined $ulo) { $uli = $ucp ; $ucp = undef ; next ; }
        if ($ulo !~ /^\Q$1\E[|][^|]*[|]$2[|] - [|]user logged off[|] -$/) {
            $uli = $ulo ; $ucp = $ulo = undef ; next ;
        }
        print "$uli$ucp$ulo";
        $uli = $ucp = $ulo = undef;
    }
' sample

使用python3usgin 只是sys读取文件并exit使用有意义的值进行调用。

#!/bin/python3
import sys

try:
    fullLine = actions = []
    prvHour = prvUsr = None
    chk_act = lambda x: x == "user logged in" or \
                 x == "user changed password" or \
                 x == "user logged off"


    with open(sys.argv[1]) as logFile:
        for line in logFile:
            hour, _, user, _, action, _ = line.split('|')
            if prvHour == hour and prvUsr == user:
                fullLine.append(line.strip())
                actions.append(action.strip())

            elif prvUsr != user and prvHour != hour:
                if len(actions) == 3 and all(map(chk_act, actions)):
                    print("\n".join(fullLine))

                prvUsr = user
                prvHour = hour
                actions = []
                actions.append(action.strip())
                fullLine = []
                fullLine.append(line.strip())


except IndexError:
    print("usage {} logfile".format(sys.argv[0]))
    sys.exit(1)

except (FileNotFoundError, PermissionError):
    print("{} not found or permission permission denied", sys.argv[1])
    sys.exit(1)

与任何sed：

#!/bin/sed -nf
N;/^\([^|]*\)|[^|]*|\([^|]*\)| - |user logged in| -\n\1|[^|]*|\2| - |user changed password| -$/{
    N;/\n\([^|]*\)|[^|]*|\([^|]*\)| - |user changed password| -\n\1|[^|]*|\2| - |user logged off| -$/{
        p;b
    }
    s/.*\n/\n/g;D
}
D

所有解决方案都避免将整个数据存储在内存中

Answer