☰

使用 awk 解析和格式化 syslog 数据

tag-icon • tag-icon awk
使用 awk 解析和格式化 syslog 数据

我正在从生成器获取系统日志数据，我想提取行开头的日期和时间以及文件末尾的操作。

到目前为止我所拥有的只是这样：

awk -F ' ' '{print $1,$2,$3,":",$NF}' test.log

我的输出有不完整的日志消息

Sep 17 20:47:02 : Warning
Sep 20 12:02:20 : Started
Sep 20 12:09:31 : Level
Sep 20 12:10:27 : Level
Sep 20 12:10:28 : Stopped
Sep 23 09:19:03 : Auto
Sep 23 09:19:04 : Cleared
Sep 23 09:19:13 : Auto
Sep 30 12:33:48 : Auto
Sep 30 12:33:55 : Started
Sep 30 12:41:14 : Level
Sep 30 12:41:56 : Stopped
Sep 30 12:41:58 : Cleared
Oct 3 14:00:52 : Started
Oct 3 14:15:17 : Stopped
Oct 3 14:20:39 : Auto
Oct 15 12:02:55 : Started
Oct 15 12:12:50 : Stopped

这就是我要打印的内容；实际日期和时间以及完整的操作消息：

Date Time Fuel has High Warning
Date Time Genset Started
Date Time  Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Date Time  Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Most Severe Level
Date Time  Genset Stopped
Date Time  Genset Not In Auto
Date Time  All Faults Cleared
Date Time  Genset in Auto
Date Time  Genset Not In Auto
Date Time  Genset Started
Date Time  Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Date Time  Genset Stopped
Date Time  All Faults Cleared
Date Time  Genset Started
Date Time  Genset Stopped
Date Time  Genset in Auto
Date Time  Genset Started
Date Time  Genset Stopped

每条线的长度没有固定的长度。

这是测试文件

Sep 17 20:47:02 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#011#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000##006#013+#006#001#004#001��s#001#001#000#004#0249/17/2023 8:47:01 PM0#023#006#013+#006#001#004#001��s#001#002#000#004#004Fuel0#023#006#013+#006#001#004#001��s#001#004#000#004#00445940#026#006#013+#006#001#004#001��s#001#003#000#004#007Warning0$#006#013+#006#001#004#001��s#001#005#000#004#025Fuel has High Warning
Sep 20 12:02:20 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#012#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/20/2023 12:02:23 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451430#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Started
Sep 20 12:09:31 192.168.22.60 0�#001(#002#001#001#004#005AE4ML��#001#032#002#001#013#002#001#000#002#001#0000�#001#0150#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/20/2023 12:09:34 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#022#006#013+#006#001#004#001��s#001#004#000#004#0031460#026#006#013+#006#001#004#001��s#001#003#000#004#007Warning0s#006#013+#006#001#004#001��s#001#005#000#004dEngine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 20 12:10:27 192.168.22.60 0�#001##002#001#001#004#005AE4ML��#001#025#002#001#014#002#001#000#002#001#0000�#001#0100#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/20/2023 12:10:30 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#022#006#013+#006#001#004#001��s#001#004#000#004#0031510#027#006#013+#006#001#004#001��s#001#003#000#004#010Shutdown0m#006#013+#006#001#004#001��s#001#005#000#004^Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Most Severe Level
Sep 20 12:10:28 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#015#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/20/2023 12:10:30 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451440#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Stopped
Sep 23 09:19:03 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#016#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000##006#013+#006#001#004#001��s#001#001#000#004#0249/23/2023 9:19:05 AM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451380#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0!#006#013+#006#001#004#001��s#001#005#000#004#022Genset Not In Auto
Sep 23 09:19:04 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#017#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000##006#013+#006#001#004#001��s#001#001#000#004#0249/23/2023 9:19:07 AM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00472190#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0!#006#013+#006#001#004#001��s#001#005#000#004#022All Faults Cleared
Sep 23 09:19:13 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#020#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000##006#013+#006#001#004#001��s#001#001#000#004#0249/23/2023 9:19:16 AM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00456640#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset in Auto
Sep 30 12:33:48 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#021#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/30/2023 12:33:50 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451380#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0!#006#013+#006#001#004#001��s#001#005#000#004#022Genset Not In Auto
Sep 30 12:33:55 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#022#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/30/2023 12:33:57 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451430#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Started
Sep 30 12:41:14 192.168.22.60 0�#001(#002#001#001#004#005AE4ML��#001#032#002#001#023#002#001#000#002#001#0000�#001#0150#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/30/2023 12:41:17 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#022#006#013+#006#001#004#001��s#001#004#000#004#0031460#026#006#013+#006#001#004#001��s#001#003#000#004#007Warning0s#006#013+#006#001#004#001��s#001#005#000#004dEngine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 30 12:41:56 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#024#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/30/2023 12:41:59 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451440#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Stopped
Sep 30 12:41:58 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#025#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000$#006#013+#006#001#004#001��s#001#001#000#004#0259/30/2023 12:42:01 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00472190#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0!#006#013+#006#001#004#001��s#001#005#000#004#022All Faults Cleared
Oct  3 14:00:52 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#026#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000##006#013+#006#001#004#001��s#001#001#000#004#02410/3/2023 2:00:55 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451430#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Started
Oct  3 14:15:17 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#027#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000##006#013+#006#001#004#001��s#001#001#000#004#02410/3/2023 2:15:20 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451440#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Stopped
Oct  3 14:20:39 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#030#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000##006#013+#006#001#004#001��s#001#001#000#004#02410/3/2023 2:20:42 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00456640#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset in Auto
Oct 15 12:02:55 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#031#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000%#006#013+#006#001#004#001��s#001#001#000#004#02610/15/2023 12:02:59 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451430#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Started
Oct 15 12:12:50 192.168.22.60 0��#002#001#001#004#005AE4ML���#002#001#032#002#001#000#002#001#0000��0#015#006#010+#006#001#002#001#001#003#000C#001#0000#030#006#012+#006#001#006#003#001#001#004#001#000#006#012+#006#001#004#001��s#001#0000%#006#013+#006#001#004#001��s#001#001#000#004#02610/15/2023 12:12:54 PM0#033#006#013+#006#001#004#001��s#001#002#000#004#01412615_lussie0#023#006#013+#006#001#004#001��s#001#004#000#004#00451440#032#006#013+#006#001#004#001��s#001#003#000#004#013Information0#035#006#013+#006#001#004#001��s#001#005#000#004#016Genset Stopped

答案1

它看起来确实有点像您可能对数据使用了错误的字符集编码。 （这可以解释数据中的 �� 字符。）您应该研究正在处理的输出的参考文档，以了解应如何解释它。

然而，考虑到它“基本上是正确的”，看起来字段分隔符基本上是#这样，所以你可以从以下开始：

awk -F'#' '{ print $1, $NF }'

样本输出

Sep 17 20:47:02 192.168.22.60 0�� 025Fuel has High Warning
Sep 20 12:02:20 192.168.22.60 0�� 016Genset Started
Sep 20 12:09:31 192.168.22.60 0� 004dEngine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 20 12:10:27 192.168.22.60 0� 004^Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Most Severe Level

现在我们需要拆分第一组空格分隔的字段，并从“最后一个”字段的开头删除根据经验看来是三个字符的内容：

awk -F'#' '{ split($1, a, " "); print a[1], a[2], a[3], substr($NF,4,length($NF)) }'

样本输出

Sep 17 20:47:02 Fuel has High Warning
Sep 20 12:02:20 Genset Started
Sep 20 12:09:31 dEngine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 20 12:10:27 ^Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Most Severe Level

没有任何明显的证据表明如何或为何应从第三行和第四行中删除前导dand以便保留它们。^

答案2

使用任何 POSIX awk：

$ awk '{sub(/[0-9]+\..*(#[0-9]{3}){4}[^[:upper:]]*/,"")} 1' file
Sep 17 20:47:02 Fuel has High Warning
Sep 20 12:02:20 Genset Started
Sep 20 12:09:31 Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 20 12:10:27 Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Most Severe Level
Sep 20 12:10:28 Genset Stopped
Sep 23 09:19:03 Genset Not In Auto
Sep 23 09:19:04 All Faults Cleared
Sep 23 09:19:13 Genset in Auto
Sep 30 12:33:48 Genset Not In Auto
Sep 30 12:33:55 Genset Started
Sep 30 12:41:14 Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 30 12:41:56 Genset Stopped
Sep 30 12:41:58 All Faults Cleared
Oct  3 14:00:52 Genset Started
Oct  3 14:15:17 Genset Stopped
Oct  3 14:20:39 Genset in Auto
Oct 15 12:02:55 Genset Started
Oct 15 12:12:50 Genset Stopped

#NNN上面假设您在要删除的部分末尾始终有一系列至少 4 个字符串，并且#NNN在要打印的剩余部分中不会有 4 个字符串，但它可能#在其他上下文中包含 s。

它还假设要打印的操作部分始终以大写字母开头，因为否则我看不到任何方法来知道我们需要根据预期输出d从输入的第 3 行开头删除。dEngine ...

-E您可以对 sed 执行相同的操作，例如使用支持ERE（例如 GNU 或 BSD sed）的sed ：

$ sed -E 's/[0-9]+\..*(#[0-9]{3}){4}[^[:upper:]]*//' file
Sep 17 20:47:02 Fuel has High Warning
Sep 20 12:02:20 Genset Started
Sep 20 12:09:31 Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 20 12:10:27 Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Most Severe Level
Sep 20 12:10:28 Genset Stopped
Sep 23 09:19:03 Genset Not In Auto
Sep 23 09:19:04 All Faults Cleared
Sep 23 09:19:13 Genset in Auto
Sep 30 12:33:48 Genset Not In Auto
Sep 30 12:33:55 Genset Started
Sep 30 12:41:14 Engine Coolant Temperature - Data Valid but Above Normal Operational Range - Moderately Severe Level
Sep 30 12:41:56 Genset Stopped
Sep 30 12:41:58 All Faults Cleared
Oct  3 14:00:52 Genset Started
Oct  3 14:15:17 Genset Stopped
Oct  3 14:20:39 Genset in Auto
Oct 15 12:02:55 Genset Started
Oct 15 12:12:50 Genset Stopped

相关内容