![Logstash grok 匹配模式不起作用,尽管 grok 构造函数说没问题](https://linux22.com/image/674428/Logstash%20grok%20%E5%8C%B9%E9%85%8D%E6%A8%A1%E5%BC%8F%E4%B8%8D%E8%B5%B7%E4%BD%9C%E7%94%A8%EF%BC%8C%E5%B0%BD%E7%AE%A1%20grok%20%E6%9E%84%E9%80%A0%E5%87%BD%E6%95%B0%E8%AF%B4%E6%B2%A1%E9%97%AE%E9%A2%98.png)
我一直在构建一些 grok 模式来解析 /var/log/secure 日志文件,一切运行正常。我创建了 grok 模式http://grokconstructor.appspot.com/然后测试它们http://grokdebug.herokuapp.com/两个站点都显示模式完全匹配。我使用的是 logstash 2.1.1、elasticsearch 2.1.1 和 kibana 4.3.1,均在 CentOS 7.1 上运行,使用 JAVA openjdk 1.8.0.65-2.b17。
然后,我采用了这些模式,并在 logstash 服务器上使用过滤器实现了它们。大多数过滤器工作正常,但出于某种原因,SECURENETREG 和 SECURENETBADGE 不匹配。logstash --configtest 显示没有问题,并且 logstash 运行正常,但当我在 Kibana 中查看应该与这些模式匹配的条目时,似乎没有任何解析工作。
这是我的模式文件,/etc/logstash/patterns.d/secure-log.grok:
SECURETIMESTAMP %{MONTH}%{SPACE}%{MONTHDAY} %{TIME}
SECUREPROG %{PROG:program}(?:\[%{POSINT:pid}\])?
SECUREHOST %{IPORHOST:host}
SECUREBASE %{SECURETIMESTAMP:secure_timestamp}%{SPACE}%{SECUREHOST}%{SPACE}%{SECUREPROG}:
SECURESU %{SECUREBASE} (runuser: |)%{PROG:pam_program}(?:\[%{POSINT:pid}\])? session %{WORD} for user %{USER:user}( by \(uid=%{NUMBER:su_caller_uid}\)|)
SECURESUDORUN %{SECUREBASE}%{SPACE}(%{USER:user} : TTY=%{DATA} ; PWD=%{DATA} ; USER=%{USER:sudo_runas_user} ; COMMAND=%{GREEDYDATA:sudo_command}|\S+:%{SPACE}(TGT verified|error reading keytab %{GREEDYDATA}|authentication %{WORD} for '%{USER:user}'%{GREEDYDATA}|%{GREEDYDATA}user=%{USER:user}))
SECURESSHDPUBKEY %{SECUREBASE} (Found matching RSA key: %{GREEDYDATA:rsa_key}|%{WORD} publickey for %{USER:user} from %{IPORHOST:src_ip}( port %{NUMBER:port} %{WORD}( \[preauth\]|: RSA %{GREEDYDATA:rsa_key}|)|)|)
SECURESSHDREST %{SECUREBASE} (Did not receive identification string from %{IPORHOST:src_ip}|pam_unix\(sshd:session\): session %{WORD} for user %{USER:user}|Starting session: command for %{USER:user} from %{IPORHOST:src_ip}|Connection from %{IPORHOST:src_ip}|Accepted (password|publickey) for %{USER:user} from %{IPORHOST:src_ip}|Received disconnect from %{IPORHOST:src_ip}|Connection closed by %{IPORHOST:src_ip}|User child is on pid %{NUMBER}|Set %{UNIXPATH} to %{NUMBER})
SECURENETREG %{SECUREBASE} connect from %{IPORHOST:src_ip}%{GREEDYDATA}
SECURENETBADGE %{SECUREBASE} (%{WORD:whois_action}|Authentication %{WORD:auth_result}): (reply from %{URI:whois_uri}: Result: %{GREEDYDATA:whois_result}|User: %{USER:user}, %{GREEDYDATA:auth_result_detail}, From: %{IPORHOST:src_ip}, %{GREEDYDATA}, URL: %{URI:netbadge_source_uri})
这是应用过滤器的配置文件,/etc/logstash/conf.d/46-filter-secure-log.conf:
filter {
if [type] == "secure" {
grok {
patterns_dir => ["/etc/logstash/patterns.d/"]
match => { "message" => [
"%{SECURESU}",
"%{SECURESUDORUN}",
"%{SECURESSHDPUBKEY}",
"%{SECURESSHDREST}",
"%{SECURENETREG}",
"%{SECURENETBADGE}"
]
}
add_field => {
"received_at" => "%{@timestamp}"
"received_from" => "%{host}"
}
}
date {
match => [ "secure_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
最后,这里有一些(已删除识别信息的)日志消息,它们在模式测试器中完全匹配,但似乎无法在 logstash 中得到正确解析(尽管我知道过滤器正在命中它们,因为我通过添加新字段进行了临时测试,只是为了确保消息正在通过过滤器运行):
Jan 8 09:22:22 netbadge-serv netbadge[3534]: verify_whois: reply from https://whois.domain.edu/whois: Result: 0:-1000:0:0:Error with submitted data Illegal characters in data stream#012
Jan 8 09:22:19 netreg-serv autoreg.pl[13867]: connect from 10.250.100.22 (10.250.100.22)
Jan 8 09:22:19 netbadge-serv netbadge[3522]: Authentication success: User: mst3k, Password: Test Test, From: 10.250.28.30, Appid: webmail_login, URL: https://www.mail.domain.edu/switchboard/
Jan 8 09:39:51 netbadge-serv netbadge[11358]: Authentication failure: User: mst3k, Invalid User/Password, From: 10.250.28.31, Appid: Shibboleth Identity Provider, URL: https://shib.domain.edu/idp/Authn/RemoteUser
我确信这是我因为长时间盯着这一切而忽略的一些简单的事情,所以我真的希望有人能告诉我这里发生了什么。
谢谢,
鲍勃
答案1
搞清楚了。在 SECURESSHDPUBKEY 的末尾,结尾 ) 之前有一个 |,它与所有其他内容匹配,因此它永远不会到达下面的模式。我知道这很简单,只是盯着那个 grok 看得太久了。