我正在尝试配置 rsyslog 将日志发送到 logstash 然后将其转发到 elasticsearch。
/etc/rsyslog.d/60-output.conf我创建了一个包含以下内容的配置文件：
*.* @localhost:10514;json-template
/etc/rsyslog.d/01-json-template.conf以及包含以下内容的模板文件：
template(name="json-template"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"@version\":\"1")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\",\"sysloghost\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"programname\":\"") property(name="programname")
constant(value="\",\"procid\":\"") property(name="procid")
constant(value="\"}\n")
}
然后我重启了 rsyslog 服务。并且我为 logstash 创建了一个配置文件，/etc/logstash/conf.d/logstash.conf内容如下：
input {
udp {
port => 10514
codec => "json"
type => "rsyslog"
}
}
filter { }
output {
if [type] == "rsyslog" {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}
}
然后我重新启动了 logstash。
当我跑步时sudo netstat -tulpn | grep 10514我得到了这个：
user@rsyslog-server:/var/log$ sudo netstat -tulpn | grep 10514
udp 0 0 0.0.0.0:10514 0.0.0.0:* 5327/java
因此 Logstash 正在监听端口 10514。
为了验证 elasticsearch 输入，我运行了 curl -XGET 'http://localhost:9200/logstash-*/_search?q=*&pretty'但它没有返回任何结果：
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 0,
"successful" : 0,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : 0.0,
"hits" : [ ]
}
}
我在用着：
- rsyslogd 8.2208.0（又名 2022.08）
- logstash 7.17.8
- elasticsearch 7.17.8
我该如何解决这个问题？
编辑：
我按如下方式运行 logstash 来查看日志：
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --verbose
输出结果如下：
sing bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2023-01-01 18:50:48.417 [main] runner - DEPRECATION WARNING: The flag ["--verbose"] has been deprecated, please use "--log.level=info" instead.
[INFO ] 2023-01-01 18:50:48.423 [main] runner - Starting Logstash {"logstash.version"=>"7.17.8", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.17+8 on 11.0.17+8 +indy +jit [linux-x86_64]"}
[INFO ] 2023-01-01 18:50:48.426 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[WARN ] 2023-01-01 18:50:48.689 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2023-01-01 18:50:49.682 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2023-01-01 18:50:50.739 [Converge PipelineAction::Create<main>] Reflections - Reflections took 50 ms to scan 1 urls, producing 119 keys and 419 values
[WARN ] 2023-01-01 18:50:51.350 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-01-01 18:50:51.396 [Converge PipelineAction::Create<main>] udp - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-01-01 18:50:51.438 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-01-01 18:50:51.479 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2023-01-01 18:50:51.658 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[INFO ] 2023-01-01 18:50:51.953 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ] 2023-01-01 18:50:52.174 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2023-01-01 18:50:52.199 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (7.17.8) {:es_version=>7}
[WARN ] 2023-01-01 18:50:52.200 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2023-01-01 18:50:52.276 [[main]-pipeline-manager] elasticsearch - Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[INFO ] 2023-01-01 18:50:52.362 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x332001c0 run>"}
[INFO ] 2023-01-01 18:50:52.380 [Ruby-0-Thread-10: :1] elasticsearch - Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[INFO ] 2023-01-01 18:50:53.026 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.66}
[INFO ] 2023-01-01 18:50:53.093 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[WARN ] 2023-01-01 18:50:53.158 [[main]<udp] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-01-01 18:50:53.186 [[main]<udp] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2023-01-01 18:50:53.188 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2023-01-01 18:50:53.206 [[main]<udp] udp - Starting UDP listener {:address=>"0.0.0.0:10514"}
[INFO ] 2023-01-01 18:50:53.244 [[main]<udp] udp - UDP listener started {:address=>"0.0.0.0:10514", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
^C[WARN ] 2023-01-01 19:02:34.721 [SIGINT handler] runner - SIGINT received. Shutting down.
[INFO ] 2023-01-01 19:02:35.721 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[INFO ] 2023-01-01 19:02:35.798 [Converge PipelineAction::StopAndDelete<main>] pipelinesregistry - Removed pipeline from registry successfully {:pipeline_id=>:main}
[INFO ] 2023-01-01 19:02:35.873 [LogStash::Runner] runner - Logstash shut down.
sudo journalctl-u logstash-f：
janv. 01 18:50:32 rsyslog-server systemd[1]: logstash.service: Deactivated successfully.
janv. 01 18:50:32 rsyslog-server systemd[1]: Stopped logstash.
janv. 01 18:50:32 rsyslog-server systemd[1]: logstash.service: Consumed 1min 31.847s CPU time.
janv. 01 19:04:10 rsyslog-server systemd[1]: Started logstash.
janv. 01 19:04:10 rsyslog-server logstash[16111]: Using bundled JDK: /usr/share/logstash/jdk
janv. 01 19:04:10 rsyslog-server logstash[16111]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
janv. 01 19:04:22 rsyslog-server logstash[16111]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
janv. 01 19:04:22 rsyslog-server logstash[16111]: [2023-01-01T19:04:22,731][INFO ][logstash.runner ] Log4j configuration path used is: /etc/logstash/log4j2.properties
janv. 01 19:04:22 rsyslog-server logstash[16111]: [2023-01-01T19:04:22,740][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.17.8", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.17+8 on 11.0.17+8 +indy +jit [linux-x86_64]"}
janv. 01 19:04:22 rsyslog-server logstash[16111]: [2023-01-01T19:04:22,741][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
janv. 01 19:04:24 rsyslog-server logstash[16111]: [2023-01-01T19:04:24,002][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
janv. 01 19:04:24 rsyslog-server logstash[16111]: [2023-01-01T19:04:24,940][INFO ][org.reflections.Reflections] Reflections took 56 ms to scan 1 urls, producing 119 keys and 419 values
janv. 01 19:04:25 rsyslog-server logstash[16111]: [2023-01-01T19:04:25,731][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
janv. 01 19:04:25 rsyslog-server logstash[16111]: [2023-01-01T19:04:25,965][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
janv. 01 19:04:26 rsyslog-server logstash[16111]: [2023-01-01T19:04:26,138][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
janv. 01 19:04:26 rsyslog-server logstash[16111]: [2023-01-01T19:04:26,148][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.17.8) {:es_version=>7}
janv. 01 19:04:26 rsyslog-server logstash[16111]: [2023-01-01T19:04:26,153][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
janv. 01 19:04:26 rsyslog-server logstash[16111]: [2023-01-01T19:04:26,206][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
janv. 01 19:04:26 rsyslog-server logstash[16111]: [2023-01-01T19:04:26,256][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
janv. 01 19:04:26 rsyslog-server logstash[16111]: [2023-01-01T19:04:26,276][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x53d2107b run>"}
janv. 01 19:04:26 rsyslog-server logstash[16111]: [2023-01-01T19:04:26,466][INFO ][logstash.outputs.elasticsearch][main] Created rollover alias {:name=>"<logstash-{now/d}-000001>"}
janv. 01 19:04:27 rsyslog-server logstash[16111]: [2023-01-01T19:04:27,008][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.73}
janv. 01 19:04:27 rsyslog-server logstash[16111]: [2023-01-01T19:04:27,069][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
janv. 01 19:04:27 rsyslog-server logstash[16111]: [2023-01-01T19:04:27,127][INFO ][logstash.inputs.udp ][main][9e1b2cf1672d2dd22a032525fa695bdf0d4d43163b1fd297f3b99f01f316ae38] Starting UDP listener {:address=>"0.0.0.0:10514"}
janv. 01 19:04:27 rsyslog-server logstash[16111]: [2023-01-01T19:04:27,157][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
janv. 01 19:04:27 rsyslog-server logstash[16111]: [2023-01-01T19:04:27,160][INFO ][logstash.inputs.udp ][main][9e1b2cf1672d2dd22a032525fa695bdf0d4d43163b1fd297f3b99f01f316ae38] UDP listener started {:address=>"0.0.0.0:10514", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
^C