Ubuntu環境部署Logstash實戰案例

　　　　　　　　　　　　Ubuntu環境部署Logstash實戰案例

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正傑

版權聲明：原創作品，謝絕轉載！否則將追究法律責任。

 

 

 

一.准備環境

1>.部署環境說明

　　Logstash可以單獨找一台機器部署，它需要安裝JDK環境，我這里為了省事，就直接和一台Elasticsearch節點復用同一個節點(es103.yinzhengjie.com)。

　　博主推薦閱讀:
　　　　https://www.cnblogs.com/yinzhengjie2020/p/12953504.html

2>.下載kibana軟件包

　　博主推薦閱讀:
　　　　https://www.cnblogs.com/yinzhengjie2020/p/12934518.html

 

二.部署Logstash實操

1>.將下載的Logstash軟件包上傳到es103.yinzhengjie.com節點並安裝

[[e-mail] ~]# dpkg -i logstash-6.8.9.deb 
Selecting previously unselected package logstash.
(Reading database ... 103047 files and directories currently installed.)
Preparing to unpack logstash-6.8.9.deb ...
Unpacking logstash (1:6.8.9-1) ...
Setting up logstash (1:6.8.9-1) ...
Using provided startup.options file: /etc/logstash/startup.options
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash
[[e-mail] ~]# 

2>.查看logstath的啟動腳本

[[e-mail] ~]# find / -name logstash.service
/etc/systemd/system/logstash.service
[[e-mail] ~]# 

[[e-mail] ~]# cat /etc/systemd/system/logstash.service
[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
[[e-mail] ~]# 

[[e-mail] ~]# vim /etc/systemd/system/logstash.service
[[e-mail] ~]# 
[[e-mail] ~]# cat /etc/systemd/system/logstash.service
[Unit]
Description=logstash

[Service]
Type=simple
User=root
Group=root
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
[[e-mail] ~]# 

 

三.以標准輸入為數據源測試Logstash可用性

1>.輸出類型為標准輸出案例

[[e-mail] ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug }}'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-04 03:01:56.765 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-06-04 03:01:56.776 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"}
[INFO ] 2020-06-04 03:02:01.365 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc
h.delay"=>50}[INFO ] 2020-06-04 03:02:01.473 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x19207ef3 run>"}
The stdin plugin is now waiting for input:
[INFO ] 2020-06-04 03:02:01.540 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin
es=>[]}[INFO ] 2020-06-04 03:02:01.778 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
尹正傑到此一游!　　　　　　　　　　　　　　　　　　　　　　　　　　#這是我輸出的數據
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
      "@version" => "1",　　　　　　　　　　　　　　　　　　　　#事件版本號，一個事件就是一個ruby對象
    "@timestamp" => 2020-06-04T03:02:40.589Z,　　　　　　　 #當前事件的發生時間
       "message" => "尹正傑到此一游!",　　　　　　　　　　　　 #消息的具體內容
          "host" => "es103.yinzhengjie.com"　　　　　　　　#標記時間發生在哪個主機
}

2>.輸出類型為文件案例

[[e-mail] ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { file {path => "/tmp/log.txt"}}'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-04 03:32:00.760 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-06-04 03:32:00.771 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"}
[INFO ] 2020-06-04 03:32:05.126 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc
h.delay"=>50}[INFO ] 2020-06-04 03:32:05.258 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4b0b27c1 run>"}
The stdin plugin is now waiting for input:
[INFO ] 2020-06-04 03:32:05.319 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin
es=>[]}[INFO ] 2020-06-04 03:32:05.522 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
尹正傑到此一游!
[INFO ] 2020-06-04 03:33:48.228 [[main]>worker1] file - Opening file {:path=>"/tmp/log.txt"}
https://www.cnblogs.com/yinzhengjie/

[[e-mail] ~]# tail -10f /tmp/log.txt 
{"@timestamp":"2020-06-04T03:33:47.887Z","host":"es103.yinzhengjie.com","message":"尹正傑到此一游!","@version":"1"}
{"@timestamp":"2020-06-04T03:34:28.523Z","host":"es103.yinzhengjie.com","message":"https://www.cnblogs.com/yinzhengjie/","@version":"1"}

3>.輸出類型為Elasticsearch案例

[[e-mail] ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch {hosts=> ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:920
0"] index=> "yinzhengjie-log-%{+YYYY.MM.dd}"}}'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-04 03:43:44.554 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-06-04 03:43:44.567 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"}
[INFO ] 2020-06-04 03:43:48.816 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc
h.delay"=>50}[INFO ] 2020-06-04 03:43:49.205 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es101.yinzhengjie.com:9200/, http://es102
.yinzhengjie.com:9200/]}}[WARN ] 2020-06-04 03:43:49.422 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es101.yinzhengjie.com:9200/"}
[INFO ] 2020-06-04 03:43:49.561 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6}
[WARN ] 2020-06-04 03:43:49.563 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version
=>6}[WARN ] 2020-06-04 03:43:49.575 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es102.yinzhengjie.com:9200/"}
[INFO ] 2020-06-04 03:43:49.626 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es101.yinzhengjie.com:9200",
 "http://es102.yinzhengjie.com:9200"]}[INFO ] 2020-06-04 03:43:49.651 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template
[INFO ] 2020-06-04 03:43:49.678 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refr
esh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[INFO ] 2020-06-04 03:43:49.720 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x696425bf run>"}
[INFO ] 2020-06-04 03:43:49.774 [Ruby-0-Thread-5: :1] elasticsearch - Installing elasticsearch template to _template/logstash
The stdin plugin is now waiting for input:
[INFO ] 2020-06-04 03:43:49.807 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin
es=>[]}[INFO ] 2020-06-04 03:43:50.234 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
尹正傑到此一游~
https://www.cnblogs.com/yinzhengjie/

　　數據是否寫入成功我們在Kibana無法直接查看到，我們可以通過Elasticsearch的head插件看到。但這並不說Kibana無法查看剛剛通過Logstash寫入的數據，而是需要創建索引。

 

四.Kibana創建索引

1>.打開Kibana的WebUI，如下圖所示，依次點擊"管理" ---> "索引模式"

2>.如下圖所示，點擊"創建索引模式"

3>.如下圖所示，輸入匹配模式后，點擊"下一步"

4>.如下圖所示，設置篩選時間的名稱為"timestamp"即可

5>.索引創建成功

6>.點擊Discover，查看剛剛創建的索引內容

 

五.博主推薦閱讀