Logstash使用快速入門


            Logstash使用快速入門

                                 作者:尹正傑

版權聲明:原創作品,謝絕轉載!否則將追究法律責任。

 

 

一.部署Logstash

  博主推薦閱讀:
    https://www.cnblogs.com/yinzhengjie2020/p/13022403.html
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash --help                #查看logstash腳本的幫助信息
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Usage:
    bin/logstash [OPTIONS]

Options:
    -n, --node.name NAME          Specify the name of this logstash instance, if no value is given
                                  it will default to the current hostname.
                                   (default: "es103.yinzhengjie.com")
    -f, --path.config CONFIG_PATH Load the logstash config from a specific file
                                  or directory.  If a directory is given, all
                                  files in that directory will be concatenated
                                  in lexicographical order and then parsed as a
                                  single config file. You can also specify
                                  wildcards (globs) and any matched files will
                                  be loaded in the order described above.
    -e, --config.string CONFIG_STRING Use the given string as the configuration
                                  data. Same syntax as the config file. If no
                                  input is specified, then the following is
                                  used as the default input:
                                  "input { stdin { type => stdin } }"
                                  and if no output is specified, then the
                                  following is used as the default output:
                                  "output { stdout { codec => rubydebug } }"
                                  If you wish to use both defaults, please use
                                  the empty string for the '-e' flag.
                                   (default: nil)
    --field-reference-parser MODE Use the given MODE when parsing field
                                  references.
                                  
                                  The field reference parser is used to expand
                                  field references in your pipeline configs,
                                  and will be becoming more strict to better
                                  handle illegal and ambbiguous inputs in a
                                  future release of Logstash.
                                  
                                  Available MODEs are:
                                   - `LEGACY`: parse with the legacy parser,
                                     which is known to handle ambiguous- and
                                     illegal-syntax in surprising ways;
                                     warnings will not be emitted.
                                   - `COMPAT`: warn once for each distinct
                                     ambiguous- or illegal-syntax input, but
                                     continue to expand field references with
                                     the legacy parser.
                                   - `STRICT`: parse in a strict manner; when
                                     given ambiguous- or illegal-syntax input,
                                     raises a runtime exception that should
                                     be handled by the calling plugin.
                                  
                                   The MODE can also be set with
                                   `config.field_reference.parser`
                                  
                                   (default: "COMPAT")
    --modules MODULES             Load Logstash modules.
                                  Modules can be defined using multiple instances
                                  '--modules module1 --modules module2',
                                     or comma-separated syntax
                                  '--modules=module1,module2'
                                  Cannot be used in conjunction with '-e' or '-f'
                                  Use of '--modules' will override modules declared
                                  in the 'logstash.yml' file.
    -M, --modules.variable MODULES_VARIABLE Load variables for module template.
                                  Multiple instances of '-M' or
                                  '--modules.variable' are supported.
                                  Ignored if '--modules' flag is not used.
                                  Should be in the format of
                                  '-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"'
                                  as in
                                  '-M "example.var.filter.mutate.fieldname=fieldvalue"'
    --setup                       Load index template into Elasticsearch, and saved searches, 
                                  index-pattern, visualizations, and dashboards into Kibana when
                                  running modules.
                                   (default: false)
    --cloud.id CLOUD_ID           Sets the elasticsearch and kibana host settings for
                                  module connections in Elastic Cloud.
                                  Your Elastic Cloud User interface or the Cloud support
                                  team should provide this.
                                  Add an optional label prefix '<label>:' to help you
                                  identify multiple cloud.ids.
                                  e.g. 'staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy'
    --cloud.auth CLOUD_AUTH       Sets the elasticsearch and kibana username and password
                                  for module connections in Elastic Cloud
                                  e.g. 'username:<password>'
    --pipeline.id ID              Sets the ID of the pipeline.
                                   (default: "main")
    -w, --pipeline.workers COUNT  Sets the number of pipeline workers to run.
                                   (default: 2)
    --java-execution              Use Java execution engine.
                                   (default: false)
    -b, --pipeline.batch.size SIZE Size of batches the pipeline is to work in.
                                   (default: 125)
    -u, --pipeline.batch.delay DELAY_IN_MS When creating pipeline batches, how long to wait while polling
                                  for the next event.
                                   (default: 50)
    --pipeline.unsafe_shutdown    Force logstash to exit during shutdown even
                                  if there are still inflight events in memory.
                                  By default, logstash will refuse to quit until all
                                  received events have been pushed to the outputs.
                                   (default: false)
    --path.data PATH              This should point to a writable directory. Logstash
                                  will use this directory whenever it needs to store
                                  data. Plugins will also have access to this path.
                                   (default: "/usr/share/logstash/data")
    -p, --path.plugins PATH       A path of where to find plugins. This flag
                                  can be given multiple times to include
                                  multiple paths. Plugins are expected to be
                                  in a specific directory hierarchy:
                                  'PATH/logstash/TYPE/NAME.rb' where TYPE is
                                  'inputs' 'filters', 'outputs' or 'codecs'
                                  and NAME is the name of the plugin.
                                   (default: [])
    -l, --path.logs PATH          Write logstash internal logs to the given
                                  file. Without this flag, logstash will emit
                                  logs to standard output.
                                   (default: "/usr/share/logstash/logs")
    --log.level LEVEL             Set the log level for logstash. Possible values are:
                                    - fatal
                                    - error
                                    - warn
                                    - info
                                    - debug
                                    - trace
                                   (default: "info")
    --config.debug                Print the compiled config ruby code out as a debug log (you must also have --log.level=debug enabled).
                                  WARNING: This will include any 'password' options passed to plugin configs as plaintext, and may result
                                  in plaintext passwords appearing in your logs!
                                   (default: false)
    -i, --interactive SHELL       Drop to shell instead of running as normal.
                                  Valid shells are "irb" and "pry"
    -V, --version                 Emit the version of logstash and its friends,
                                  then exit.
    -t, --config.test_and_exit    Check configuration for valid syntax and then exit.
                                   (default: false)
    -r, --config.reload.automatic Monitor configuration changes and reload
                                  whenever it is changed.
                                  NOTE: use SIGHUP to manually reload the config
                                   (default: false)
    --config.reload.interval RELOAD_INTERVAL How frequently to poll the configuration location
                                  for changes, in seconds.
                                   (default: 3000000000)
    --http.host HTTP_HOST         Web API binding host (default: "127.0.0.1")
    --http.port HTTP_PORT         Web API http port (default: 9600..9700)
    --log.format FORMAT           Specify if Logstash should write its own logs in JSON form (one
                                  event per line) or in plain text (using Ruby's Object#inspect)
                                   (default: "plain")
    --path.settings SETTINGS_DIR  Directory containing logstash.yml file. This can also be
                                  set through the LS_SETTINGS_DIR environment variable.
                                   (default: "/usr/share/logstash/config")
    --verbose                     Set the log level to info.
                                  DEPRECATED: use --log.level=info instead.
    --debug                       Set the log level to debug.
                                  DEPRECATED: use --log.level=debug instead.
    --quiet                       Set the log level to info.
                                  DEPRECATED: use --log.level=info instead.
    -h, --help                    print help
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash --help                #查看logstash腳本的幫助信息

 

二.input(file) ---> output(stdout)案例

1>.編寫配置文件並檢查語法是否錯誤

[root@es103.yinzhengjie.com ~]# vim /etc/logstash/conf.d/file-stdout.conf
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# cat /etc/logstash/conf.d/file-stdout.conf
input {
    file {
        type => "syslog"
        path => "/var/log/syslog"
        start_position => "beginning"
        stat_interval => 3
    }
}

output {
    stdout {
        codec => "rubydebug"
    }
}
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# vim /etc/logstash/conf.d/file-stdout.conf
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file-stdout.conf -t        #檢查配置文件
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-05 00:27:36.243 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK                #注意哈,如果出現了"Configuration OK"說明配置文件語法正確
[INFO ] 2020-06-05 00:27:40.363 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file-stdout.conf -t        #檢查配置文件

2>.以root身份啟動logstash任務

[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file-stdout.conf          #以root身份啟動logstash進程
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-05 00:35:22.337 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-06-05 00:35:22.347 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"}
[INFO ] 2020-06-05 00:35:27.742 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc
h.delay"=>50}[INFO ] 2020-06-05 00:35:28.106 [[main]-pipeline-manager] file - No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/fil
e/.sincedb_f5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[INFO ] 2020-06-05 00:35:28.151 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x3dddf5cf run>"}
[INFO ] 2020-06-05 00:35:28.229 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin
es=>[]}[INFO ] 2020-06-05 00:35:28.244 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2020-06-05 00:35:28.706 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
......                          #如下所示,"/var/log/syslog"的每一行都被抽取成JSON格式發送給stdout。
{
    "@timestamp" => 2020-06-05T00:33:24.840Z,
          "type" => "syslog",
       "message" => "Jun  5 00:11:42 es103 systemd[3705]: Reached target Basic System.",
          "host" => "es103.yinzhengjie.com",
          "path" => "/var/log/syslog",
      "@version" => "1"
}
{
    "@timestamp" => 2020-06-05T00:33:24.841Z,
          "type" => "syslog",
       "message" => "Jun  5 00:11:42 es103 systemd[3705]: Startup finished in 319ms.",
          "host" => "es103.yinzhengjie.com",
          "path" => "/var/log/syslog",
      "@version" => "1"
}
{
    "@timestamp" => 2020-06-05T00:33:24.841Z,
          "type" => "syslog",
       "message" => "Jun  5 00:17:01 es103 CRON[3934]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)",
          "host" => "es103.yinzhengjie.com",
          "path" => "/var/log/syslog",
      "@version" => "1"
}
......
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file-stdout.conf          #以root身份啟動logstash進程

3>.啟動logstash任務后查看生成的隱藏文件信息

[root@es103.yinzhengjie.com ~]# ll /usr/share/logstash/data/plugins/inputs/file/          #當我們啟動一個input插件為file的logstash任務時,會在該目錄生成相應的配置信息。
total 12
drwxr-xr-x 2 root root 4096 Jun  5 00:36 ./
drwxr-xr-x 3 root root 4096 Jun  5 00:33 ../
-rw-r--r-- 1 root root   58 Jun  5 00:36 .sincedb_f5fdf6ea0ea92860c6a6b2b354bfcbbc
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# cat /usr/share/logstash/data/plugins/inputs/file/.sincedb_f5fdf6ea0ea92860c6a6b2b354bfcbbc       #注意觀察該隱藏文件的內容
5245734 0 2050 1418255 1591317366.7913418 /var/log/syslog
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# ls -il /var/log/syslog                          #不難發現該inode節點信息和logstash的隱藏文件記錄的是一致的。
5245734 -rw-r----- 1 syslog adm 1418255 Jun  5 00:18 /var/log/syslog
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# ll /usr/share/logstash/data/plugins/inputs/file/          #當我們啟動一個input插件為file的logstash任務時,會在該目錄生成相應的配置信息。

 

三.input(file) ---> output(elasticsearch)案例

1>.編寫配置文件並檢查語法是否錯誤

[root@es103.yinzhengjie.com ~]# vim /etc/logstash/conf.d/file-elasticsearch.conf 
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# cat /etc/logstash/conf.d/file-elasticsearch.conf 
input {
    file {
        type => "syslog"
        path => "/var/log/syslog"
        start_position => "beginning"
        stat_interval => 3
    }
}

output {
    elasticsearch {
        hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"]
        index => "syslog-172.200.5.103-%{+YYYY.MM.dd}"
    }
}
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# vim /etc/logstash/conf.d/file-elasticsearch.conf
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file-elasticsearch.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-05 00:52:51.381 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[INFO ] 2020-06-05 00:52:55.289 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file-elasticsearch.conf -t

2>.輸入源文件無權限訪問導致logstash任務運行失敗案例

[root@es103.yinzhengjie.com ~]# ll /var/log/logstash/
total 8
drwxrwxr-x  2 logstash root   4096 May  4 18:27 ./
drwxrwxr-x 11 root     syslog 4096 Jun  4 02:46 ../
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# ll /var/log/syslog                   
-rw-r----- 1 syslog adm 1418255 Jun  5 00:18 /var/log/syslog          #不難發現,該文件對於其它用戶(logstatsh)是沒有訪問權限的
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# systemctl start logstash.service 
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# ll /var/log/logstash/
total 16
drwxrwxr-x  2 logstash root     4096 Jun  5 00:58 ./
drwxrwxr-x 11 root     syslog   4096 Jun  4 02:46 ../
-rw-r--r--  1 logstash logstash 5285 Jun  5 01:03 logstash-plain.log      #一般情況下,我們通過查看該文件就可以看到logstash的日志信息,如果任務失敗在該文件可以找到原因
-rw-r--r--  1 logstash logstash    0 Jun  5 00:58 logstash-slowlog-plain.log
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# tail -10f /var/log/logstash/logstash-plain.log          #相信你不難從WARN日志級別找到"Permission denied"的關鍵詞。
[2020-06-05T00:58:01,397][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[2020-06-05T00:58:01,422][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
[2020-06-05T00:58:01,777][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.8.9"}
[2020-06-05T00:58:01,798][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"d8171294-9203-4745-ab19-e671d626ac67", :path=>"/var/lib/logstash/uuid
"}[2020-06-05T00:58:07,849][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2020-06-05T00:58:08,338][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es101.yinzhengjie.com:9200/, http://es102.yinzheng
jie.com:9200/, http://es103.yinzhengjie.com:9200/]}}[2020-06-05T00:58:08,651][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es101.yinzhengjie.com:9200/"}
[2020-06-05T00:58:08,722][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2020-06-05T00:58:08,725][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2020-06-05T00:58:08,732][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es102.yinzhengjie.com:9200/"}
[2020-06-05T00:58:08,755][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es103.yinzhengjie.com:9200/"}
[2020-06-05T00:58:08,796][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es101.yinzhengjie.com:9200", "http://
es102.yinzhengjie.com:9200", "http://es103.yinzhengjie.com:9200"]}[2020-06-05T00:58:08,815][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2020-06-05T00:58:08,832][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_i
nterval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[2020-06-05T00:58:09,033][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f
5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-05T00:58:09,054][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f
5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-05T00:58:09,083][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x55c99cf3 run>"}
[2020-06-05T00:58:09,138][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2020-06-05T00:58:09,146][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2020-06-05T00:58:09,175][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-06-05T00:58:09,556][WARN ][filewatch.tailmode.handlers.createinitial] failed to open /var/log/syslog: #<Errno::EACCES: Permission denied - /var/log/syslog>, ["org/jruby/RubyIO.java:12
36:in `sysopen'", "org/jruby/RubyFile.java:367:in `initialize'", "org/jruby/RubyIO.java:1155:in `open'"]
[2020-06-05T00:58:09,567][WARN ][filewatch.tailmode.handlers.createinitial] failed to open /var/log/syslog: #<Errno::EACCES: Permission denied - /var/log/syslog>, ["org/jruby/RubyIO.java:12
36:in `sysopen'", "org/jruby/RubyFile.java:367:in `initialize'", "org/jruby/RubyIO.java:1155:in `open'"]
[2020-06-05T00:58:09,613][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
......
[root@es103.yinzhengjie.com ~]# tail -10f /var/log/logstash/logstash-plain.log       #相信你不難從WARN日志級別找到"Permission denied"的關鍵詞。

3>.上一步驟問題解決方案

[root@es103.yinzhengjie.com ~]# ll /var/log/syslog 
-rw-r----- 1 syslog adm 1425400 Jun  5 01:08 /var/log/syslog
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# chmod 644 /var/log/syslog                 #修改日志的權限,使得"logstash"用戶可以訪問
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# ll /var/log/syslog 
-rw-r--r-- 1 syslog adm 1425400 Jun  5 01:08 /var/log/syslog
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# chmod 644 /var/log/syslog                   #修改日志文件的權限,使得"logstash"用戶可以訪問
[root@es103.yinzhengjie.com ~]# systemctl restart logstash.service               #將源文件的權限配置為644后,需要重啟服務,Logstash就回去加載"/etc/logstash/conf.d"目錄下的所有配置文件並運行相應的job。
[root@es103.yinzhengjie.com ~]# tail -10f /var/log/logstash/logstash-plain.log                          #再次查看日志就可以看到正常的logstash收集數據的信息啦~
[2020-06-05T01:13:54,918][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es101.yinzhengjie.com:9200", "http://
es102.yinzhengjie.com:9200", "http://es103.yinzhengjie.com:9200"]}[2020-06-05T01:13:54,942][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2020-06-05T01:13:54,992][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_i
nterval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[2020-06-05T01:13:55,226][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f
5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-05T01:13:55,256][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f
5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-05T01:13:55,291][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2bea6da8 run>"}
[2020-06-05T01:13:55,333][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2020-06-05T01:13:55,335][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2020-06-05T01:13:55,406][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-06-05T01:13:55,913][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[root@es103.yinzhengjie.com ~]# tail -10f /var/log/logstash/logstash-plain.log        #再次查看日志就可以看到正常的logstash收集數據的信息啦~

 

四.多個輸入源和多個輸出源(多日志if語句使用)案例

1>.編寫配置文件並檢查語法是否錯誤

[root@es103.yinzhengjie.com ~]# vim /etc/logstash/conf.d/multiple-file-elasticsearch.conf 
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# cat /etc/logstash/conf.d/multiple-file-elasticsearch.conf 
input {
    file {
        type => "syslog"
        path => "/var/log/syslog"
        start_position => "beginning"
        stat_interval => 3
    }

    
    file {
        type => "nginx-log"
        path => "/var/log/nginx/access.log"
        start_position => "beginning"
        stat_interval => 3
    }
}

output {
    if [type] == "syslog" {
        elasticsearch {
            hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"]
            index => "syslog-172.200.5.103-%{+YYYY.MM.dd}"
        }
       
        file {
            path => "/tmp/syslog.txt"
        }
    }

    if [type] == "nginx-log" {
        elasticsearch {
            hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"]
            index => "nginx-log-172.200.5.103-%{+YYYY.MM.dd}"
        }
    }
}
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# vim /etc/logstash/conf.d/multiple-file-elasticsearch.conf
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/multiple-file-elasticsearch.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-05 01:47:16.051 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[INFO ] 2020-06-05 01:47:29.814 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/multiple-file-elasticsearch.conf -t

2>.安裝Nginx服務並產生測試數據

[root@es103.yinzhengjie.com ~]# apt-get install nginx
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0 libjpeg-turbo8 libjpeg8 libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter
  libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6 libxpm4 nginx-common nginx-core
Suggested packages:
  libgd-tools fcgiwrap nginx-doc ssl-cert
The following NEW packages will be installed:
  fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0 libjpeg-turbo8 libjpeg8 libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter
  libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6 libxpm4 nginx nginx-common nginx-core
0 upgraded, 18 newly installed, 0 to remove and 79 not upgraded.
Need to get 2,462 kB of archives.
After this operation, 8,210 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libjpeg-turbo8 amd64 1.5.2-0ubuntu5.18.04.3 [110 kB]
Get:2 http://mirrors.aliyun.com/ubuntu bionic/main amd64 fonts-dejavu-core all 2.37-1 [1,041 kB]
Get:3 http://mirrors.aliyun.com/ubuntu bionic/main amd64 fontconfig-config all 2.12.6-0ubuntu2 [55.8 kB]
Get:4 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libfontconfig1 amd64 2.12.6-0ubuntu2 [137 kB]
Get:5 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libjpeg8 amd64 8c-2ubuntu8 [2,194 B]
Get:6 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libjbig0 amd64 2.1-3.1build1 [26.7 kB]
Get:7 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libtiff5 amd64 4.0.9-5ubuntu0.3 [153 kB]
Get:8 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libwebp6 amd64 0.6.1-2 [185 kB]
Get:9 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libxpm4 amd64 1:3.5.12-1 [34.0 kB]
Get:10 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libgd3 amd64 2.2.5-4ubuntu0.4 [119 kB]
Get:11 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 nginx-common all 1.14.0-0ubuntu1.7 [37.4 kB]
Get:12 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-http-geoip amd64 1.14.0-0ubuntu1.7 [11.2 kB]
Get:13 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-http-image-filter amd64 1.14.0-0ubuntu1.7 [14.6 kB]
Get:14 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-http-xslt-filter amd64 1.14.0-0ubuntu1.7 [13.0 kB]
Get:15 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-mail amd64 1.14.0-0ubuntu1.7 [41.8 kB]
Get:16 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-stream amd64 1.14.0-0ubuntu1.7 [63.7 kB]
Get:17 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 nginx-core amd64 1.14.0-0ubuntu1.7 [413 kB]
Get:18 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 nginx all 1.14.0-0ubuntu1.7 [3,596 B]
Fetched 2,462 kB in 4s (635 kB/s)
Preconfiguring packages ...
Selecting previously unselected package libjpeg-turbo8:amd64.
(Reading database ... 119418 files and directories currently installed.)
Preparing to unpack .../00-libjpeg-turbo8_1.5.2-0ubuntu5.18.04.3_amd64.deb ...
Unpacking libjpeg-turbo8:amd64 (1.5.2-0ubuntu5.18.04.3) ...
Selecting previously unselected package fonts-dejavu-core.
Preparing to unpack .../01-fonts-dejavu-core_2.37-1_all.deb ...
Unpacking fonts-dejavu-core (2.37-1) ...
Selecting previously unselected package fontconfig-config.
Preparing to unpack .../02-fontconfig-config_2.12.6-0ubuntu2_all.deb ...
Unpacking fontconfig-config (2.12.6-0ubuntu2) ...
Selecting previously unselected package libfontconfig1:amd64.
Preparing to unpack .../03-libfontconfig1_2.12.6-0ubuntu2_amd64.deb ...
Unpacking libfontconfig1:amd64 (2.12.6-0ubuntu2) ...
Selecting previously unselected package libjpeg8:amd64.
Preparing to unpack .../04-libjpeg8_8c-2ubuntu8_amd64.deb ...
Unpacking libjpeg8:amd64 (8c-2ubuntu8) ...
Selecting previously unselected package libjbig0:amd64.
Preparing to unpack .../05-libjbig0_2.1-3.1build1_amd64.deb ...
Unpacking libjbig0:amd64 (2.1-3.1build1) ...
Selecting previously unselected package libtiff5:amd64.
Preparing to unpack .../06-libtiff5_4.0.9-5ubuntu0.3_amd64.deb ...
Unpacking libtiff5:amd64 (4.0.9-5ubuntu0.3) ...
Selecting previously unselected package libwebp6:amd64.
Preparing to unpack .../07-libwebp6_0.6.1-2_amd64.deb ...
Unpacking libwebp6:amd64 (0.6.1-2) ...
Selecting previously unselected package libxpm4:amd64.
Preparing to unpack .../08-libxpm4_1%3a3.5.12-1_amd64.deb ...
Unpacking libxpm4:amd64 (1:3.5.12-1) ...
Selecting previously unselected package libgd3:amd64.
Preparing to unpack .../09-libgd3_2.2.5-4ubuntu0.4_amd64.deb ...
Unpacking libgd3:amd64 (2.2.5-4ubuntu0.4) ...
Selecting previously unselected package nginx-common.
Preparing to unpack .../10-nginx-common_1.14.0-0ubuntu1.7_all.deb ...
Unpacking nginx-common (1.14.0-0ubuntu1.7) ...
Selecting previously unselected package libnginx-mod-http-geoip.
Preparing to unpack .../11-libnginx-mod-http-geoip_1.14.0-0ubuntu1.7_amd64.deb ...
Unpacking libnginx-mod-http-geoip (1.14.0-0ubuntu1.7) ...
Selecting previously unselected package libnginx-mod-http-image-filter.
Preparing to unpack .../12-libnginx-mod-http-image-filter_1.14.0-0ubuntu1.7_amd64.deb ...
Unpacking libnginx-mod-http-image-filter (1.14.0-0ubuntu1.7) ...
Selecting previously unselected package libnginx-mod-http-xslt-filter.
Preparing to unpack .../13-libnginx-mod-http-xslt-filter_1.14.0-0ubuntu1.7_amd64.deb ...
Unpacking libnginx-mod-http-xslt-filter (1.14.0-0ubuntu1.7) ...
Selecting previously unselected package libnginx-mod-mail.
Preparing to unpack .../14-libnginx-mod-mail_1.14.0-0ubuntu1.7_amd64.deb ...
Unpacking libnginx-mod-mail (1.14.0-0ubuntu1.7) ...
Selecting previously unselected package libnginx-mod-stream.
Preparing to unpack .../15-libnginx-mod-stream_1.14.0-0ubuntu1.7_amd64.deb ...
Unpacking libnginx-mod-stream (1.14.0-0ubuntu1.7) ...
Selecting previously unselected package nginx-core.
Preparing to unpack .../16-nginx-core_1.14.0-0ubuntu1.7_amd64.deb ...
Unpacking nginx-core (1.14.0-0ubuntu1.7) ...
Selecting previously unselected package nginx.
Preparing to unpack .../17-nginx_1.14.0-0ubuntu1.7_all.deb ...
Unpacking nginx (1.14.0-0ubuntu1.7) ...
Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Setting up libjbig0:amd64 (2.1-3.1build1) ...
Setting up fonts-dejavu-core (2.37-1) ...
Setting up nginx-common (1.14.0-0ubuntu1.7) ...
Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /lib/systemd/system/nginx.service.
Setting up libjpeg-turbo8:amd64 (1.5.2-0ubuntu5.18.04.3) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for systemd (237-3ubuntu10.38) ...
Setting up libnginx-mod-mail (1.14.0-0ubuntu1.7) ...
Setting up libxpm4:amd64 (1:3.5.12-1) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Setting up libnginx-mod-http-xslt-filter (1.14.0-0ubuntu1.7) ...
Setting up libnginx-mod-http-geoip (1.14.0-0ubuntu1.7) ...
Setting up libwebp6:amd64 (0.6.1-2) ...
Setting up libjpeg8:amd64 (8c-2ubuntu8) ...
Setting up fontconfig-config (2.12.6-0ubuntu2) ...
Setting up libnginx-mod-stream (1.14.0-0ubuntu1.7) ...
Setting up libtiff5:amd64 (4.0.9-5ubuntu0.3) ...
Setting up libfontconfig1:amd64 (2.12.6-0ubuntu2) ...
Setting up libgd3:amd64 (2.2.5-4ubuntu0.4) ...
Setting up libnginx-mod-http-image-filter (1.14.0-0ubuntu1.7) ...
Setting up nginx-core (1.14.0-0ubuntu1.7) ...
Setting up nginx (1.14.0-0ubuntu1.7) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# apt-get install nginx
[root@es103.yinzhengjie.com ~]# systemctl start nginx
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# ss -ntl
State                  Recv-Q                  Send-Q                                              Local Address:Port                                     Peer Address:Port                  
LISTEN                 0                       128                                                       0.0.0.0:80                                            0.0.0.0:*                     
LISTEN                 0                       128                                                 127.0.0.53%lo:53                                            0.0.0.0:*                     
LISTEN                 0                       128                                                       0.0.0.0:22                                            0.0.0.0:*                     
LISTEN                 0                       128                                                          [::]:80                                               [::]:*                     
LISTEN                 0                       128                                        [::ffff:172.200.5.103]:9200                                                *:*                     
LISTEN                 0                       128                                        [::ffff:172.200.5.103]:9300                                                *:*                     
LISTEN                 0                       128                                                          [::]:22                                               [::]:*                     
LISTEN                 0                       50                                             [::ffff:127.0.0.1]:9600                                                *:*                     
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# systemctl start nginx          #服務啟動成功后,可以進行訪問測試,如下圖所示。
[root@es103.yinzhengjie.com ~]# systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-06-05 01:43:50 UTC; 8min ago
     Docs: man:nginx(8)
  Process: 6237 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 6226 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 6240 (nginx)
    Tasks: 3 (limit: 4632)
   CGroup: /system.slice/nginx.service
           ├─6240 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           ├─6242 nginx: worker process
           └─6243 nginx: worker process

Jun 05 01:43:49 es103.yinzhengjie.com systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 05 01:43:50 es103.yinzhengjie.com systemd[1]: Started A high performance web server and a reverse proxy server.
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# systemctl status nginx
[root@es103.yinzhengjie.com ~]# systemctl enable nginx
Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable nginx
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# systemctl enable nginx
[root@es103.yinzhengjie.com ~]# cat /var/log/nginx/access.log 
172.200.0.1 - - [05/Jun/2020:01:54:33 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
"172.200.0.1 - - [05/Jun/2020:01:54:33 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "http://es103.yinzhengjie.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like G
ecko) Chrome/83.0.4103.61 Safari/537.36"172.200.5.103 - - [05/Jun/2020:01:55:07 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.58.0"
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# cat /var/log/nginx/access.log
[root@es103.yinzhengjie.com ~]# ll /var/log/nginx/access.log 
-rw-r----- 1 www-data adm 511 Jun  5 01:55 /var/log/nginx/access.log
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# chmod 644 /var/log/nginx/access.log 
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# ll /var/log/nginx/access.log 
-rw-r--r-- 1 www-data adm 511 Jun  5 01:55 /var/log/nginx/access.log
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# chmod 644 /var/log/nginx/access.log    #千萬別忘記加執行權限哈~否則無法收集Nginx日志喲~

3>.重啟Elasticsearch服務

[root@es103.yinzhengjie.com ~]# systemctl restart logstash                   #重啟Logstash服務,使得配置文件生效。
[root@es103.yinzhengjie.com ~]# tail -100f /var/log/logstash/logstash-plain.log 
......
[2020-06-05T02:09:03,939][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2020-06-05T02:09:04,383][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es101.yinzhengjie.com:9200/, http://es102.yinzheng
jie.com:9200/, http://es103.yinzhengjie.com:9200/]}}[2020-06-05T02:09:04,580][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es101.yinzhengjie.com:9200/"}
[2020-06-05T02:09:04,631][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2020-06-05T02:09:04,635][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2020-06-05T02:09:04,641][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es102.yinzhengjie.com:9200/"}
[2020-06-05T02:09:04,653][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es103.yinzhengjie.com:9200/"}
[2020-06-05T02:09:04,691][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es101.yinzhengjie.com:9200", "http://
es102.yinzhengjie.com:9200", "http://es103.yinzhengjie.com:9200"]}[2020-06-05T02:09:04,719][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2020-06-05T02:09:04,733][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es101.yinzhengjie.com:9200/, http://es102.yinzheng
jie.com:9200/, http://es103.yinzhengjie.com:9200/]}}[2020-06-05T02:09:04,743][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es101.yinzhengjie.com:9200/"}
[2020-06-05T02:09:04,748][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2020-06-05T02:09:04,749][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2020-06-05T02:09:04,753][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es102.yinzhengjie.com:9200/"}
[2020-06-05T02:09:04,761][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_i
nterval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[2020-06-05T02:09:04,787][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es103.yinzhengjie.com:9200/"}
[2020-06-05T02:09:04,796][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2020-06-05T02:09:04,799][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_i
nterval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[2020-06-05T02:09:04,799][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es101.yinzhengjie.com:9200", "http://
es102.yinzhengjie.com:9200", "http://es103.yinzhengjie.com:9200"]}[2020-06-05T02:09:04,848][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es101.yinzhengjie.com:9200/, http://es102.yinzheng
jie.com:9200/, http://es103.yinzhengjie.com:9200/]}}[2020-06-05T02:09:04,854][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es101.yinzhengjie.com:9200/"}
[2020-06-05T02:09:04,859][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2020-06-05T02:09:04,860][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2020-06-05T02:09:04,864][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es102.yinzhengjie.com:9200/"}
[2020-06-05T02:09:04,880][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es103.yinzhengjie.com:9200/"}
[2020-06-05T02:09:04,887][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2020-06-05T02:09:04,889][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_i
nterval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[2020-06-05T02:09:04,891][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es101.yinzhengjie.com:9200", "http://
es102.yinzhengjie.com:9200", "http://es103.yinzhengjie.com:9200"]}[2020-06-05T02:09:05,228][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f
5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-05T02:09:05,251][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f
5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-05T02:09:05,256][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f
5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-05T02:09:05,263][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_d
883144359d3b4f516b37dba51fab2a2", :path=>["/var/log/nginx/access.log"]}[2020-06-05T02:09:05,304][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x3875545b run>"}
[2020-06-05T02:09:05,331][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2020-06-05T02:09:05,331][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2020-06-05T02:09:05,331][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2020-06-05T02:09:05,363][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2020-06-05T02:09:05,473][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-06-05T02:09:06,734][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-06-05T02:09:07,267][INFO ][logstash.outputs.file    ] Opening file {:path=>"/tmp/syslog.txt"}
[root@es103.yinzhengjie.com ~]# tail -100f /var/log/logstash/logstash-plain.log      #記得查看日志變化喲~

4>.在Kibana界面上添加索引

  關於添加索引的步驟我之前有演示過,這里就不羅嗦了,直接上圖。

  博主推薦閱讀:
    https://www.cnblogs.com/yinzhengjie2020/p/13022403.html

5>.查看"/tmp/syslog.txt"是否生成

[root@es103.yinzhengjie.com ~]# ll -h /tmp/syslog.txt             #很明顯,該文件已經存在啦~
-rw-r--r-- 1 logstash logstash 646M Jun  5 02:31 /tmp/syslog.txt
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# ll -h /tmp/syslog.txt             #很明顯,該文件已經存在啦~

 

五.收集tomcat日志案例

1>.安裝tomcat

[root@es102.yinzhengjie.com ~]# wget https://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-8/v8.5.55/bin/apache-tomcat-8.5.55.tar.gz
--2020-06-05 04:33:38--  https://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-8/v8.5.55/bin/apache-tomcat-8.5.55.tar.gz
Resolving mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)... 101.6.8.193, 2402:f000:1:408:8100::1
Connecting to mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)|101.6.8.193|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10371538 (9.9M) [application/x-gzip]
Saving to: ‘apache-tomcat-8.5.55.tar.gz’

apache-tomcat-8.5.55.tar.gz                     100%[====================================================================================================>]   9.89M  2.93MB/s    in 3.4s    

2020-06-05 04:33:41 (2.93 MB/s) - ‘apache-tomcat-8.5.55.tar.gz’ saved [10371538/10371538]

[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# wget https://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-8/v8.5.55/bin/apache-tomcat-8.5.55.tar.gz        #下載tomcat軟件包
[root@es102.yinzhengjie.com ~]# ls
apache-tomcat-8.5.55.tar.gz
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# tar -zxf apache-tomcat-8.5.55.tar.gz -C /yinzhengjie/softwares/
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# ll /yinzhengjie/softwares/
total 0
drwxr-xr-x 4 root root  54 Jun  5 04:34 ./
drwxr-xr-x 4 root root  35 Jun  3 02:53 ../
drwxr-xr-x 9 root root 220 Jun  5 04:34 apache-tomcat-8.5.55/
drwxr-xr-x 7 uucp  143 245 Dec 15  2018 jdk1.8.0_201/
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# tar -zxf apache-tomcat-8.5.55.tar.gz -C /yinzhengjie/softwares/
[root@es102.yinzhengjie.com ~]# /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/catalina.sh start
Using CATALINA_BASE:   /yinzhengjie/softwares/apache-tomcat-8.5.55
Using CATALINA_HOME:   /yinzhengjie/softwares/apache-tomcat-8.5.55
Using CATALINA_TMPDIR: /yinzhengjie/softwares/apache-tomcat-8.5.55/temp
Using JRE_HOME:        /yinzhengjie/softwares/jdk1.8.0_201/jre
Using CLASSPATH:       /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/bootstrap.jar:/yinzhengjie/softwares/apache-tomcat-8.5.55/bin/tomcat-juli.jar
Tomcat started.
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# ss -ntl
State                  Recv-Q                  Send-Q                                              Local Address:Port                                     Peer Address:Port                  
LISTEN                 0                       128                                                 127.0.0.53%lo:53                                            0.0.0.0:*                     
LISTEN                 0                       128                                                       0.0.0.0:22                                            0.0.0.0:*                     
LISTEN                 0                       100                                                             *:8080                                                *:*                     
LISTEN                 0                       128                                        [::ffff:172.200.5.102]:9200                                                *:*                     
LISTEN                 0                       128                                        [::ffff:172.200.5.102]:9300                                                *:*                     
LISTEN                 0                       128                                                          [::]:22                                               [::]:*                     
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/catalina.sh start            #啟動tomcat服務
  1 [root@es102.yinzhengjie.com ~]# cat /yinzhengjie/softwares/apache-tomcat-8.5.55/conf/server.xml 
  2 <?xml version="1.0" encoding="UTF-8"?>
  3 <!--
  4   Licensed to the Apache Software Foundation (ASF) under one or more
  5   contributor license agreements.  See the NOTICE file distributed with
  6   this work for additional information regarding copyright ownership.
  7   The ASF licenses this file to You under the Apache License, Version 2.0
  8   (the "License"); you may not use this file except in compliance with
  9   the License.  You may obtain a copy of the License at
 10 
 11       http://www.apache.org/licenses/LICENSE-2.0
 12 
 13   Unless required by applicable law or agreed to in writing, software
 14   distributed under the License is distributed on an "AS IS" BASIS,
 15   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 16   See the License for the specific language governing permissions and
 17   limitations under the License.
 18 -->
 19 <!-- Note:  A "Server" is not itself a "Container", so you may not
 20      define subcomponents such as "Valves" at this level.
 21      Documentation at /docs/config/server.html
 22  -->
 23 <Server port="8005" shutdown="SHUTDOWN">
 24   <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
 25   <!-- Security listener. Documentation at /docs/config/listeners.html
 26   <Listener className="org.apache.catalina.security.SecurityListener" />
 27   -->
 28   <!--APR library loader. Documentation at /docs/apr.html -->
 29   <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
 30   <!-- Prevent memory leaks due to use of particular java/javax APIs-->
 31   <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
 32   <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
 33   <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
 34 
 35   <!-- Global JNDI resources
 36        Documentation at /docs/jndi-resources-howto.html
 37   -->
 38   <GlobalNamingResources>
 39     <!-- Editable user database that can also be used by
 40          UserDatabaseRealm to authenticate users
 41     -->
 42     <Resource name="UserDatabase" auth="Container"
 43               type="org.apache.catalina.UserDatabase"
 44               description="User database that can be updated and saved"
 45               factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
 46               pathname="conf/tomcat-users.xml" />
 47   </GlobalNamingResources>
 48 
 49   <!-- A "Service" is a collection of one or more "Connectors" that share
 50        a single "Container" Note:  A "Service" is not itself a "Container",
 51        so you may not define subcomponents such as "Valves" at this level.
 52        Documentation at /docs/config/service.html
 53    -->
 54   <Service name="Catalina">
 55 
 56     <!--The connectors can use a shared executor, you can define one or more named thread pools-->
 57     <!--
 58     <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
 59         maxThreads="150" minSpareThreads="4"/>
 60     -->
 61 
 62 
 63     <!-- A "Connector" represents an endpoint by which requests are received
 64          and responses are returned. Documentation at :
 65          Java HTTP Connector: /docs/config/http.html
 66          Java AJP  Connector: /docs/config/ajp.html
 67          APR (HTTP/AJP) Connector: /docs/apr.html
 68          Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
 69     -->
 70     <Connector port="8080" protocol="HTTP/1.1"
 71                connectionTimeout="20000"
 72                redirectPort="8443" />
 73     <!-- A "Connector" using the shared thread pool-->
 74     <!--
 75     <Connector executor="tomcatThreadPool"
 76                port="8080" protocol="HTTP/1.1"
 77                connectionTimeout="20000"
 78                redirectPort="8443" />
 79     -->
 80     <!-- Define an SSL/TLS HTTP/1.1 Connector on port 8443
 81          This connector uses the NIO implementation. The default
 82          SSLImplementation will depend on the presence of the APR/native
 83          library and the useOpenSSL attribute of the
 84          AprLifecycleListener.
 85          Either JSSE or OpenSSL style configuration may be used regardless of
 86          the SSLImplementation selected. JSSE style configuration is used below.
 87     -->
 88     <!--
 89     <Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
 90                maxThreads="150" SSLEnabled="true">
 91         <SSLHostConfig>
 92             <Certificate certificateKeystoreFile="conf/localhost-rsa.jks"
 93                          type="RSA" />
 94         </SSLHostConfig>
 95     </Connector>
 96     -->
 97     <!-- Define an SSL/TLS HTTP/1.1 Connector on port 8443 with HTTP/2
 98          This connector uses the APR/native implementation which always uses
 99          OpenSSL for TLS.
100          Either JSSE or OpenSSL style configuration may be used. OpenSSL style
101          configuration is used below.
102     -->
103     <!--
104     <Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"
105                maxThreads="150" SSLEnabled="true" >
106         <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
107         <SSLHostConfig>
108             <Certificate certificateKeyFile="conf/localhost-rsa-key.pem"
109                          certificateFile="conf/localhost-rsa-cert.pem"
110                          certificateChainFile="conf/localhost-rsa-chain.pem"
111                          type="RSA" />
112         </SSLHostConfig>
113     </Connector>
114     -->
115 
116     <!-- Define an AJP 1.3 Connector on port 8009 -->
117     <!--
118     <Connector protocol="AJP/1.3"
119                address="::1"
120                port="8009"
121                redirectPort="8443" />
122     -->
123 
124     <!-- An Engine represents the entry point (within Catalina) that processes
125          every request.  The Engine implementation for Tomcat stand alone
126          analyzes the HTTP headers included with the request, and passes them
127          on to the appropriate Host (virtual host).
128          Documentation at /docs/config/engine.html -->
129 
130     <!-- You should set jvmRoute to support load-balancing via AJP ie :
131     <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
132     -->
133     <Engine name="Catalina" defaultHost="localhost">
134 
135       <!--For clustering, please take a look at documentation at:
136           /docs/cluster-howto.html  (simple how to)
137           /docs/config/cluster.html (reference documentation) -->
138       <!--
139       <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
140       -->
141 
142       <!-- Use the LockOutRealm to prevent attempts to guess user passwords
143            via a brute-force attack -->
144       <Realm className="org.apache.catalina.realm.LockOutRealm">
145         <!-- This Realm uses the UserDatabase configured in the global JNDI
146              resources under the key "UserDatabase".  Any edits
147              that are performed against this UserDatabase are immediately
148              available for use by the Realm.  -->
149         <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
150                resourceName="UserDatabase"/>
151       </Realm>
152 
153       <Host name="localhost"  appBase="webapps"
154             unpackWARs="true" autoDeploy="true">
155 
156         <!-- SingleSignOn valve, share authentication between web applications
157              Documentation at: /docs/config/valve.html -->
158         <!--
159         <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
160         -->
161 
162         <!-- Access log processes all example.
163              Documentation at: /docs/config/valve.html
164              Note: The pattern used is equivalent to using pattern="common" -->
165         <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
166                prefix="localhost_access_log" suffix=".txt"
167                pattern="%h %l %u %t "%r" %s %b" />
168 
169       </Host>
170     </Engine>
171   </Service>
172 </Server>
173 [root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# cat /yinzhengjie/softwares/apache-tomcat-8.5.55/conf/server.xml              #查看tomcat的默認配置文件,注意觀察第165行。
  1 [root@es102.yinzhengjie.com ~]# vim /yinzhengjie/softwares/apache-tomcat-8.5.55/conf/server.xml              #將tomcat日志轉換為JSON格式
  2 [root@es102.yinzhengjie.com ~]# 
  3 [root@es102.yinzhengjie.com ~]# cat /yinzhengjie/softwares/apache-tomcat-8.5.55/conf/server.xml 
  4 <?xml version="1.0" encoding="UTF-8"?>
  5 <!--
  6   Licensed to the Apache Software Foundation (ASF) under one or more
  7   contributor license agreements.  See the NOTICE file distributed with
  8   this work for additional information regarding copyright ownership.
  9   The ASF licenses this file to You under the Apache License, Version 2.0
 10   (the "License"); you may not use this file except in compliance with
 11   the License.  You may obtain a copy of the License at
 12 
 13       http://www.apache.org/licenses/LICENSE-2.0
 14 
 15   Unless required by applicable law or agreed to in writing, software
 16   distributed under the License is distributed on an "AS IS" BASIS,
 17   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 18   See the License for the specific language governing permissions and
 19   limitations under the License.
 20 -->
 21 <!-- Note:  A "Server" is not itself a "Container", so you may not
 22      define subcomponents such as "Valves" at this level.
 23      Documentation at /docs/config/server.html
 24  -->
 25 <Server port="8005" shutdown="SHUTDOWN">
 26   <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
 27   <!-- Security listener. Documentation at /docs/config/listeners.html
 28   <Listener className="org.apache.catalina.security.SecurityListener" />
 29   -->
 30   <!--APR library loader. Documentation at /docs/apr.html -->
 31   <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
 32   <!-- Prevent memory leaks due to use of particular java/javax APIs-->
 33   <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
 34   <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
 35   <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
 36 
 37   <!-- Global JNDI resources
 38        Documentation at /docs/jndi-resources-howto.html
 39   -->
 40   <GlobalNamingResources>
 41     <!-- Editable user database that can also be used by
 42          UserDatabaseRealm to authenticate users
 43     -->
 44     <Resource name="UserDatabase" auth="Container"
 45               type="org.apache.catalina.UserDatabase"
 46               description="User database that can be updated and saved"
 47               factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
 48               pathname="conf/tomcat-users.xml" />
 49   </GlobalNamingResources>
 50 
 51   <!-- A "Service" is a collection of one or more "Connectors" that share
 52        a single "Container" Note:  A "Service" is not itself a "Container",
 53        so you may not define subcomponents such as "Valves" at this level.
 54        Documentation at /docs/config/service.html
 55    -->
 56   <Service name="Catalina">
 57 
 58     <!--The connectors can use a shared executor, you can define one or more named thread pools-->
 59     <!--
 60     <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
 61         maxThreads="150" minSpareThreads="4"/>
 62     -->
 63 
 64 
 65     <!-- A "Connector" represents an endpoint by which requests are received
 66          and responses are returned. Documentation at :
 67          Java HTTP Connector: /docs/config/http.html
 68          Java AJP  Connector: /docs/config/ajp.html
 69          APR (HTTP/AJP) Connector: /docs/apr.html
 70          Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
 71     -->
 72     <Connector port="8080" protocol="HTTP/1.1"
 73                connectionTimeout="20000"
 74                redirectPort="8443" />
 75     <!-- A "Connector" using the shared thread pool-->
 76     <!--
 77     <Connector executor="tomcatThreadPool"
 78                port="8080" protocol="HTTP/1.1"
 79                connectionTimeout="20000"
 80                redirectPort="8443" />
 81     -->
 82     <!-- Define an SSL/TLS HTTP/1.1 Connector on port 8443
 83          This connector uses the NIO implementation. The default
 84          SSLImplementation will depend on the presence of the APR/native
 85          library and the useOpenSSL attribute of the
 86          AprLifecycleListener.
 87          Either JSSE or OpenSSL style configuration may be used regardless of
 88          the SSLImplementation selected. JSSE style configuration is used below.
 89     -->
 90     <!--
 91     <Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
 92                maxThreads="150" SSLEnabled="true">
 93         <SSLHostConfig>
 94             <Certificate certificateKeystoreFile="conf/localhost-rsa.jks"
 95                          type="RSA" />
 96         </SSLHostConfig>
 97     </Connector>
 98     -->
 99     <!-- Define an SSL/TLS HTTP/1.1 Connector on port 8443 with HTTP/2
100          This connector uses the APR/native implementation which always uses
101          OpenSSL for TLS.
102          Either JSSE or OpenSSL style configuration may be used. OpenSSL style
103          configuration is used below.
104     -->
105     <!--
106     <Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"
107                maxThreads="150" SSLEnabled="true" >
108         <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
109         <SSLHostConfig>
110             <Certificate certificateKeyFile="conf/localhost-rsa-key.pem"
111                          certificateFile="conf/localhost-rsa-cert.pem"
112                          certificateChainFile="conf/localhost-rsa-chain.pem"
113                          type="RSA" />
114         </SSLHostConfig>
115     </Connector>
116     -->
117 
118     <!-- Define an AJP 1.3 Connector on port 8009 -->
119     <!--
120     <Connector protocol="AJP/1.3"
121                address="::1"
122                port="8009"
123                redirectPort="8443" />
124     -->
125 
126     <!-- An Engine represents the entry point (within Catalina) that processes
127          every request.  The Engine implementation for Tomcat stand alone
128          analyzes the HTTP headers included with the request, and passes them
129          on to the appropriate Host (virtual host).
130          Documentation at /docs/config/engine.html -->
131 
132     <!-- You should set jvmRoute to support load-balancing via AJP ie :
133     <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
134     -->
135     <Engine name="Catalina" defaultHost="localhost">
136 
137       <!--For clustering, please take a look at documentation at:
138           /docs/cluster-howto.html  (simple how to)
139           /docs/config/cluster.html (reference documentation) -->
140       <!--
141       <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
142       -->
143 
144       <!-- Use the LockOutRealm to prevent attempts to guess user passwords
145            via a brute-force attack -->
146       <Realm className="org.apache.catalina.realm.LockOutRealm">
147         <!-- This Realm uses the UserDatabase configured in the global JNDI
148              resources under the key "UserDatabase".  Any edits
149              that are performed against this UserDatabase are immediately
150              available for use by the Realm.  -->
151         <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
152                resourceName="UserDatabase"/>
153       </Realm>
154 
155       <Host name="localhost"  appBase="webapps"
156             unpackWARs="true" autoDeploy="true">
157 
158         <!-- SingleSignOn valve, share authentication between web applications
159              Documentation at: /docs/config/valve.html -->
160         <!--
161         <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
162         -->
163 
164         <!-- Access log processes all example.
165              Documentation at: /docs/config/valve.html
166              Note: The pattern used is equivalent to using pattern="common" -->
167         <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
168                prefix="es102.yinzhengjie.com_access_log" suffix=".log"
169                pattern="{&quot;client&quot;:&quot;%h&quot;,  &quot;client user&quot;:&quot;%l&quot;,   &quot;authenticated&quot;:&quot;%u&quot;,   &quot;access time&quot;:&quot;%t&quot;,   &quot;method&quot;:&quot;%r&quot;,   &quot;status&quot;:&quot;%s&quot;,  &quot;send bytes&quot;:&quot;%b&quot;,  &quot;Query?string&quot;:&quot;%q&quot;,  &quot;partner&quot;:&quot;%{Referer}i&quot;,  &quot;Agent version&quot;:&quot;%{User-Agent}i&quot;}" />
170       </Host>
171     </Engine>
172   </Service>
173 </Server>
174 [root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# vim /yinzhengjie/softwares/apache-tomcat-8.5.55/conf/server.xml              #將tomcat日志轉換為JSON格式,注意和默認配置文件進行對比
[root@es102.yinzhengjie.com ~]# rm -f /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/*            #重啟tomcat服務之前記得刪除之前的訪問日志喲~(重啟會自動生成新的文件)
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# ll /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/
total 0
drwxr-x--- 2 root root   6 Jun  6 03:13 ./
drwxr-xr-x 9 root root 220 Jun  5 04:34 ../
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# 
 
[root@es102.yinzhengjie.com ~]# rm -f /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/*                  #重啟tomcat服務之前記得刪除之前的訪問日志喲~(重啟會自動生成新的文件)
[root@es102.yinzhengjie.com ~]# ss -ntl
State                  Recv-Q                  Send-Q                                              Local Address:Port                                     Peer Address:Port                  
LISTEN                 0                       128                                                 127.0.0.53%lo:53                                            0.0.0.0:*                     
LISTEN                 0                       128                                                       0.0.0.0:22                                            0.0.0.0:*                     
LISTEN                 0                       100                                                             *:8080                                                *:*                     
LISTEN                 0                       128                                        [::ffff:172.200.5.102]:9200                                                *:*                     
LISTEN                 0                       128                                        [::ffff:172.200.5.102]:9300                                                *:*                     
LISTEN                 0                       128                                                          [::]:22                                               [::]:*                     
LISTEN                 0                       1                                              [::ffff:127.0.0.1]:8005                                                *:*                     
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/catalina.sh stop
Using CATALINA_BASE:   /yinzhengjie/softwares/apache-tomcat-8.5.55
Using CATALINA_HOME:   /yinzhengjie/softwares/apache-tomcat-8.5.55
Using CATALINA_TMPDIR: /yinzhengjie/softwares/apache-tomcat-8.5.55/temp
Using JRE_HOME:        /yinzhengjie/softwares/jdk1.8.0_201/jre
Using CLASSPATH:       /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/bootstrap.jar:/yinzhengjie/softwares/apache-tomcat-8.5.55/bin/tomcat-juli.jar
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# ss -ntl
State                  Recv-Q                  Send-Q                                              Local Address:Port                                     Peer Address:Port                  
LISTEN                 0                       128                                                 127.0.0.53%lo:53                                            0.0.0.0:*                     
LISTEN                 0                       128                                                       0.0.0.0:22                                            0.0.0.0:*                     
LISTEN                 0                       128                                        [::ffff:172.200.5.102]:9200                                                *:*                     
LISTEN                 0                       128                                        [::ffff:172.200.5.102]:9300                                                *:*                     
LISTEN                 0                       128                                                          [::]:22                                               [::]:*                     
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/catalina.sh stop
[root@es102.yinzhengjie.com ~]# ss -ntl
State                  Recv-Q                  Send-Q                                              Local Address:Port                                     Peer Address:Port                  
LISTEN                 0                       128                                                 127.0.0.53%lo:53                                            0.0.0.0:*                     
LISTEN                 0                       128                                                       0.0.0.0:22                                            0.0.0.0:*                     
LISTEN                 0                       128                                        [::ffff:172.200.5.102]:9200                                                *:*                     
LISTEN                 0                       128                                        [::ffff:172.200.5.102]:9300                                                *:*                     
LISTEN                 0                       128                                                          [::]:22                                               [::]:*                     
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/catalina.sh start
Using CATALINA_BASE:   /yinzhengjie/softwares/apache-tomcat-8.5.55
Using CATALINA_HOME:   /yinzhengjie/softwares/apache-tomcat-8.5.55
Using CATALINA_TMPDIR: /yinzhengjie/softwares/apache-tomcat-8.5.55/temp
Using JRE_HOME:        /yinzhengjie/softwares/jdk1.8.0_201/jre
Using CLASSPATH:       /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/bootstrap.jar:/yinzhengjie/softwares/apache-tomcat-8.5.55/bin/tomcat-juli.jar
Tomcat started.
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# ss -ntl
State                  Recv-Q                  Send-Q                                              Local Address:Port                                     Peer Address:Port                  
LISTEN                 0                       128                                                 127.0.0.53%lo:53                                            0.0.0.0:*                     
LISTEN                 0                       128                                                       0.0.0.0:22                                            0.0.0.0:*                     
LISTEN                 0                       100                                                             *:8080                                                *:*                     
LISTEN                 0                       128                                        [::ffff:172.200.5.102]:9200                                                *:*                     
LISTEN                 0                       128                                        [::ffff:172.200.5.102]:9300                                                *:*                     
LISTEN                 0                       128                                                          [::]:22                                               [::]:*                     
LISTEN                 0                       1                                              [::ffff:127.0.0.1]:8005                                                *:*                     
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/catalina.sh start
[root@es102.yinzhengjie.com ~]# ll /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/
total 24
drwxr-x--- 2 root root  209 Jun  6 03:17 ./
drwxr-xr-x 9 root root  220 Jun  5 04:34 ../
-rw-r----- 1 root root 6395 Jun  6 03:17 catalina.2020-06-06.log
-rw-r----- 1 root root 6395 Jun  6 03:17 catalina.out
-rw-r----- 1 root root  762 Jun  6 03:18 es102.yinzhengjie.com_access_log.2020-06-06.log
-rw-r----- 1 root root    0 Jun  6 03:16 host-manager.2020-06-06.log
-rw-r----- 1 root root  459 Jun  6 03:17 localhost.2020-06-06.log
-rw-r----- 1 root root    0 Jun  6 03:16 manager.2020-06-06.log
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# tail -10f /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log 
{"client":"172.200.0.1",  "client user":"-",   "authenticated":"-",   "access time":"[06/Jun/2020:03:18:19 +0000]",     "method":"GET / HTTP/1.1",   "status":"200",  "send bytes":"11215",  
"Query?string":"",  "partner":"-",  "Agent version":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"}{"client":"172.200.0.1",  "client user":"-",   "authenticated":"-",   "access time":"[06/Jun/2020:03:18:19 +0000]",     "method":"GET /favicon.ico HTTP/1.1",   "status":"200",  "send bytes"
:"21630",  "Query?string":"",  "partner":"http://es102.yinzhengjie.com:8080/",  "Agent version":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"}
[root@es102.yinzhengjie.com ~]# tail -10f /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log
[root@es102.yinzhengjie.com ~]# ll /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log 
-rw-r----- 1 root root 762 Jun  6 03:18 /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# chmod 644 /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log 
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# ll /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log 
-rw-r--r-- 1 root root 762 Jun  6 03:18 /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# chmod 644 /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log      #記得授權哈~不然啟動時logstash沒有權限訪問
[root@es102.yinzhengjie.com ~]# ll /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/ -d
drwxr-x--- 2 root root 209 Jun  6 03:17 /yinzhengjie/softwares/apache-tomcat-8.5.55/logs//
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# chmod o+x /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/ 
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# ll /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/ -d
drwxr-x--x 2 root root 209 Jun  6 03:17 /yinzhengjie/softwares/apache-tomcat-8.5.55/logs//
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# chmod o+x /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/                  #千萬別忘記給目錄添加可訪問權限,否則logstash無法進入到logs目錄喲~

2>.編寫配置文件並檢查語法是否錯誤

[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/java-elasticsearch.conf 
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# cat /etc/logstash/conf.d/java-elasticsearch.conf 
input {
    file {
        type => "java-log"
        path => "/var/log/logstash/logstash-plain.log"
        start_position => "beginning"
        stat_interval => 3
    }

    
    file {
        type => "tomcat-access-log"
        path => "/yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.*.log"
        start_position => "beginning"
        stat_interval => 3
        codec => "json"
    }
}

output {
    if [type] == "java-log" {
        elasticsearch {
            hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"]
            index => "java-log-172.200.5.102-%{+YYYY.MM.dd}"
        }
       
    }

    if [type] == "tomcat-access-log" {
        elasticsearch {
            hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"]
            index => "tomcat-access-log-172.200.5.102-%{+YYYY.MM.dd}"
        }

        file {
            path => "/tmp/tomcat-access-log"
        }
    }
}
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/java-elasticsearch.conf
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/java-elasticsearch.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-06 04:50:06.980 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[INFO ] 2020-06-06 04:50:11.656 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/java-elasticsearch.conf -t

3>.啟動logstash服務

[root@es102.yinzhengjie.com ~]# vim /etc/systemd/system/logstash.service
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# cat /etc/systemd/system/logstash.service
[Unit]
Description=logstash

[Service]
Type=simple
User=root
Group=root
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# vim /etc/systemd/system/logstash.service          #將logstash服務以root身份啟動(這樣避免logstash某些日志文件無權限訪問)
[root@es102.yinzhengjie.com ~]# systemctl daemon-reload                    #使得上述配置生效
[root@es102.yinzhengjie.com ~]#   
[root@es102.yinzhengjie.com ~]# systemctl restart logstash.service              #重啟logstash服務 
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# ll /tmp/tomcat-access-log 
-rw-r--r-- 1 root root 1152 Jun  6 05:42 /tmp/tomcat-access-log
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# cat /tmp/tomcat-access-log 
{"access time":"[06/Jun/2020:03:18:19 +0000]","@timestamp":"2020-06-06T05:42:17.263Z","method":"GET / HTTP/1.1","type":"tomcat-access-log","send bytes":"11215","Agent version":"Mozilla/5.0 
(Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","@version":"1","client":"172.200.0.1","client user":"-","partner":"-","path":"/yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log","Query?string":"","host":"es102.yinzhengjie.com","status":"200","authenticated":"-"}{"access time":"[06/Jun/2020:03:18:19 +0000]","@timestamp":"2020-06-06T05:42:19.224Z","method":"GET /favicon.ico HTTP/1.1","type":"tomcat-access-log","send bytes":"21630","Agent version":"M
ozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","@version":"1","client":"172.200.0.1","client user":"-","partner":"http://es102.yinzhengjie.com:8080/","path":"/yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log","Query?string":"","host":"es102.yinzhengjie.com","status":"200","authenticated":"-"}[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# cat /tmp/tomcat-access-log                #數據已經成功寫入到本地文件啦,同時也寫入到Elastticsearch集群啦~如下圖所示,可以通過kibana創建索引。

4>.在kibana查看寫入Elasticsearch集群的JSON日志

5>.統計日志的IP地址個數

[root@es102.yinzhengjie.com ~]# ll /tmp/tomcat-access-log 
-rw-r--r-- 1 root root 1152 Jun  6 05:42 /tmp/tomcat-access-log
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# cat /tmp/tomcat-access-log 
{"access time":"[06/Jun/2020:03:18:19 +0000]","@timestamp":"2020-06-06T05:42:17.263Z","method":"GET / HTTP/1.1","type":"tomcat-access-log","send bytes":"11215","Agent version":"Mozilla/5.0 
(Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","@version":"1","client":"172.200.0.1","client user":"-","partner":"-","path":"/yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log","Query?string":"","host":"es102.yinzhengjie.com","status":"200","authenticated":"-"}{"access time":"[06/Jun/2020:03:18:19 +0000]","@timestamp":"2020-06-06T05:42:19.224Z","method":"GET /favicon.ico HTTP/1.1","type":"tomcat-access-log","send bytes":"21630","Agent version":"M
ozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","@version":"1","client":"172.200.0.1","client user":"-","partner":"http://es102.yinzhengjie.com:8080/","path":"/yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log","Query?string":"","host":"es102.yinzhengjie.com","status":"200","authenticated":"-"}[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# cat /tmp/tomcat-access-log 
[root@es102.yinzhengjie.com ~]# cat log.py 
#!/usr/bin/python3
#__conding__:uft-8

status_200=[]
status_404=[]

with open("/tmp/tomcat-access-log") as f:
    for line in f.readlines():
        line = eval(line)
        print(line.get("client"))
        if line.get("status") == "200":
            status_200.append(line.get)
        elif line.get(status) == "404":
            status_404.append(line.get)
        else:
            print("狀態碼錯誤")

print("狀態碼200的有: ",len(status_200))
print("狀態碼404的有: ",len(status_404))
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# python3 log.py
172.200.0.1
172.200.0.1
狀態碼200的有:  2
狀態碼404的有:  0
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# python3 log.py                        #統計日志中的IP地址個數

 

六.日志的多行合並案例

  關於多行合並官方也有相應的案例說明,我這里就不再贅述了,感興趣的小伙伴可自行查看官網。

  博主推薦閱讀:
    https://www.elastic.co/guide/en/logstash/6.8/plugins-codecs-multiline.html

1>.編寫配置文件

[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/stdin-stdout.conf 
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# cat /etc/logstash/conf.d/stdin-stdout.conf 
input {
    stdin {
        codec => multiline {
            pattern => "^\["
            negate => "true"
            what => "previous"
        }
    }
}

output {
    stdout {
        codec => "rubydebug"
    }
}
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/stdin-stdout.conf

2>.檢查配置語法是否錯誤

[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f  /etc/logstash/conf.d/stdin-stdout.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-06 04:29:38.082 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[INFO ] 2020-06-06 04:29:42.272 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/stdin-stdout.conf -t

3>.測試多行合並

[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f  /etc/logstash/conf.d/stdin-stdout.conf 
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-06 04:19:29.466 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-06-06 04:19:29.479 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"}
[INFO ] 2020-06-06 04:19:29.505 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"79da8bcd-0a33-4dab-a25a-df89c5387e12", :path=>"/usr/share/logstash/dat
a/uuid"}[INFO ] 2020-06-06 04:19:34.517 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc
h.delay"=>50}[INFO ] 2020-06-06 04:19:34.635 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7c854c5f run>"}
The stdin plugin is now waiting for input:
[INFO ] 2020-06-06 04:19:34.718 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin
es=>[]}[INFO ] 2020-06-06 04:19:35.779 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601}
yinzhengjie
2020
blog[https://www.cnblogs.com/yinzhengjie/]
bigdata
[2020520
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
      "@version" => "1",
    "@timestamp" => 2020-06-06T04:20:41.993Z,
       "message" => "yinzhengjie\n2020\nblog[https://www.cnblogs.com/yinzhengjie/]\nbigdata",
          "tags" => [
        [0] "multiline"
    ],
          "host" => "es102.yinzhengjie.com"
}
yinzhengjie&jasonYin[op]
LOL  
Python
Golang
[6666
{
      "@version" => "1",
    "@timestamp" => 2020-06-06T04:24:05.829Z,
       "message" => "[2020520\nyinzhengjie&jasonYin[op]\nLOL\nPython\nGolang",
          "tags" => [
        [0] "multiline"
    ],
          "host" => "es102.yinzhengjie.com"
}
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/stdin-stdout.conf

 

七.收集nginx日志

1>.安裝nginx服務

[root@es102.yinzhengjie.com ~]# apt-get -y install nginx
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0 libjpeg-turbo8 libjpeg8 libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter
  libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6 libxpm4 nginx-common nginx-core
Suggested packages:
  libgd-tools fcgiwrap nginx-doc ssl-cert
The following NEW packages will be installed:
  fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0 libjpeg-turbo8 libjpeg8 libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter
  libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6 libxpm4 nginx nginx-common nginx-core
0 upgraded, 18 newly installed, 0 to remove and 79 not upgraded.
Need to get 2,462 kB of archives.
After this operation, 8,210 kB of additional disk space will be used.
Get:1 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libjpeg-turbo8 amd64 1.5.2-0ubuntu5.18.04.3 [110 kB]
Get:2 http://mirrors.aliyun.com/ubuntu bionic/main amd64 fonts-dejavu-core all 2.37-1 [1,041 kB]
Get:3 http://mirrors.aliyun.com/ubuntu bionic/main amd64 fontconfig-config all 2.12.6-0ubuntu2 [55.8 kB]
Get:4 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libfontconfig1 amd64 2.12.6-0ubuntu2 [137 kB]
Get:5 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libjpeg8 amd64 8c-2ubuntu8 [2,194 B]
Get:6 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libjbig0 amd64 2.1-3.1build1 [26.7 kB]
Get:7 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libtiff5 amd64 4.0.9-5ubuntu0.3 [153 kB]
Get:8 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libwebp6 amd64 0.6.1-2 [185 kB]
Get:9 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libxpm4 amd64 1:3.5.12-1 [34.0 kB]
Get:10 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libgd3 amd64 2.2.5-4ubuntu0.4 [119 kB]
Get:11 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 nginx-common all 1.14.0-0ubuntu1.7 [37.4 kB]
Get:12 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-http-geoip amd64 1.14.0-0ubuntu1.7 [11.2 kB]
Get:13 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-http-image-filter amd64 1.14.0-0ubuntu1.7 [14.6 kB]
Get:14 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-http-xslt-filter amd64 1.14.0-0ubuntu1.7 [13.0 kB]
Get:15 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-mail amd64 1.14.0-0ubuntu1.7 [41.8 kB]
Get:16 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-stream amd64 1.14.0-0ubuntu1.7 [63.7 kB]
Get:17 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 nginx-core amd64 1.14.0-0ubuntu1.7 [413 kB]
Get:18 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 nginx all 1.14.0-0ubuntu1.7 [3,596 B]
Fetched 2,462 kB in 2s (1,168 kB/s)
Preconfiguring packages ...
Selecting previously unselected package libjpeg-turbo8:amd64.
(Reading database ... 119418 files and directories currently installed.)
Preparing to unpack .../00-libjpeg-turbo8_1.5.2-0ubuntu5.18.04.3_amd64.deb ...
Unpacking libjpeg-turbo8:amd64 (1.5.2-0ubuntu5.18.04.3) ...
Selecting previously unselected package fonts-dejavu-core.
Preparing to unpack .../01-fonts-dejavu-core_2.37-1_all.deb ...
Unpacking fonts-dejavu-core (2.37-1) ...
Selecting previously unselected package fontconfig-config.
Preparing to unpack .../02-fontconfig-config_2.12.6-0ubuntu2_all.deb ...
Unpacking fontconfig-config (2.12.6-0ubuntu2) ...
Selecting previously unselected package libfontconfig1:amd64.
Preparing to unpack .../03-libfontconfig1_2.12.6-0ubuntu2_amd64.deb ...
Unpacking libfontconfig1:amd64 (2.12.6-0ubuntu2) ...
Selecting previously unselected package libjpeg8:amd64.
Preparing to unpack .../04-libjpeg8_8c-2ubuntu8_amd64.deb ...
Unpacking libjpeg8:amd64 (8c-2ubuntu8) ...
Selecting previously unselected package libjbig0:amd64.
Preparing to unpack .../05-libjbig0_2.1-3.1build1_amd64.deb ...
Unpacking libjbig0:amd64 (2.1-3.1build1) ...
Selecting previously unselected package libtiff5:amd64.
Preparing to unpack .../06-libtiff5_4.0.9-5ubuntu0.3_amd64.deb ...
Unpacking libtiff5:amd64 (4.0.9-5ubuntu0.3) ...
Selecting previously unselected package libwebp6:amd64.
Preparing to unpack .../07-libwebp6_0.6.1-2_amd64.deb ...
Unpacking libwebp6:amd64 (0.6.1-2) ...
Selecting previously unselected package libxpm4:amd64.
Preparing to unpack .../08-libxpm4_1%3a3.5.12-1_amd64.deb ...
Unpacking libxpm4:amd64 (1:3.5.12-1) ...
Selecting previously unselected package libgd3:amd64.
Preparing to unpack .../09-libgd3_2.2.5-4ubuntu0.4_amd64.deb ...
Unpacking libgd3:amd64 (2.2.5-4ubuntu0.4) ...
Selecting previously unselected package nginx-common.
Preparing to unpack .../10-nginx-common_1.14.0-0ubuntu1.7_all.deb ...
Unpacking nginx-common (1.14.0-0ubuntu1.7) ...
Selecting previously unselected package libnginx-mod-http-geoip.
Preparing to unpack .../11-libnginx-mod-http-geoip_1.14.0-0ubuntu1.7_amd64.deb ...
Unpacking libnginx-mod-http-geoip (1.14.0-0ubuntu1.7) ...
Selecting previously unselected package libnginx-mod-http-image-filter.
Preparing to unpack .../12-libnginx-mod-http-image-filter_1.14.0-0ubuntu1.7_amd64.deb ...
Unpacking libnginx-mod-http-image-filter (1.14.0-0ubuntu1.7) ...
Selecting previously unselected package libnginx-mod-http-xslt-filter.
Preparing to unpack .../13-libnginx-mod-http-xslt-filter_1.14.0-0ubuntu1.7_amd64.deb ...
Unpacking libnginx-mod-http-xslt-filter (1.14.0-0ubuntu1.7) ...
Selecting previously unselected package libnginx-mod-mail.
Preparing to unpack .../14-libnginx-mod-mail_1.14.0-0ubuntu1.7_amd64.deb ...
Unpacking libnginx-mod-mail (1.14.0-0ubuntu1.7) ...
Selecting previously unselected package libnginx-mod-stream.
Preparing to unpack .../15-libnginx-mod-stream_1.14.0-0ubuntu1.7_amd64.deb ...
Unpacking libnginx-mod-stream (1.14.0-0ubuntu1.7) ...
Selecting previously unselected package nginx-core.
Preparing to unpack .../16-nginx-core_1.14.0-0ubuntu1.7_amd64.deb ...
Unpacking nginx-core (1.14.0-0ubuntu1.7) ...
Selecting previously unselected package nginx.
Preparing to unpack .../17-nginx_1.14.0-0ubuntu1.7_all.deb ...
Unpacking nginx (1.14.0-0ubuntu1.7) ...
Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Setting up libjbig0:amd64 (2.1-3.1build1) ...
Setting up fonts-dejavu-core (2.37-1) ...
Setting up nginx-common (1.14.0-0ubuntu1.7) ...
Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /lib/systemd/system/nginx.service.
Setting up libjpeg-turbo8:amd64 (1.5.2-0ubuntu5.18.04.3) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for systemd (237-3ubuntu10.38) ...
Setting up libnginx-mod-mail (1.14.0-0ubuntu1.7) ...
Setting up libxpm4:amd64 (1:3.5.12-1) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Setting up libnginx-mod-http-xslt-filter (1.14.0-0ubuntu1.7) ...
Setting up libnginx-mod-http-geoip (1.14.0-0ubuntu1.7) ...
Setting up libwebp6:amd64 (0.6.1-2) ...
Setting up libjpeg8:amd64 (8c-2ubuntu8) ...
Setting up fontconfig-config (2.12.6-0ubuntu2) ...
Setting up libnginx-mod-stream (1.14.0-0ubuntu1.7) ...
Setting up libtiff5:amd64 (4.0.9-5ubuntu0.3) ...
Setting up libfontconfig1:amd64 (2.12.6-0ubuntu2) ...
Setting up libgd3:amd64 (2.2.5-4ubuntu0.4) ...
Setting up libnginx-mod-http-image-filter (1.14.0-0ubuntu1.7) ...
Setting up nginx-core (1.14.0-0ubuntu1.7) ...
Setting up nginx (1.14.0-0ubuntu1.7) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# apt-get -y install nginx
[root@es102.yinzhengjie.com ~]# vim /etc/nginx/nginx.conf 
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# cat /etc/nginx/nginx.conf 
worker_processes  4;
worker_cpu_affinity 00000001 00000010 00000100 00001000; 

events {
    worker_connections  100000;
    use epoll;
    accept_mutex on;
    multi_accept on; 
}

http {
    include       mime.types;
    
    default_type  text/html;
    
    charset utf-8;

    log_format my_access_json '{"@timestamp":"$time_iso8601",' 
        '"host":"$server_addr",' 
        '"clientip":"$remote_addr",' 
        '"size":$body_bytes_sent,' 
        '"responsetime":$request_time,' 
        '"upstreamtime":"$upstream_response_time",' 
        '"upstreamhost":"$upstream_addr",' 
        '"http_host":"$host",' 
        '"uri":"$uri",' 
        '"domain":"$host",' 
        '"xff":"$http_x_forwarded_for",' 
        '"referer":"$http_referer",' 
        '"tcp_xff":"$proxy_protocol_addr",' 
        '"http_user_agent":"$http_user_agent",' 
        '"status":"$status"}';

    access_log /var/log/nginx/access.log my_access_json;
    
    error_log /var/log/nginx/error.log;
    
    gzip on;
    
    include /etc/nginx/conf.d/*.conf;
    
    include /etc/nginx/sites-enabled/*;

}
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# vim /etc/nginx/nginx.conf            #修改Nginx默認的配置文件,使得其日志存儲為JSON格式。
[root@es102.yinzhengjie.com ~]# systemctl start nginx                #啟動Nginx服務
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# systemctl enable nginx                #將nginx設置為開機自啟動
Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable nginx
[root@es102.yinzhengjie.com ~]# 

2>.編寫配置文件並檢查語法是否錯誤

[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/nginx-elasticsearch.conf
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# cat /etc/logstash/conf.d/nginx-elasticsearch.conf
input {
    file {
        type => "nginx-access-log"
        path => "/var/log/nginx/access.log"
        start_position => "beginning"
        stat_interval => 3
        codec => "json"
    }
}

output {
    if [type] == "nginx-access-log" {
        elasticsearch {
            hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"]
            index => "nginx-access-log-172.200.5.102-%{+YYYY.MM.dd}"
        }
    }
}
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/nginx-elasticsearch.conf
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx-elasticsearch.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-06 07:57:30.291 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[INFO ] 2020-06-06 07:57:35.681 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx-elasticsearch.conf -t

3>.啟動logstash服務

[root@es102.yinzhengjie.com ~]# systemctl start logstash

4>.在Kibana頁面中添加索引

5>.查看nginx的日志

 

八.收集TCP日志

1>.編寫配置文件並檢查語法是否錯誤

[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/tcp-elasticsearch.conf 
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# cat /etc/logstash/conf.d/tcp-elasticsearch.conf 
input {
    tcp {
        port => "8888"
        codec => "json"
    }
}

output {
    stdout {
        codec => "rubydebug"
    }
}
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/tcp-elasticsearch.conf      #測試文件,輸出到標准輸出(當前終端),下面的測試均采用該配置文件試驗的
[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/tcp-elasticsearch.conf
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# cat /etc/logstash/conf.d/tcp-elasticsearch.conf
input {
    tcp {
        port => "8888"
        codec => "json"
    }
}

output {
    elasticsearch {
        hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"]
        index => "tcp-log-172.200.5.102-%{+YYYY.MM.dd}"
    }
}
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/tcp-elasticsearch.conf      #將數據寫入到Elasticsearch集群(上面測試成功后,可以使用這種方案)
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcp-elasticsearch.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-06 08:25:30.447 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[INFO ] 2020-06-06 08:25:34.496 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcp-elasticsearch.conf -t

2>.傳數測試數據

[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcp-elasticsearch.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-06 08:26:02.135 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-06-06 08:26:02.150 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"}
[INFO ] 2020-06-06 08:26:07.011 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc
h.delay"=>50}[INFO ] 2020-06-06 08:26:07.155 [[main]-pipeline-manager] tcp - Automatically switching from json to json_lines codec {:plugin=>"tcp"}
[INFO ] 2020-06-06 08:26:07.498 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xfcc6877 run>"}
[INFO ] 2020-06-06 08:26:07.564 [[main]<tcp] tcp - Starting tcp input listener {:address=>"0.0.0.0:8888", :ssl_enable=>"false"}
[INFO ] 2020-06-06 08:26:07.612 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin
es=>[]}[INFO ] 2020-06-06 08:26:08.000 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
    "@timestamp" => 2020-06-06T08:34:26.207Z,
          "port" => 57610,
      "password" => "123",
      "@version" => "1",
          "host" => "es103.yinzhengjie.com",
      "username" => "yinzhengjie"
}
{
    "@timestamp" => 2020-06-06T08:36:02.000Z,
          "port" => 57614,
      "password" => "666",
      "@version" => "1",
          "host" => "es103.yinzhengjie.com",
      "username" => "yinzhengjie"
}
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcp-elasticsearch.conf        #啟動logstash單個任務,會自動監聽本機的8888端口用於接收其它主機發送來的數據~
[root@es103.yinzhengjie.com ~]# apt-get -y install nmap
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libblas3 liblinear3 liblua5.3-0
Suggested packages:
  liblinear-tools liblinear-dev ndiff
The following NEW packages will be installed:
  libblas3 liblinear3 liblua5.3-0 nmap
0 upgraded, 4 newly installed, 0 to remove and 79 not upgraded.
Need to get 5,467 kB of archives.
After this operation, 25.0 MB of additional disk space will be used.
Get:1 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libblas3 amd64 3.7.1-4ubuntu1 [140 kB]
Get:2 http://mirrors.aliyun.com/ubuntu bionic/main amd64 liblinear3 amd64 2.1.0+dfsg-2 [39.3 kB]
Get:3 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 liblua5.3-0 amd64 5.3.3-1ubuntu0.18.04.1 [115 kB]
Get:4 http://mirrors.aliyun.com/ubuntu bionic/main amd64 nmap amd64 7.60-1ubuntu5 [5,174 kB]
Fetched 5,467 kB in 1s (4,379 kB/s)
Selecting previously unselected package libblas3:amd64.
(Reading database ... 119654 files and directories currently installed.)
Preparing to unpack .../libblas3_3.7.1-4ubuntu1_amd64.deb ...
Unpacking libblas3:amd64 (3.7.1-4ubuntu1) ...
Selecting previously unselected package liblinear3:amd64.
Preparing to unpack .../liblinear3_2.1.0+dfsg-2_amd64.deb ...
Unpacking liblinear3:amd64 (2.1.0+dfsg-2) ...
Selecting previously unselected package liblua5.3-0:amd64.
Preparing to unpack .../liblua5.3-0_5.3.3-1ubuntu0.18.04.1_amd64.deb ...
Unpacking liblua5.3-0:amd64 (5.3.3-1ubuntu0.18.04.1) ...
Selecting previously unselected package nmap.
Preparing to unpack .../nmap_7.60-1ubuntu5_amd64.deb ...
Unpacking nmap (7.60-1ubuntu5) ...
Setting up libblas3:amd64 (3.7.1-4ubuntu1) ...
update-alternatives: using /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 to provide /usr/lib/x86_64-linux-gnu/libblas.so.3 (libblas.so.3-x86_64-linux-gnu) in auto mode
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Setting up liblinear3:amd64 (2.1.0+dfsg-2) ...
Setting up liblua5.3-0:amd64 (5.3.3-1ubuntu0.18.04.1) ...
Setting up nmap (7.60-1ubuntu5) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# apt-get -y install nmap          #安裝測試工具
[root@es103.yinzhengjie.com ~]# echo "{\"username\":\"yinzhengjie\",\"password\":\"123\"}" | nc -q 1 es102.yinzhengjie.com 8888        #nc命令默認是安裝的,可以直接使用
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# echo "{\"username\":\"yinzhengjie\",\"password\":\"666\"}" | ncat es102.yinzhengjie.com 8888           #需要安裝nmap工具才能使用ncat命令喲~
[root@es103.yinzhengjie.com ~]# 

3>.傳輸文件到logstash服務器

[root@es103.yinzhengjie.com ~]# cat /var/log/syslog | ncat es102.yinzhengjie.com 8888

 


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM