一、安裝multiline
在使用elk 傳輸記錄 java 日志時，如下
一個java的報錯
在elk中會按每一行 產生多條記錄，不方便查閱
這里修改配置文件 使用 multiline 插件 即可實現多行合一的 輸出模式
修改配置文件
# vi /etc/logstash/conf.d/logstash.conf input { file { path => "/w_logs/error.log.2018-06-05" type => "test" } } filter { multiline { pattern => "^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}" negate => true what => "previous" } grok { match => [ "message", "%{NOTSPACE:day} %{NOTSPACE:datetime} %{NOTSPACE:level} %{GREEDYDATA:msginfo} " ] } } output { if [type] == "test" { elasticsearch { hosts => ["10.10.15.95:9200"] index => "12.83-test" } } }
修改完 重啟logstash
報錯：
[ERROR] 2018-07-13 15:37:59.834 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] registry - Tried to load a plugin's code, but failed.
{:exception=>#<LoadError: no such file to load -- logstash/filters/multiline>, :path=>"logstash/filters/multiline", :type=>"filter", :name=>"multiline"}
[ERROR] 2018-07-13 15:37:59.838 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent -
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any filter plugin named 'multiline'. Are you sure this is correct? Trying to load the multiline filter plugin resulted in this error: no such file to load -- logstash/filters/multiline", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:192:in `lookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:140:in `lookup'", "/usr/share/logstash/logstash-core/lib/logs
提示缺少 插件 filters/multiline
我們看看logstash都安裝了哪些插件
# /usr/share/logstash/bin/logstash-plugin list logstash-codec-cef logstash-codec-collectd logstash-codec-dots logstash-codec-edn logstash-codec-edn_lines logstash-codec-es_bulk logstash-codec-fluent logstash-codec-graphite logstash-codec-json logstash-codec-json_lines logstash-codec-line logstash-codec-msgpack logstash-codec-multiline logstash-codec-netflow logstash-codec-plain logstash-codec-rubydebug logstash-filter-aggregate logstash-filter-anonymize logstash-filter-cidr logstash-filter-clone logstash-filter-csv logstash-filter-date logstash-filter-de_dot logstash-filter-dissect logstash-filter-dns logstash-filter-drop logstash-filter-elasticsearch logstash-filter-fingerprint logstash-filter-geoip logstash-filter-grok logstash-filter-jdbc_static logstash-filter-jdbc_streaming logstash-filter-json logstash-filter-kv logstash-filter-metrics logstash-filter-mutate logstash-filter-ruby logstash-filter-sleep logstash-filter-split logstash-filter-syslog_pri logstash-filter-throttle logstash-filter-translate logstash-filter-truncate logstash-filter-urldecode logstash-filter-useragent logstash-filter-xml logstash-input-beats logstash-input-dead_letter_queue logstash-input-elasticsearch logstash-input-exec logstash-input-file logstash-input-ganglia logstash-input-gelf logstash-input-generator logstash-input-graphite logstash-input-heartbeat logstash-input-http logstash-input-http_poller logstash-input-imap logstash-input-jdbc logstash-input-kafka logstash-input-pipe logstash-input-rabbitmq logstash-input-redis logstash-input-s3 logstash-input-snmptrap logstash-input-sqs logstash-input-stdin logstash-input-syslog logstash-input-tcp logstash-input-twitter logstash-input-udp logstash-input-unix logstash-output-cloudwatch logstash-output-csv logstash-output-elasticsearch logstash-output-email logstash-output-file logstash-output-graphite logstash-output-http logstash-output-kafka logstash-output-lumberjack logstash-output-nagios logstash-output-null logstash-output-pagerduty logstash-output-pipe logstash-output-rabbitmq logstash-output-redis logstash-output-s3 logstash-output-sns logstash-output-sqs logstash-output-stdout logstash-output-tcp logstash-output-udp logstash-output-webhdfs logstash-patterns-core
有一個logstash-codec-multiline
並沒有我們需要的 logstash-filter-multiline
我們來安裝這個插件，先看一下 logstash-plugin 的用法
Usage: bin/logstash-plugin [OPTIONS] SUBCOMMAND [ARG] ... Parameters: SUBCOMMAND subcommand [ARG] ... subcommand arguments Subcommands: list List all installed Logstash plugins install Install a Logstash plugin remove Remove a Logstash plugin update Update a plugin pack Package currently installed plugins, Deprecated: Please use prepare-offline-pack instead unpack Unpack packaged plugins, Deprecated: Please use prepare-offline-pack instead generate Create the foundation for a new plugin uninstall Uninstall a plugin. Deprecated: Please use remove instead prepare-offline-pack Create an archive of specified plugins to use for offline installation Options: -h, --help print help
安裝插件是 # logstash-plugin install logstash-filter-multiline
# logstash-plugin install logstash-filter-multiline Validating logstash-filter-multiline Installing logstash-filter-multiline Installation successfu
二、multiline 使用方法
codec =>multiline { charset=>... #可選 字符編碼 max_bytes=>... #可選 bytes類型 設置最大的字節數 max_lines=>... #可選 number類型 設置最大的行數,默認是500行 multiline_tag... #可選 string類型 設置一個事件標簽,默認是multiline pattern=>... #必選 string類型 設置匹配的正則表達式 patterns_dir=>... #可選 array類型 可以設置多個正則表達式 negate=>... #可選 boolean類型 默認false不顯示，可設置ture what=>... #必選 向前previous ， 向后 next
}
## negate 只支持布爾值，true 或者false，默認為false。
如果設置為true，表示與正則表達式（pattern）不匹配的內容都需要整合，
具體整合在前還是在后，看what參數。如果設置為false，即與pattern匹配的內容
## what 前一行 或者后一行，指出上面對應的規則與前一行內容收集為一行，還是與后一行整合在一起
簡單來說：
negate默認是 false，不顯示 與patten匹配的行 由what決定 向前或向后 匹配
negate 設置為true
則與patten 不匹配的行
由what決定 向前或向后 匹配