ELK学习笔记之logstash安装logstash-filter-multiline(在线离线安装)

ELK学习笔记之logstash安装logstash-filter-multiline(在线离线安装)

0x00 概述

ELK-logstash在搬运日志的时候会出现多行日志，普通的搬运会造成保存到ES中单条单条，很丑，而且不方便读取，logstash-filter-multiline可以解决该问题

github地址：​​原生安装

# /usr/share/logstash/bin/logstash-plugin install logstash-filter-multilineValidating logstash-filter-multilineInstalling logstash-filter-multilineWARNING: can not set Session#timeout=(0) no session context

0x02 改镜像源安装

给 Ruby 加上国内的镜像站：yum install -y gem# gem -v2.0.14.1# gem update --system# gem -v2.6.13

2. 检查并修改镜像源

# gem sources -l*** CURRENT SOURCES *** gem sources --add --remove added to sourcesremoved from sources

# cat ~/.gemrc ---:backtrace: false:bulk_threshold: 1000:sources:- true:verbose: true

3. 安装 bundle 并更改镜像源

# gem install bundler# bundle config mirror.logstash的 gem 镜像源

# vim /usr/share/logstash/Gemfile # This is a Logstash generated Gemfile.# If you modify this file manually all comments and formatting will be lost. source ""logstash-core", :path => "./logstash-core"......

# 更改默认的 为logstash-filter-multiline

# /usr/share/logstash/bin/logstash-plugin install logstash-filter-multilineValidating logstash-filter-multilineInstalling logstash-filter-multilineInstallation successful

我们看看logstash都安装了哪些插件

# /usr/share/logstash/bin/logstash-plugin listlogstash-codec-ceflogstash-codec-collectdlogstash-codec-dotslogstash-codec-ednlogstash-codec-edn_lineslogstash-codec-es_bulklogstash-codec-fluentlogstash-codec-graphitelogstash-codec-jsonlogstash-codec-json_lineslogstash-codec-linelogstash-codec-msgpacklogstash-codec-multilinelogstash-filter-multilinelogstash-codec-netflowlogstash-codec-plainlogstash-codec-rubydebuglogstash-filter-aggregatelogstash-filter-anonymizelogstash-filter-cidrlogstash-filter-clonelogstash-filter-csvlogstash-filter-datelogstash-filter-de_dotlogstash-filter-dissectlogstash-filter-dnslogstash-filter-droplogstash-filter-elasticsearchlogstash-filter-fingerprintlogstash-filter-geoiplogstash-filter-groklogstash-filter-jdbc_staticlogstash-filter-jdbc_streaminglogstash-filter-jsonlogstash-filter-kvlogstash-filter-metricslogstash-filter-mutatelogstash-filter-rubylogstash-filter-sleeplogstash-filter-splitlogstash-filter-syslog_prilogstash-filter-throttlelogstash-filter-translatelogstash-filter-truncatelogstash-filter-urldecodelogstash-filter-useragentlogstash-filter-xmllogstash-input-beatslogstash-input-dead_letter_queuelogstash-input-elasticsearchlogstash-input-execlogstash-input-filelogstash-input-ganglialogstash-input-gelflogstash-input-generatorlogstash-input-graphitelogstash-input-heartbeatlogstash-input-离线打包

这里安装好了，那么就可以打个离线的 zip 包，方便生产环境直接安装。离线包将包含所有依赖的包。

# cd /usr/share/logstash/bin# ./logstash-plugin prepare-offline-pack --overwrite --output logstash-filter-multiline.zip logstash-filter-multilineOffline package created at: logstash-filter-multiline.zip You can install it with this command `bin/logstash-plugin install file:///usr/share/logstash/bin/logstash-filter-multiline.zip`

7. logstash-filter-multiline使用方法

codec =>multiline { charset=>... #可选 字符编码 max_bytes=>... #可选 bytes类型 设置最大的字节数 max_lines=>... #可选 number类型 设置最大的行数,默认是500行 multiline_tag... #可选 string类型 设置一个事件标签,默认是multiline pattern=>... #必选 string类型 设置匹配的正则表达式 patterns_dir=>... #可选 array类型 可以设置多个正则表达式 negate=>... #可选 boolean类型 默认false不显示，可设置ture what=>... #必选 向前previous ， 向后 next }

## negate 只支持布尔值，true 或者false，默认为false。如果设置为true，表示与正则表达式（pattern）不匹配的内容都需要整合，具体整合在前还是在后，看what参数。如果设置为false，即与pattern匹配的内容## what 前一行 或者后一行，指出上面对应的规则与前一行内容收集为一行，还是与后一行整合在一起negate默认是 false，不显示与patten匹配的行由what决定 向前或向后 匹配negate 设置为true则与patten 不匹配的行由what决定 向前或向后 匹配

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。