在CentOS7中部署ELK日志分析系统

在CentOS7中部署ELK日志分析系统

在CentOS7中部署ELK日志分析系统

ELK原理介绍

什么是ELK

ELK是三个开源软件的缩写，分别表示：Elasticsearch , Logstash, Kibana , 它们都是开源软件。新增了一个FileBeat，它是一个轻量级的日志收集处理工具(Agent)，Filebeat占用资源少，适合于在各个服务器上搜集日志后传输给Logstash，官方也推荐此工具。

Elasticsearch是实时全文搜索和分析引擎，提供搜集、分析、存储数据三大功能；是一套开放REST和JAVA API等结构提供高效搜索功能，可扩展的分布式系统。它构建于Apache Lucene搜索引擎库之上。

Logstash是一个用来搜集、分析、过滤日志的工具。它支持几乎任何类型的日志，包括系统日志、错误日志和自定义应用程序日志。它可以从许多来源接收日志，这些来源包括 syslog、消息传递（例如 RabbitMQ）和JMX，它能够以多种方式输出数据，包括电子邮件、websockets和Elasticsearch。

Kibana是一个基于Web的图形界面，用于搜索、分析和可视化存储在 Elasticsearch指标中的日志数据。它利用Elasticsearch的REST接口来检索数据，不仅允许用户创建他们自己的数据的定制仪表板视图，还允许他们以特殊的方式查询和过滤数据。

实验环境

实验实施

安装elasticsearch

我们这次做的是搭建两个elasticsearch节点，做分布式搜索及存储，首先修改yum源，使用yum安装elasticsearch，注意elasticsearch服务器内存需要大于2G

[root@promote ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch #导入GPG校验密钥 [root@promote ~]# vim /etc/yum.repos.d/elasticsearch.repo #创建repo的源文件，代码如下 [elasticsearch-2.x] name=Elasticsearch repository for 2.x packages baseurl=http://packages.elastic.co/elasticsearch/2.x/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enable=1 [root@promote ~]# yum install elasticsearch -y #使用yum安装elasticsearch软件包

安装java环境，直接使用yum安装

[root@promote ~]# yum install java -y #使用java -version测试java环境是否搭建好 [root@promote ~]# java -version openjdk version "1.8.0_181" OpenJDK Runtime Environment (build 1.8.0_181-b13) OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode) #可以看到java已经更新到最新版本

修改elasticsearch配置文件

[root@promote ~]# cd /etc/elasticsearch/ [root@promote elasticsearch]# vim elasticsearch.yml

[root@promote elasticsearch]# mkdir -p /data/es-data [root@promote elasticsearch]# chown -R elasticsearch:elasticsearch /data/es-data/

启动服务，并查看9200端口是否开启

[root@promote elasticsearch]# systemctl start elasticsearch.service [root@promote elasticsearch]# netstat -ntap | grep 9200 tcp6 0 0 :::9200 :::* LISTEN 90165/java #可以看到9200端口已经开启

[root@promote elasticsearch]# curl -i -XGET '-d '{> "query": { > "match_all": {} > } > }' HTTP/1.1 200 OK Content-Type: application/json; charset=UTF-8 Content-Length: 95 { "count" : 0, "_shards" : { "total" : 0, "successful" : 0, "failed" : 0 } } #测试成功

可以看到上面两种交互方式并不友好，我们可以通过安装head插件，进行更加友好的访问。

[root@promote elasticsearch]# /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head -> Installing mobz/elasticsearch-head... Trying ... ....省略 Verifying checksums if available ... NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify) Installed head into /usr/share/elasticsearch/plugins/head

[root@promote elasticsearch]# systemctl start elasticsearch.service [root@promote elasticsearch]# netstat -ntap | grep 9200 tcp6 0 0 :::9200 :::* LISTEN 2194/java

[root@promote elasticsearch]# /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf -> Installing lmenezes/elasticsearch-kopf... Trying ... ....省略 Verifying checksums if available ... NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify) Installed kopf into /usr/share/elasticsearch/plugins/kopf

安装logstash

配置yum源文件

[root@~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch #导入软件包校验密钥 [root@~]# vim /etc/yum.repos.d/logstash.repo [logstash-2.1] name=Logstash repository for 2.1.x packages baseurl=http://packages.elastic.co/logstash/2.1/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enable=1 [root@yum.repos.d]# yum install logstash -y #安装logstash服务

可以测试logstash

[root@yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }' OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Settings: Default filter workers: 1 Logstash startup completed abc123 2018-08-21T14:07:37.666Z abc123 test 2018-08-21T14:07:46.156Z test #可以看到我们输入什么，后面就会直接输出什么内容

按住Ctrl+c退出后，换一种格式输入输出

[root@yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug } }' OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Settings: Default filter workers: 1 Logstash startup completed abc123 { "message" => "abc123", "@version" => "1", "@timestamp" => "2018-08-21T14:09:18.094Z", "host" => "www1.yx.com" } #这是详细格式输出，可以看到更加详细的内容

同样，我们可以将输入内容输出到elasticsearch中。

[root@yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["192.168.58.147:9200"] } }' OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Settings: Default filter workers: 1 Logstash startup completed abc123 test123 123456

[root@promote ~]# ln -s /opt/logstash/bin/logstash /usr/bin/ [root@promote ~]# vim file.conf input { file { path => "/var/log/messages" type => "system" start_position => "beginning" } } output { elasticsearch { hosts => ["192.168.58.147:9200"] index => "system-%{+YYYY.MM.dd}" } }

下面我们尝试多个服务日志，修改file.conf.

input { file { path => "/var/log/messages" type => "system" start_position => "beginning" } file { path => "/var/log/httpd/access_log" type => "httpd" start_position => "beginning" } } output { if [type] == "system" { elasticsearch { hosts => ["192.168.58.147:9200"] index => "system-%{+YYYY.MMdd}" } } if [type] == "{ elasticsearch { hosts => ["192.168.58.147:9200"] index => "httpd-%{+YYYY.MMdd}" } } }

安装kibana

下载kibana

[root@localhost ~]# wget https://download.elastic.co/kibana/kibana/kibana-4.3.1-linux-x64.tar.gz --2018-08-21 23:02:18-- https://download.elastic.co/kibana/kibana/kibana-4.3.1-linux-x64.tar.gz 正在解析主机 download.elastic.co (download.elastic.co)... 54.235.171.120, 54.225.214.74, 54.225.221.128, ... 正在连接 download.elastic.co (download.elastic.co)|54.235.171.120|:443... 已连接。 已发出 HTTP 请求，正在等待回应... 200 OK 长度：30408272 (29M) [binary/octet-stream] 正在保存至: “kibana-4.3.1-linux-x64.tar.gz” 100%[==================================================>] 30,408,272 512KB/s 用时 82s 2018-08-21 23:03:43 (361 KB/s) - 已保存 “kibana-4.3.1-linux-x64.tar.gz” [30408272/30408272]

解压kibana到指定目录

[root@localhost ~]# tar zxvf kibana-4.3.1-linux-x64.tar.gz -C /opt/

将解压的目录重命名为kibana

[root@localhost ~]# mv /opt/kibana-4.3.1-linux-x64/ /opt/kibana/

修改kibana配置文件

[root@localhost config]# vim /opt/kibana/config/kibana.yml

[root@localhost config]# /opt/kibana/bin/kibana

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。