kubernetes(k8s)构建elk(filebeat)日志收集系统 - k8s系列(四)

kubernetes(k8s)构建elk(filebeat)日志收集系统 - k8s系列(四)

环境背景

已实现 k8s系列(一) - 使用kubeadm安装kubernetes(k8s) 已实现 k8s系列(二) - jenkins+kubernetes(k8s)+docker持续集成与部署(CI/CD) - 收集日志使用 km - 2cpu - 4g内存 - ip - 192.168.23.39 node1 - 2cpu - 2G内存 - ip - 192.168.23.40 node1 - 2cpu - 2G内存 - ip - 192.168.23.41 基于 k8s系列(二) 已有可正常运行的k8s工作任务 所有操作基于官方文档说明

镜像准备

1.拉取所需镜像

elasticsearch/kibana/filebeat版本必须相同

docker pull elasticsearch:8.1.1 docker pull kibana:8.1.1 docker pull docker.elastic.co/beats/filebeat:8.1.1

2.标记自定义tag

docker tag elasticsearch:8.1.1 192.168.23.39:5000/elasticsearch:8.1.1 docker tag kibana:8.1.1 192.168.23.39:5000/kibana:8.1.1 docker tag docker.elastic.co/beats/filebeat:8.1.1 192.168.23.39:5000/filebeat:8.1.1

3.推送到私有镜像仓库

docker push 192.168.23.39:5000/elasticsearch:8.1.1 docker push 192.168.23.39:5000/kibana:8.1.1 docker push 192.168.23.39:5000/filebeat:8.1.1

搭建elk(filebeat)

1.添加Elastic官方自定义资源对象

wget https://download.elastic.co/downloads/eck/2.1.0/crds.yaml wget https://download.elastic.co/downloads/eck/2.1.0/operator.yaml kubectl apply -f crds.yaml kubectl apply -f operator.yaml

2.创建存储卷 - pv

不创建存储卷将会报错 - pod has unbound immediate PersistentVolumeClaims

1).创建pv.yaml文件

apiVersion: v1 kind: PersistentVolume metadata: name: elasticsearch-data spec: capacity: storage: 1Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local-storage local: path: /www/wwwroot/mnt nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - km

2).创建PV

kubectl apply -f pv.yaml

2.将master标记为可调度

当master节点不可调度的情况下，将会报错 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 Insufficient memory

#标记master可调度 kubectl taint nodes --all node-role.kubernetes.io/master- #取消master可调度 kubectl taint nodes km node-role.kubernetes.io/master=true:NoSchedule

3.创建elasticsearch

1).创建elasticsearch.yaml文件

apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: es spec: image: 192.168.23.39:5000/elasticsearch:8.1.1 version: 8.1.1 nodeSets: - name: master count: 1 config: node.store.allow_mmap: false volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: - ReadWriteOnce resources: requests: memory: 1Gi cpu: 1 storage: 1Gi limits: memory: 1Gi storageClassName: local-storage

2).创建elasticsearch

kubectl apply -f elasticsearch.yaml

3).查看pod信息

kubectl describe pods es-es-master-0

4.创建kibana

1).创建kibana.yaml文件

apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: kibana spec: image: 192.168.23.39:5000/kibana:8.1.1 version: 8.1.1 count: 1 elasticsearchRef: name: es config: i18n.locale: "zh-CN" http: service: spec: type: LoadBalancer tls: selfSignedCertificate: subjectAltNames: - ip: 192.168.23.39

2).创建kibana

kubectl apply -f elasticsearch.yaml

3).查看相关信息

#查看svc kubectl get service kibana-kb-http #显示如下 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kibana-kb- LoadBalancer 10.111.40.122 5601:30767/TCP 2m27s #kubectl port-forward service/kibana-kb-5601 #PASSWORD=$(kubectl get secret es-es-elastic-user -o go-template='{{.data.elastic | base64decode}}') #查看管理员elastic密码 kubectl get secret es-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode; echo

#本机访问 - - svc 中 PORT(S) 5601:30767/TCP 账号:elastic 密码:如上 如果想使用请在kibana.yaml文件http相关配置,添加以下代码 http: tls: selfSignedCertificate: disabled: true

4.创建filebeat

1).创建filebeat.yaml文件

apiVersion: beat.k8s.elastic.co/v1beta1 kind: Beat metadata: name: beat spec: image: 192.168.23.39:5000/filebeat:8.1.1 type: filebeat version: 8.1.1 elasticsearchRef: name: es config: filebeat.inputs: - type: container paths: - /var/log/containers/*.log daemonSet: podTemplate: spec: dnsPolicy: ClusterFirstWithHostNet hostNetwork: true securityContext: runAsUser: 0 containers: - name: filebeat volumeMounts: - name: varlogcontainers mountPath: /var/log/containers - name: varlogpods mountPath: /var/log/pods - name: varlibdockercontainers mountPath: /var/lib/docker/containers volumes: - name: varlogcontainers hostPath: path: /var/log/containers - name: varlogpods hostPath: path: /var/log/pods - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers

2).创建kibana

kubectl apply -f filebeat.yaml

查看elk(filebeat)日志记录

1.创建数据视图

2.模拟请求创建ELK(filebeat)日志记录

#查看已启动的项目 kubectl get svc #模拟请求 curl 10.101.102.226/static/none.jpg > /dev/null

3.查看ELK(filebeat)日志记录

结束

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。