Pormetheus k8s服务发现配置详解

Pormetheus k8s服务发现配置详解

创建sa账号，在k8s集群的master节点操作

#创建一个sa账号

对sa账号授权，这样普罗米修斯才能对k8s集群有一定的权限，采集其他节点的信息。、

[root@master ~]# kubectl create serviceaccount monitor -n monitorserviceaccount/monitor created

#把sa账号monitor通过clusterrolebing绑定到clusterrole上

[root@master prometheus]# kubectl create clusterrolebinding monitor-clusterrolebinding -n monitor --clusterrole=cluster-admin --serviceaccount=monitor:monitorclusterrolebinding.rbac.authorization.k8s.io/monitor-clusterrolebinding created

这个clusterrole具有管理员的角色，那么这个sa就可以访问k8s上面所有的资源了。

​​Relabeler - The playground for Prometheus relabeling rules​​

Kubernetes 基于角色node自动发现  node_exporter  cadvisor

scrape_interval: 15s #数据采集间隔 scrape_timeout: 10s # 数据采集超时时间，默认10s evaluation_interval: 1m # 评估告警周期

scrape_configs：配置数据源，称为target，每个target用job_name命名。又分为静态配置和服务发现 。

k8s服务发现角色有很多，如果使用node就会使用kubelet提供的 - source_labels: [__address__] #配置的原始标签，匹配地址 regex: '(.*):10250' #匹配带有10250端口的ip:10250 replacement: '${1}:9100' #把匹配到的ip:10250的ip保留替换成${1} target_label: __address__ #新生成的地址 action: replace

[root@master ~]# netstat -tpln | grep 10250tcp6 0 0 :::10250 :::* LISTEN 482/kubelet

[root@master prometheus]# netstat -tpln | grep 9100tcp6 0 0 :::9100 :::* LISTEN 22132/node_exporter

labelmap #匹配到下面正则表达式的标签会被保留

scrape_configs:#scrape_configs：配置数据源，称为target，每个target用job_name命名。又分为静态配置和服务发现 - job_name: 'kubernetes-node' kubernetes_sd_configs:#使用的是k8s的服务发现 - role: node# 使用node角色，它使用默认的kubelet提供的 relabel_configs:#重新标记 - source_labels: [__address__] #配置的原始标签，匹配地址 regex: '(.*):10250' #匹配带有10250端口的ip:10250 replacement: '${1}:9100' #把匹配到的ip:10250的ip保留替换成${1} target_label: __address__ #新生成的地址 action: replace - action: labelmap #匹配到下面正则表达式的标签会被保留 regex: __meta_kubernetes_node_label_(.+)

​​node​​

该角色发现每个群集节点的一个目标，该地址默认为 Kubelet 的 HTTP 端口。目标地址默认为地址类型顺序 、和 中的 Kubernetes 节点对象的第一个现有地址。​​node``NodeInternalIP``NodeExternalIP``NodeLegacyHostIP``NodeHostName​​

可用的元标签：

​​__meta_kubernetes_node_name​​：节点对象的名称。​​__meta_kubernetes_node_label_​​：节点对象中的每个标签。​​__meta_kubernetes_node_labelpresent_​​​：对于节点对象的每个标签。​​true​​​​__meta_kubernetes_node_annotation_​​：节点对象中的每个注释。​​__meta_kubernetes_node_annotationpresent_​​​：对于节点对象的每个注释。​​true​​​​__meta_kubernetes_node_address_​​：每个节点地址类型的第一个地址（如果存在）。

此外，节点的标签将设置为从 API 服务器检索的节点名称。​​instance​​

- job_name: 'kubernetes-node-cadvisor'# 抓取cAdvisor数据，是获取kubelet上/metrics/cadvisor接口数据来获取容器的资源使用情况 kubernetes_sd_configs: - role: node scheme: tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

[root@master ~]# kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEkubernetes ClusterIP 10.233.0.1 443/TCP 45h

上面基于node角色的服务发现，最后抓取指标的路径为 scheme + __address__ + __metrics_path__

node_exporter：${1}:9100  +  /metrics

cadvisor：kubernetes.default.svc:443  +  /api/v1/nodes/${1}/proxy/metrics/cadvisor

Kubernetes 基于角色endpoints 自动发现 Apiserver

基于不同的角色的服务发现，源标签是不一样的。

基于k8s的服务发现，这里使用的角色是endpoints

- job_name: 'kubernetes-apiserver' kubernetes_sd_configs: - role: endpoints scheme: tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace #endpoint这个对象的名称空间,__meta_kubernetes_service_name #endpoint对象的服务名字, __meta_kubernetes_endpoint_port_name #endpoint的端口名称] action: keep regex: default;kubernetes;~]# kubectl get pod -n kube-system -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATEScoredns-867b49865c-f6qbh 1/1 Running 2 45h 10.233.96.13 node2 coredns-867b49865c-m9hx4 1/1 Running 2 45h 10.233.90.9 node1

[root@master ~]# kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEcoredns ClusterIP 10.233.0.3 53/UDP,53/TCP,9153/TCP 45h[root@master ~]# kubectl get ep -n kube-system NAME ENDPOINTS AGEcoredns 10.233.90.9:53,10.233.96.13:53,10.233.90.9:53 + 3 more... 45h

[root@master ~]# kubectl get svc -n kube-systemNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEcoredns ClusterIP 10.233.0.3 53/UDP,53/TCP,9153/TCP 52dkube-scheduler-prometheus-discovery ClusterIP None 10259/TCP 36dkube-state-metrics ClusterIP 10.233.15.152 8080/TCP 47d[root@master ~]# kubectl get svc coredns -o yaml -n kube-systemapiVersion: v1kind: Servicemetadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"prometheus.io/port":"9153","prometheus.io/scrape":"true"},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","k8s-app":"kube-dns","kubernetes.io/cluster-service":"true","kubernetes.io/name":"coredns"},"name":"coredns","namespace":"kube-system"},"spec":{"clusterIP":"10.233.0.3","ports":[{"name":"dns","port":53,"protocol":"UDP"},{"name":"dns-tcp","port":53,"protocol":"TCP"},{"name":"metrics","port":9153,"protocol":"TCP"}],"selector":{"k8s-app":"kube-dns"}}} prometheus.io/port: "9153" prometheus.io/scrape: "true"

[root@master ~]# kubectl get svc coredns -n kube-system -o yamlapiVersion: v1kind: Servicemetadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"prometheus.io/port":"9153","prometheus.io/scrape":"true"},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","k8s-app":"kube-dns","kubernetes.io/cluster-service":"true","kubernetes.io/name":"coredns"},"name":"coredns","namespace":"kube-system"},"spec":{"clusterIP":"10.233.0.3","ports":[{"name":"dns","port":53,"protocol":"UDP"},{"name":"dns-tcp","port":53,"protocol":"TCP"},{"name":"metrics","port":9153,"protocol":"TCP"}],"selector":{"k8s-app":"kube-dns"}}} prometheus.io/port: "9153" prometheus.io/scrape: "true"

[root@master prometheus]# curl 10.233.90.9:9153/metrics | head -n 10 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0# HELP coredns_build_info A metric with a constant '1' value labeled by version, revision, and goversion from which CoreDNS was built.# TYPE coredns_build_info gaugecoredns_build_info{goversion="go1.14.1",revision="1766568",version="1.6.9"} 1# HELP coredns_cache_hits_total The count of cache hits.# TYPE coredns_cache_hits_total countercoredns_cache_hits_total{server="dns://:53",type="denial"} 1# HELP coredns_cache_misses_total The count of cache misses.# TYPE coredns_cache_misses_total countercoredns_cache_misses_total{server="dns://:53"} 6# HELP coredns_cache_size The number of elements in the cache.100 12115 0 12115 0 0 4249k 0 --:--:-- --:--:-- --:--:-- 5915kcurl: (23) Failed writing body (123 != 2048)

可以看到通过服务发现endpoints角色也能抓取到CoreDns暴露的据！！！！！！！！！！！！！

Pormetheus 完整yaml文件

prometheus配置文件

[root@master prometheus]# cat prometheus-cfg.yaml ---kind: ConfigMapapiVersion: v1metadata: labels: app: prometheus name: prometheus-config namespace: monitordata: prometheus.yml: | global: scrape_interval: 15s scrape_timeout: 10s evaluation_interval: 1m scrape_configs: - job_name: 'kubernetes-node' kubernetes_sd_configs: - role: node relabel_configs: - source_labels: [__address__] regex: '(.*):10250' replacement: '${1}:9100' target_label: __address__ action: replace - action: labelmap regex: __meta_kubernetes_node_label_(.+) - job_name: 'kubernetes-node-cadvisor' kubernetes_sd_configs: - role: node scheme: tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor - job_name: 'kubernetes-apiserver' kubernetes_sd_configs: - role: endpoints scheme: tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes; - job_name: 'kubernetes-service-endpoints' kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: __scheme__ regex: ( - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_service_name] action: replace target_label: kubernetes_name

prometheus deploy文件

[root@master prometheus]# cat prometheus-deploy.yaml ---apiVersion: apps/v1kind: Deploymentmetadata: name: prometheus-server namespace: monitor labels: app: prometheusspec: replicas: 1 selector: matchLabels: app: prometheus component: server #matchExpressions: #- {key: app, operator: In, values: [prometheus]} #- {key: component, operator: In, values: [server]} template: metadata: labels: app: prometheus component: server annotations: prometheus.io/scrape: 'false' spec: serviceAccountName: monitor containers: - name: prometheus image: prom/prometheus:v2.2.1 imagePullPolicy: IfNotPresent command: - prometheus - --config.file=/etc/prometheus/prometheus.yml - --storage.tsdb.path=/prometheus - --storage.tsdb.retention=720h - --web.enable-lifecycle ports: - containerPort: 9090 protocol: TCP volumeMounts: - mountPath: /etc/prometheus/prometheus.yml name: prometheus-config subPath: prometheus.yml - mountPath: /prometheus/ name: prometheus-storage-volume volumes: - name: prometheus-config configMap: name: prometheus-config items: - key: prometheus.yml path: prometheus.yml mode: 0644 - name: prometheus-storage-volume persistentVolumeClaim: claimName: prometheus---apiVersion: v1kind: PersistentVolumeClaimmetadata: name: prometheus namespace: monitorspec: storageClassName: "managed-nfs-storage" accessModes: - ReadWriteMany resources: requests: storage: 5GiYou have new mail in /var/spool/mail/root

prometheus server不被抓取到。

prometheus.io/scrape: 'false'

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。