使用prometheus-operator构建监控系统(二) —— 配置一个redis的监控告警示例

使用prometheus-operator构建监控系统(二) —— 配置一个redis的监控告警示例

Prometheus Operator默认的监控指标并不能完全满足实际的监控需求，这时候就需要我们自己根据业务添加自定义监控。这里定义一个redis的服务，并抽取其数据指标，然后根据获取的指标定义告警规则,配置邮箱告警。

1.创建redis服务

这里我们把redis_exporter以sidecar的形式和redis服务部署在用一个Pod另外为其添加了annotations：prometheus.io/scrape: "true" 和 prometheus.io/port: "9121"描述信息

创建redis的名称空间 kubectl create ns redis vim k8s-redis-and-exporter-deployment.yaml 资源清单如下: apiVersion: apps/v1 kind: Deployment metadata: namespace: redis name: redis spec: replicas: 1 selector: matchLabels: app: redis template: metadata: annotations: prometheus.io/scrape: "true" prometheus.io/port: "9121" labels: app: redis spec: containers: - name: redis image: redis resources: requests: cpu: 100m memory: 100Mi ports: - containerPort: 6379 - name: redis-exporter image: oliver006/redis_exporter:latest resources: requests: cpu: 100m memory: 100Mi ports: - containerPort: 9121

创建Redis Service

apiVersion: v1 kind: Service metadata: name: redis-svc namespace: redis labels: app: redis spec: type: NodePort ports: - name: redis port: 6379 targetPort: 6379 - name: redis-exporter port: 9121 targetPort: 9121 selector: app: redis

2.查看部署好的服务并验证metrics是否能够获取到数据

3.配置redis的Targets，rules规则

vim prometheus-serviceMonitorRedis.yaml

apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: redis-k8s namespace: monitoring labels: app: redis spec: jobLabel: redis endpoints: - port: redis-exporter interval: 30s scheme: http selector: matchLabels: app: redis namespaceSelector: matchNames: - redis

注意事项

当创建serviceMonitor后但查看prometheus配置发现serviceMonitor不生效的情况。查看prometheus状态，发现监控没有生效，查看日志，看到如下报错:

services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list services in the namespace \"monitoring\

这是因为prometheus的RBAC权限不足，访问不到对应的资源，需要修改prometheus的clusterrrole，添加一个apiGroups，修改后重启prometheus即可。

# kubectl edit ClusterRole prometheus-k8s# 添加的apiGroups

- apiGroups: - "" resources: - services - endpoints - pods verbs: - get - list - watch

下面配置rules的告警规则vim redisRules.yaml

apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus: k8s role: alert-rules name: redis-rules namespace: monitoring spec: groups: - name: redis rules: - alert: RedisUnavailable annotations: summary: redis instance info description: If redis_up == 0, redis will be unavailable //告警发送的描述信息 expr: | redis_up == 0 // 定义参数指标 for: 3m labels: severity: critical

4.配置邮箱告警

vim alertmanager.yaml

global: resolve_timeout: 5m smtp_smarthost: 'smtp.163.com:25' smtp_from: 'admin@163.com' smtp_auth_username: 'admin@163.com' smtp_auth_password: 'yihkxxxxdsawqrghfjkuiasewqeruyli;afnajyukdhwlkdsla' smtp_require_tls: false route: group_by: ['alertname'] group_wait: 30s group_interval: 10s repeat_interval: 30s // 每30s发送一个告警信息 receiver: 'mail' receivers: - name: 'mail' email_configs: - to: 'admin@163.com' send_resolved: true

此时prometheus监控系统邮箱对接完成，当服务运行触发了rule中定义的告警规则时，告警信息会自动的发送到邮箱中。

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。