腾讯云运行Ingress控制器——实战

腾讯云运行Ingress控制器——实战

背景

每次业务变更，nginx worker进程都得执行 reload。随着业务体量增加，reload 会越来越频繁，拆分 ingress 可以有效避免业务互相影响。

​​ingress-nginx​​​提供了运行多个​​nginx​​​入口控制器的能力，例如一个服务于公网流量，一个服务于内网流量，也可以根据业务来拆分，使得不同的​​ingress​​​使用不同的​​ingress controller​​。

Ingress原理

原理：

外部负载均衡器​​externalLB​​​请求调至到 ​​nodeport​​ 里面 service服务 ---> 调度到内部Pod(ingress controller里面) -----> 根据ingree定义，是虚拟主机，还是url代理 ----> 假设是主机名，一组主机名对应后端的pod资源Pod1，Pod2，Pod3。Pod怎么分组通过 service 进行分组。才能被 ingress 引用。

动态生效：

Pod变化 -> Service变化 -> ingress变化 -> 注入 ingress controller

环境准备

集群节点

集群内运行两台节点，用做测试

[root@VM-1-6-centos ingress]# kubectl get nodeNAME STATUS ROLES AGE VERSION172.16.1.3 Ready 5h22m v1.20.6-tke.3172.16.1.6 Ready 5h22m v1.20.6-tke.3

分别给两台节点打标签，一台运行​​pub​​​的​​ingress controller​​​，一台运行​​internal​​​的​​ingress controller​​

[root@VM-1-6-centos ingress]# kubectl label node 172.16.1.3 ingress-role=pub[root@VM-1-6-centos ingress]# kubectl label node 172.16.1.6 ingress-role=internal

创建负载均衡

部署

rbac

apiVersion: v1kind: ServiceAccountmetadata: name: nginx-ingress-serviceaccount namespace: ingress-nginx---apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRolemetadata: name: nginx-ingress-clusterrolerules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "extensions" resources: - ingresses verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "extensions" resources: - ingresses/status verbs: - update---apiVersion: rbac.authorization.k8s.io/v1beta1kind: Rolemetadata: name: nginx-ingress-role namespace: ingress-nginxrules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "-" # Here: "-" # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. - "ingress-controller-leader-nginx" verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - "" resources: - endpoints verbs: - get---apiVersion: rbac.authorization.k8s.io/v1beta1kind: RoleBindingmetadata: name: nginx-ingress-role-nisa-binding namespace: ingress-nginxroleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-ingress-rolesubjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx---apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata: name: nginx-ingress-clusterrole-nisa-bindingroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: nginx-ingress-clusterrolesubjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx

configmap

apiVersion: v1kind: ConfigMapmetadata: name: nginx-configuration namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx

default-apps/v1 kind: Deploymentmetadata: name: default- labels: app: default- namespace: ingress-nginxspec: selector: matchLabels: app: default- replicas: 1 template: metadata: labels: app: default- spec: terminationGracePeriodSeconds: 60 containers: - name: default- image: registry.cn-hangzhou.aliyuncs.com/google_containers/defaultbackend:1.4 livenessProbe: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 ports: - containerPort: 8080 resources: limits: cpu: 10m memory: 20Mi requests: cpu: 10m memory: 20Mi---apiVersion: v1kind: Servicemetadata: name: default- namespace: ingress-nginx labels: app: default- ports: - port: 80 targetPort: 8080 selector: app: default-准备 yaml 文件目录[root@VM-1-6-centos data]# mkdir ingress/{pub,internal} -p[root@VM-1-6-centos data]# cd ingress/pub

apiVersion: apps/v1 kind: DaemonSetmetadata: name: nginx-ingress-controller-pub namespace: ingress-nginx spec: selector: matchLabels: app.kubernetes.io/name: ingress-nginx-pub app.kubernetes.io/instance: ingress-nginx-pub app.kubernetes.io/component: controller-pub template: metadata: labels: app.kubernetes.io/name: ingress-nginx-pub app.kubernetes.io/instance: ingress-nginx-pub app.kubernetes.io/component: controller-pub annotations: prometheus.io/port: '10254' prometheus.io/scrape: 'true' spec: serviceAccountName: nginx-ingress-serviceaccount hostNetwork: true nodeSelector: ingress-role: "pub" containers: - name: nginx-ingress-controller image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.20.0 args: - /nginx-ingress-controller ## 重要 - --ingress-class=nginx-pub - --default-backend-service=$(POD_NAMESPACE)/default- - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: containerPort: 80 - name: containerPort: 443 livenessProbe: failureThreshold: 3 path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 readinessProbe: failureThreshold: 3 path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1---apiVersion: v1kind: Servicemetadata: labels: helm.sh/chart: ingress-nginx-2.11.1 app.kubernetes.io/name: ingress-nginx-pub app.kubernetes.io/instance: ingress-nginx-pub app.kubernetes.io/version: 0.34.1 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller-pub annotations: service.cloud.tencent.com/local-svc-weighted-balance: "true" service.kubernetes.io/local-svc-only-bind-node-with-pod: "true" service.kubernetes.io/tke-existed-lbid: lb-g82xo9yj name: ingress-nginx-controller-pub namespace: ingress-nginxspec: type: LoadBalancer externalTrafficPolicy: Local ports: - name: port: 80 protocol: TCP targetPort: - name: port: 443 protocol: TCP targetPort: selector: app.kubernetes.io/name: ingress-nginx-pub app.kubernetes.io/instance: ingress-nginx-pub app.kubernetes.io/component: controller-pub

ingress-controller-internal

apiVersion: apps/v1 kind: DaemonSetmetadata: name: nginx-ingress-controller-internal namespace: ingress-nginx spec: selector: matchLabels: app.kubernetes.io/name: ingress-nginx-internal app.kubernetes.io/instance: ingress-nginx-internal app.kubernetes.io/component: controller-internal template: metadata: labels: app.kubernetes.io/name: ingress-nginx-internal app.kubernetes.io/instance: ingress-nginx-internal app.kubernetes.io/component: controller-internal annotations: prometheus.io/port: '10254' prometheus.io/scrape: 'true' spec: serviceAccountName: nginx-ingress-serviceaccount hostNetwork: true nodeSelector: ingress-role: "internal" containers: - name: nginx-ingress-controller image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.20.0 args: - /nginx-ingress-controller ## 重要 - --ingress-class=nginx-internal - --default-backend-service=$(POD_NAMESPACE)/default- - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: containerPort: 80 - name: containerPort: 443 livenessProbe: failureThreshold: 3 path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 readinessProbe: failureThreshold: 3 path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 ---apiVersion: v1kind: Servicemetadata: labels: helm.sh/chart: ingress-nginx-2.11.1 app.kubernetes.io/name: ingress-nginx-internal app.kubernetes.io/instance: ingress-nginx-internal app.kubernetes.io/version: 0.34.1 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller-internal annotations: service.cloud.tencent.com/local-svc-weighted-balance: "true" service.kubernetes.io/local-svc-only-bind-node-with-pod: "true" service.kubernetes.io/tke-existed-lbid: lb-7pztvmq7 name: ingress-nginx-controller-internal namespace: ingress-nginxspec: type: LoadBalancer externalTrafficPolicy: Local ports: - name: port: 80 protocol: TCP targetPort: - name: port: 443 protocol: TCP targetPort: selector: app.kubernetes.io/name: ingress-nginx-internal app.kubernetes.io/instance: ingress-nginx-internal app.kubernetes.io/component: controller-internal

全部部署后，查看资源

EXTERNAL-IP 应该对应CLB的VIP地址

[root@VM-1-6-centos ~]# kubectl get po -n ingress-nginxNAME READY STATUS RESTARTS AGEdefault- 1/1 Running 0 5h26mnginx-ingress-controller-internal-jvwsw 1/1 Running 0 4h34mnginx-ingress-controller-pub-qb8t9 1/1 Running 0 5h9m[root@VM-1-6-centos ~]# kubectl get svc -n ingress-nginxNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEdefault- ClusterIP 172.17.253.191 80/TCP 5h27mingress-nginx-controller-internal LoadBalancer 172.17.254.224 172.16.1.15 80:32665/TCP,443:32702/TCP 4h46mingress-nginx-controller-pub LoadBalancer 172.17.253.203 1.15.158.231 80:30572/TCP,443:32013/TCP 5h7m

部署服务验证

公网流量服务部署

注意: ​​kubernetes.io/ingress.class: nginx-pub​​

apiVersion: apps/v1kind: Deploymentmetadata: name: nginx-pub spec: replicas: 1 selector: matchLabels: app: nginx-pub template: metadata: labels: app: nginx-pub spec: containers: - name: nginx image: nginx:1.19.5---apiVersion: v1kind: Servicemetadata: name: nginx-pubspec: ports: - port: 80 protocol: TCP targetPort: 80 selector: app: nginx-pub sessionAffinity: None type: ClusterIP---apiVersion: networking.k8s.io/v1kind: Ingressmetadata: annotations: kubernetes.io/ingress.class: nginx-pub name: ingress-pubspec: rules: - host: pub.test.com paths: - backend: service: name: nginx-pub port: number: 80 path: / pathType: ImplementationSpecific

内网流量服务部署

apiVersion: apps/v1kind: Deploymentmetadata: name: nginx-internalspec: replicas: 1 selector: matchLabels: app: nginx-internal template: metadata: labels: app: nginx-internal spec: containers: - name: nginx image: nginx:1.19.5---apiVersion: v1kind: Servicemetadata: name: nginx-internalspec: ports: - port: 80 protocol: TCP targetPort: 80 selector: app: nginx-internal sessionAffinity: None type: ClusterIP---apiVersion: networking.k8s.io/v1kind: Ingressmetadata: annotations: kubernetes.io/ingress.class: nginx-internal name: ingress-internalspec: rules: - host: internal.test.com paths: - backend: service: name: nginx-internal port: number: 80 path: / pathType: ImplementationSpecific

测试

分别改变​​pub​​​和​​internal​​​ 的 Pod 中的html后，绑定​​hosts​​测试

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。