k8s ingress实现http/https7层和tcp四层代理

k8s ingress实现http/https7层和tcp四层代理

ingress做为k8s集群的入口非常重要，能实现ingress功能的软件很多，可根据自身需求选择。本篇博客主要使用nginx官方提供的nginx-ingress完成了ce的版本为19.03.8-33，五台主机操作系统版本为centos7，kernel版本3.10.0-9574，使用五台主机部署，3台master节点+2台work节点5，Ingress测试域名t.myk8s.com

部署nginx和apache应用

使用Deployment部署nginx

nginx服务的manifest：

--- apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: nginx spec: selector: matchLabels: app: nginx replicas: 2 # tells deployment to run 2 pods template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.17 livenessProbe: # 设置存活探针，如果检测失败则重启pod httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 30 # 启动容器手册监控检查的等待时间 timeoutSeconds: 5 # 健康检查请求的超时时间，如果超时，则重启该pod readinessProbe: # 设置存活探针，通过探针检测的Pod才会像其转发请求 httpGet: path: / port: 80 scheme: HTTP ports: - containerPort: 80 # 设置为pod中应用监听的端口 --- apiVersion: v1 kind: Service metadata: name: nginx namespace: default labels: app: nginx spec: ports: - name: http port: 80 # service服务访问的端口 protocol: TCP targetPort: 80 # 设置为pod中应用监听的端口 selector: app: nginx

apache服务的manifests：

--- apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: httpd spec: selector: matchLabels: app: httpd replicas: 1 # tells deployment to run 2 pods template: metadata: labels: app: httpd spec: containers: - name: httpd image: httpd:2.4 livenessProbe: # 设置存活探针，如果检测失败则重启pod httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 30 # 启动容器手册监控检查的等待时间 timeoutSeconds: 5 # 健康检查请求的超时时间，如果超时，则重启该pod readinessProbe: # 设置存活探针，通过探针检测的Pod才会像其转发请求 httpGet: path: / port: 80 scheme: HTTP ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: httpd namespace: default labels: app: httpd spec: ports: - name: http port: 80 protocol: TCP targetPort: 80 selector: app: kubectl apply -f nginx-dp.yaml --record # kubectl apply -f --record

查看应用状态：

# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 1/1 Running 0 3m15s 10.107.199.77 work2 nginx-5577575ddd-499wr 1/1 Running 0 46m 10.109.125.196 work3 nginx-5577575ddd-7zwmv 1/1 Running 0 48m 10.99.1.85 work1

查看services状态，通过services暴露出来的固定ip可以实现对应应用的访问：

# kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ClusterIP 10.104.235.200 80/TCP 45m kubernetes ClusterIP 10.96.0.1 443/TCP 6d5h nginx ClusterIP 10.111.5.31 80/TCP 57m

Ingress配置vim http-ingress.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: http-ingress spec: rules: - host: t.myk8s.com http: paths: - path: /index.html backend: serviceName: nginx servicePort: 80 - path: / backend: serviceName: httpd servicePort: 80 # kubectl apply -f kubectl get ingress -o wide NAME CLASS HOSTS ADDRESS PORTS AGE t.myk8s.com 80 102m

master集群中其中一台master的ip为172.18.2.175，在本地pc测试访问:

~ curl -H 'Host: t.myk8s.com' 172.18.2.175 It works! ➜ ~ curl -H 'Host: t.myk8s.com' 172.18.2.175/index.html Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

Ingress配置openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ingress.key -out ingress.crt -subj "/CN=t.myk8s.com/O=t.myk8s.com"

通过secret存储证书:

# kubectl create secret tls ingress-secret --key ingress.key --cert ingress.crt # kubectl get secrets NAME TYPE DATA AGE ingress-secret kubernetes.io/tls 2 2s

注意如下ingress资源要和上面的secret资源在同一个namespace下面：

# vim nginx-ingress.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: http-ingress spec: tls: - hosts: - t.myk8s.com secretName: ingress-secret rules: - host: t.myk8s.com http: paths: - path: /index.html backend: serviceName: nginx servicePort: 80 - path: / backend: serviceName: httpd servicePort: 80

使用本地pc测试访问，默认 ~ curl -k It works! ➜ ~ curl -I -k HTTP/1.1 301 Moved Permanently Server: nginx/1.17.10 Date: Thu, 21 May 2020 03:53:06 GMT Content-Type: text/html Content-Length: 170 Connection: keep-alive Location: https://t.myk8s.com:443/ ➜ ~ curl -k Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

可以通过配置nginx-ingress控制器实现map的name以及namespace要和nginx -ingress控制器使用的相同：

# vim nginx-cf.yaml kind: ConfigMap apiVersion: v1 metadata: name: nginx-config data: ssl-redirect: "false" # kubectl apply -f nginx-cf.yaml -n nginx-ingress # kubectl get configmap -n nginx-ingress NAME DATA AGE nginx-config 1 6d19h

默认nginx-ingress控制器不提供tcp和udp代理功能，但是nginx是支持的，故也可以通过调整nginx-ingress控制器来提供tcp和udp代理服务

配置nginx-ingress做4层tcp代理

打开nginx官方的nginx-ingress仓库，然后修改mainifests，使nginx加载tcp和udp代理的相关配置（即在nginx-ingress.yaml的args下面添加global-configuration配置）：

# git clone https://github.com/nginxinc/kubernetes-ingress/ # cd kubernetes-ingress/deployments # git checkout v1.7.0 # vim daemon-set/nginx-ingress.yaml - -global-configuration=$(POD_NAMESPACE)/nginx-configuration # kubectl apply -f daemon-set/nginx-ingress.yaml

生成tcp代理相关的manifests：

# vim global-configuration.yaml apiVersion: k8s.nginx.org/v1alpha1 kind: GlobalConfiguration metadata: name: nginx-configuration namespace: nginx-ingress spec: listeners: - name: nginx-tcp port: 8000 protocol: TCP # vim global-configuration.yaml apiVersion: k8s.nginx.org/v1alpha1 kind: GlobalConfiguration metadata: name: nginx-configuration namespace: nginx-ingress spec: listeners: - name: nginx-tcp port: 8000 protocol: TCP [root@master1 manifests]# cat transport-server-passthrough.yaml apiVersion: k8s.nginx.org/v1alpha1 kind: TransportServer metadata: name: nginx-tcp spec: listener: name: nginx-tcp protocol: TCP upstreams: - name: nginx-app service: nginx port: 80 action: pass: nginx-app # kubectl apply -f global-configuration.yaml # kubectl apply -f transport-server-passthrough.yaml

在本地pc测试访问

➜ ~ curl http://172.18.2.175:8000/index.html Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

手动对nginx-ingress的后端nginx服务扩缩容和升级回滚

由2台水平扩容为3台，扩容期间服务无影响：

# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 1/1 Running 0 27h 10.107.199.77 work2 nginx-5577575ddd-pz59n 1/1 Running 0 18s 10.99.1.91 work1 nginx-5577575ddd-qsrb2 1/1 Running 0 18s 10.109.125.207 work3 # kubectl scale deployment nginx --replicas 3 # kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 1/1 Running 0 27h 10.107.199.77 work2 nginx-5577575ddd-j4nq4 1/1 Running 0 13s 10.107.199.84 work2 nginx-5577575ddd-pz59n 1/1 Running 0 68s 10.99.1.91 work1 nginx-5577575ddd-qsrb2 1/1 Running 0 68s 10.109.125.207 work3

由3台水平缩容为1台，缩容期间服务无影响：

# kubectl scale deployment nginx --replicas 1 # kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 1/1 Running 0 27h 10.107.199.77 work2 nginx-5577575ddd-pz59n 1/1 Running 0 102s 10.99.1.91 work1

平滑升级，先启动一个新版本的pod，然后流量切到新pod之后，再关闭老版本的pod

# kubectl set image deployment nginx nginx=nginx:1.18 --record # kubectl rollout status deployment nginx Waiting for deployment "nginx" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "nginx" rollout to finish: 1 old replicas are pending termination... deployment "nginx" successfully rolled out # kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 1/1 Running 0 27h 10.107.199.77 work2 nginx-648f9b4559-m2b8m 1/1 Running 0 19s 10.109.125.208 work3 # kubectl get deployment nginx -o yaml|grep '\- image:' - image: nginx:1.18

查看历史操作deployment的命令

# kubectl rollout history deployment nginx deployment.apps/nginx REVISION CHANGE-CAUSE 1 kubectl apply --filename=nginx-dp.yaml --record=true 2 kubectl set image deployment nginx nginx=nginx:1.18 --record=true

可以通过两种方式回滚：第一种：回滚最近一次操作

# kubectl rollout undo deployment nginx # kubectl get deployment nginx -o yaml|grep '\- image:' - image: nginx:1.17

第二种：回滚到指定的命令版本，即执行对应的命令

# kubectl rollout undo deployment nginx --to-revision=1

参考文档

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。