Kubelet 证书⾃动续期解决⽅案

Kubelet 证书⾃动续期解决⽅案

Kubelet 证书⾃动续期解决⽅案

标签（空格分隔）：kubernetes系列

[toc]

一：关于kubelet 证书的问题

1.1 k8s 证书问题

1.2 TLS bootstrapping 证书⾃动续期⽅案实施

事实上，⾃ Kubernetes 1.8 版本开始就包含了⼀个 beta 特性 —— kubelet certificate rotation，在当前的证书有 效期到达总有效期的 70% ～ 90% 时，kubelet 便会发起⼀个 的 CSR 请求从 apiserver 重新申请证书，新 证书可⽤后 就会⾃动批准续签。 注意： K8s 版本为 1.8 或更⾼时，kubelet certificate relation 会⾃动加载新证书，⽆需⼿动重启服务；K8s 版本低于 1.8 则需要⼿动重启服务才能加载新证书

1.3 配置 kubelet client/server 证书轮换

1）kube-controller-manager 配置参数 所有 master 节点都需要操作。

修改配置文件：kube-controller-manager.conf ⽂件路径: kubernetes/cfg/kube-controller-manager.conf ，完整配置⽂件内容如下： KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/data/application/kubernetes/logs \ --leader-elect=true \ --master=127.0.0.1:8080 \ --bind-address=127.0.0.1 \ --allocate-node-cidrs=true \ --cluster-cidr=10.244.0.0/16 \ --service-cluster-ip-range=10.0.0.0/16 \ --cluster-signing-cert-file=/data/application/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/data/application/kubernetes/ssl/ca-key.pem \ --root-ca-file=/data/application/kubernetes/ssl/ca.pem \ --service-account-private-key-file=/data/application/kubernetes/ssl/ca-key.pem \ --experimental-cluster-signing-duration=87600h0m0s \ --feature-gates=RotateKubeletServerCertificate=true"

1.4 k8s 1.11 之前的版本

2）kubelet 配置参数 1.K8s 1.11 版本及以前（了解即可） 注意: K8s 1.11 版本之前，RotateKubeletServerCertificate 默认是关闭的功能，需要在 kubelet 和 kube- controller-manager 中⼿动开启

修改配置⽂件: kubelet.conf 注意: 所有 node 节点都需要进⾏操作（包括作为 node 加⼊集群的 master 节点）。 vim kubernetes/cfg/kubelet.conf ----- KUBELET_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/data/application/kubernetes/logs \ --hostname-override=t-k8sN-001 \ --network-plugin=cni \ --kubeconfig=/data/application/kubernetes/cfg/kubelet.kubeconfig \ --bootstrap-kubeconfig=/data/application/kubernetes/cfg/bootstrap.kubeconfig \ --config=/data/application/kubernetes/cfg/kubelet-config.yml \ --cert-dir=/data/application/kubernetes/ssl \ --pod-infra-container-image=lizhenliang/pause-amd64:3.0 \ --node-labels=node.kubernetes.io/k8s-node=true \ --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true --rotate-certificates" -----

1.5 k8s 1.11 之后的版本

新增参数说明

文件路径: kubernetes/cfg/kubelet.conf （不推荐，后期版本逐步废弃），⽂件内容如下: KUBELET_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/data/application/kubernetes/logs \ --hostname-override=t-k8sN-001 \ --network-plugin=cni \ --kubeconfig=/data/application/kubernetes/cfg/kubelet.kubeconfig \ --bootstrap-kubeconfig=/data/application/kubernetes/cfg/bootstrap.kubeconfig \ --cert-dir=/data/application/kubernetes/ssl \ --config=/data/application/kubernetes/cfg/kubelet-config.yml \ --pod-infra-container-image=lizhenliang/pause-amd64:3.0 \ --node-labels=node.kubernetes.io/k8s-node=true \ --feature-gates=RotateKubeletServerCertificate=true \ --rotate-certificates \ --rotate-server-certificates"

1.6 创建 RBAC 规则

selfnodeclient: 自动批复 system:nodes 组⽤户更新 kubelet ⾃身与 apiserver 通讯⽤证书的 CSR 请求 （ kubelet-client-xxxx.pem ） selfnodeserver: 自动批复求system:nodes 组⽤户更新 kubelet 10250 端口API 鉴权⽤证书的 CSR 请求（ kubelet.crt）

1.7 K8s 1.8 版本前 （了解即可）

创建 clusterole vim approve-renewal-csr.yaml ----- # A ClusterRole which instructs the CSR approver to approve a user requesting # node client credentials. kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: approve-node-client-csr rules: - apiGroups: ["certificates.k8s.io"] resources: ["certificatesigningrequests/nodeclient"] verbs: ["create"] --- # A ClusterRole which instructs the CSR approver to approve a node renewing its # own client credentials. kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: approve-node-client-renewal-csr rules: - apiGroups: ["certificates.k8s.io"] resources: ["certificatesigningrequests/selfnodeclient"] verbs: ["create"] --- # A ClusterRole which instructs the CSR approver to approve a node requesting a # serving cert matching its client cert. kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: approve-node-server-renewal-csr rules: - apiGroups: ["certificates.k8s.io"] resources: ["certificatesigningrequests/selfnodeserver"] verbs: ["create"] ---- 创建资源： kubectl apply -f approve-renewal-csr.yaml 创建：ClusterRoleBinding 自动批准首次申请证书 还没有证书，所以是组system:node-bootsrapper 组 kubectl create clusterrolebinding node-client-auto-approve-csr --clusterrole=approve- node-client-csr --group=system:node-bootstrapper 自动批准更新 kubelet 自身与 apiserver 通讯用证书 已有证书，所以是组system:nodes组 kubectl create clusterrolebinding node-client-auto-renew-crt --clusterrole=approve- node-client-renewal-csr --group=system:nodes 自动批准更新 kubelet 10250 端口 API鉴权⽤证书 已有证书，所以是system：nodes组 kubectl create clusterrolebinding node-server-auto-renew-crt --clusterrole=approve- node-server-renewal-csr --group=system:nodes 如上创建后可以通过 kubectl get clusterrolebinding|egrep "node-(.*)-auto" 进⾏查看，结果如下:

1.8 k8s 1.8 之后的版本

我们上⾯说到，K8s 提供了三种 ClusterRole 来进⾏ CSR 的自动批复⼯作。在 K8s 1.8 版本前都需要手动创 建，而从 1.8 版本起 kube-apiserver 会自动创建nodeclient 和 selfnodeclient ，所以我们只需要再⼿ 动创建selfnodeserver就可以了。

创建clusteRole vim auto-approve-csr-clusterrole.yml # A ClusterRole which instructs the CSR approver to approve a node requesting # a serving cert matching its client cert. kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver rules: - apiGroups: ["certificates.k8s.io"] resources: ["certificatesigningrequests/selfnodeserver"] verbs: ["create"]

创建ClusterRoleBinding vim auto-approve-csr-clusterrolebinding.yaml # 批复 "system:node-bootstrapper" 组的所有 CSR apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: node-client-auto-approve-csr subjects: - kind: Group name: system:node-bootstrapper apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:nodeclient apiGroup: rbac.authorization.k8s.io --- # 批复 "system:nodes" 组 kubelet 与 apiserver 通信证书续约请求 apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: node-client-auto-renew-crt subjects: - kind: Group name: system:nodes apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient apiGroup: rbac.authorization.k8s.io --- # 批复 "system:nodes" 组 kubelet 10250端⼝ API 鉴权证书续约请求 apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: node-server-auto-renew-crt subjects: - kind: Group name: system:nodes apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver apiGroup: rbac.authorization.k8s.io

kubectl get clusterrolebinding|egrep "node-(.*)-auto"

二： 重启服务与删除证书

2.1 重启kube-controller-manger

重启kube-controller-manager 如果有多个master 就逐个重启 systemctl daemon-reload systemctl restart kube-controller-manager.service

2.2 删掉kubelet的证书

查看默认证书时间 openssl x509 -in kubelet-client-current.pem -noout -text | grep "Not"

rm -rf /opt/kubernetes/ssl/kubelet* [所有节点的证书] 重新启动 节点kubelet service kubelet restart

查看原 kubelet server 证书的有效期 会少了两个文件： kubelet.crt , kubelet.key 文件 默认情况下签发的 kubelet server 证书的有效期只有 1 年，并不受 kube-controller-manager 控制其证书有效期。

2.3 重新签发证书

重新签发证书： kubectl get csr

kubectl get csr|grep "Pending"|awk 'NR>1{print $1}'|xargs kubectl certificate approve kubectl certificate approve csr-f7jl6

证书管理： openssl x509 -in /opt/kubernetes/ssl/kubelet-client-current.pem -noout -text| grep Not

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。