linux cpu占用率如何看
275
2022-09-09
#云原生征文#自建高可用k8s集群搭建
#云原生征文#自建高可用k8s集群搭建
自建高可用k8s集群搭建
一、所有节点基础环境
192.168.0.x ： 为机器的网段10.96.0.0/16: 为Service网段196.16.0.0/16: 为Pod网段
1、环境准备与内核升级
先升级所有机器内核
#我的机器版本cat /etc/redhat-release # CentOS Linux release 7.9.2009 (Core)#修改域名，一定不是localhosthostnamectl set-hostname k8s-xxx#集群规划k8s-master1 k8s-master2 k8s-master3 k8s-master-lb k8s-node01 k8s-node02 ... k8s-nodeN# 每个机器准备域名vi /etc/hosts192.168.0.10 k8s-master1192.168.0.11 k8s-master2192.168.0.12 k8s-master3192.168.0.13 k8s-node1192.168.0.14 k8s-node2192.168.0.15 k8s-node3192.168.0.250 k8s-master-lb # 非高可用，可以不用这个。这个使用keepalive配置
# 关闭selinuxsetenforce 0sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinuxsed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
# 关闭swapswapoff -a && sysctl -w vm.swappiness=0sed -ri 's/.*swap.*/#&/' /etc/fstab
#修改limitulimit -SHn 65535vi /etc/security/limits.conf# 末尾添加如下内容* soft nofile 655360* hard nofile 131072* soft nproc 655350* hard nproc 655350* soft memlock unlimited* hard memlock unlimited
#为了方便以后操作配置ssh免密连接，master1运行ssh-keygen -t rsafor i in k8s-master1 k8s-master2 k8s-master3 k8s-node1 k8s-node2 k8s-node3;do ssh-copy-id -i .ssh/id_rsa.pub $i;done
#安装后续用的一些工具yum install wget git jq psmisc net-tools yum-utils device-mapper-persistent-data lvm2 -y
# 所有节点# 安装ipvs工具,方便以后操作ipvs，ipset，conntrack等yum install ipvsadm ipset sysstat conntrack libseccomp -y# 所有节点配置ipvs模块,执行以下命令，在内核4.19+版本改为nf_conntrack， 4.18下改为nf_conntrack_ipv4modprobe -- ip_vsmodprobe -- ip_vs_rrmodprobe -- ip_vs_wrrmodprobe -- ip_vs_shmodprobe -- nf_conntrack#修改ipvs配置，加入以下内容vi /etc/modules-load.d/ipvs.confip_vsip_vs_lcip_vs_wlcip_vs_rrip_vs_wrrip_vs_lblcip_vs_lblcrip_vs_dhip_vs_ship_vs_foip_vs_nqip_vs_sedip_vs_ftpip_vs_shnf_conntrackip_tablesip_setxt_setipt_setipt_rpfilteript_REJECTipip# 执行命令systemctl enable --now systemd-modules-load.service #--now = enable+start#检测是否加载lsmod | grep -e ip_vs -e nf_conntrack
## 所有节点cat <
# 所有节点配置完内核后，重启服务器，保证重启后内核依旧加载rebootlsmod | grep -e ip_vs -e nf_conntrack
2、安装Docker
# 安装dockeryum remove docker*yum install -y yum-utilsyum-config-manager --add-repo install -y docker-ce-19.03.9 docker-ce-cli-19.03.9 containerd.io-1.4.4
# 修改docker配置,新版kubelet建议使用systemd，所以可以把docker的CgroupDriver改成systemdmkdir /etc/dockercat > /etc/docker/daemon.json < #也可以自己下载rpm离线包进行安装localinstall xxxx 二、PKI 百度百科：公钥基础设施_百度百科 Kubernetes 需要 PKI 才能执行以下操作： Kubelet 的客户端证书，用于 API 服务器身份验证API 服务器端点的证书集群管理员的客户端证书，用于 API 服务器身份认证API 服务器的客户端证书，用于和 Kubelet 的会话API 服务器的客户端证书，用于和 etcd 的会话控制器管理器的客户端证书/kubeconfig，用于和 API 服务器的会话调度器的客户端证书/kubeconfig，用于和 API 服务器的会话前端代理的客户端及服务端证书 说明： 只有当你运行 kube-proxy 并要支持扩展 API 服务器 时，才需要 front-proxy 证书 etcd 还实现了双向 TLS 来对客户端和对其他对等节点进行身份验证 PKI 证书和要求 | Kubernetes 三、证书工具准备 # 准备文件夹存放所有证书信息。看看kubeadm 如何组织有序的结构的# 三个节点都执行mkdir -p /etc/kubernetes/pki 1、下载证书工具 # 下载cfssl核心组件wget +x cfssl*#批量重命名for name in `ls cfssl*`; do mv $name ${name%_1.5.0_linux_amd64}; done#移动到文件mv cfssl* /usr/bin 2、ca根配置 ca-config.json mkdir -p /etc/kubernetes/pkicd /etc/kubernetes/pkivi ca-config.json { "signing": { "default": { "expiry": "87600h" }, "profiles": { "server": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth" ] }, "client": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "client auth" ] }, "peer": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] }, "kubernetes": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] }, "etcd": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } }} 3、ca签名请求 CSR是Certificate Signing Request的英文缩写，即证书签名请求文件 ca-csr.json vi /etc/kubernetes/pki/ca-csr.json { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "Kubernetes", "OU": "Kubernetes" } ], "ca": { "expiry": "87600h" }} CN(Common Name): 公用名（Common Name）必须填写，一般可以是网站域 O(Organization): Organization（组织名）是必须填写的，如果申请的是OV、EV型证书，组织名称必须严格和企业在政府登记名称一致，一般需要和营业执照上的名称完全一致。不可以使用缩写或者商标。如果需要使用英文名称，需要有DUNS编码或者律师信证明。 OU(Organization Unit) OU单位部门，这里一般没有太多限制，可以直接填写IT DEPT等皆可。 C(City) City是指申请单位所在的城市。 ST(State/Province) ST是指申请单位所在的省份。 C(Country Name） C是指国家名称，这里用的是两位大写的国家代码，中国是CN。 4、生成证书 生成ca证书和私钥 cfssl gencert -initca ca-csr.json | cfssljson -bare ca -# ca.csr ca.pem(ca公钥) ca-key.pem(ca私钥,妥善保管) 5、k8s集群是如何使用证书的 参考官方文档：PKI 证书和要求 | Kubernetes 四、etcd高可用搭建 1、etcd文档 etcd示例：Demo | etcd 参照示例学习etcd使用 etcd构建：Install | etcd 参照etcd-k8s集群量规划指南，大家参照这个标准建立集群 etcd部署：Operations guide | etcd 参照部署手册，学习etcd配置和集群部署 2、下载etcd # 给所有master节点，发送etcd包准备部署etcd高可用wget 复制到其他节点for i in k8s-master1 k8s-master2 k8s-master3;do scp etcd-* root@$i:/root/;done## 解压到 /usr/local/bintar -zxvf etcd-v3.4.16-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.4.16-linux-amd64/etcd{,ctl}##验证etcdctl #只要有打印就ok 3、etcd证书 Hardware recommendations | etcd安装参考 ：Hardware recommendations | etcd 生成etcd证书 etcd-ca-csr.json { "CN": "etcd", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "etcd", "OU": "etcd" } ], "ca": { "expiry": "87600h" }} # 生成etcd根ca证书cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/etcd/ca - etcd-itdachang-csr.json { "CN": "etcd-itdachang", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "k8s-master1", "k8s-master2", "k8s-master3", "192.168.0.10", "192.168.0.11", "192.168.0.12" ], "names": [ { "C": "CN", "L": "beijing", "O": "etcd", "ST": "beijing", "OU": "System" } ]}// 注意：hosts用自己的主机名和ip// 也可以在签发的时候再加上 -hostname=127.0.0.1,k8s-master1,k8s-master2,k8s-master3,// 可以指定受信的主机列表// "hosts": [// "k8s-master1",// "example.net"// ], # 签发itdachang的etcd证书cfssl gencert \ -ca=/etc/kubernetes/pki/etcd/ca.pem \ -ca-key=/etc/kubernetes/pki/etcd/ca-key.pem \ -config=/etc/kubernetes/pki/ca-config.json \ -profile=etcd \ etcd-itdachang-csr.json | cfssljson -bare /etc/kubernetes/pki/etcd/etcd # 把生成的etcd证书，复制给其他机器for i in k8s-master2 k8s-master3;do scp -r /etc/kubernetes/pki/etcd root@$i:/etc/kubernetes/pki;done 4、etcd高可用安装 etcd配置文件示例： Configuration flags | etcd etcd高可用安装示例： Clustering Guide | etcd 为了保证启动配置一致性，我们编写etcd配置文件，并将etcd做成service启动 # etcd yaml示例。# This is the configuration file for the etcd server.# Human-readable name for this member.name: 'default'# Path to the data directory.data-dir:# Path to the dedicated wal directory.wal-dir:# Number of committed transactions to trigger a snapshot to disk.snapshot-count: 10000# Time (in milliseconds) of a heartbeat interval.heartbeat-interval: 100# Time (in milliseconds) for an election to timeout.election-timeout: 1000# Raise alarms when backend size exceeds the given quota. 0 means use the# default quota.quota-backend-bytes: 0# List of comma separated URLs to listen on for peer traffic.listen-peer-urls: List of comma separated URLs to listen on for client traffic.listen-client-urls: Maximum number of snapshot files to retain (0 is unlimited).max-snapshots: 5# Maximum number of wal files to retain (0 is unlimited).max-wals: 5# Comma-separated white list of origins for CORS (cross-origin resource sharing).cors:# List of this member's peer URLs to advertise to the rest of the cluster.# The URLs needed to be a comma-separated list.initial-advertise-peer-urls: List of this member's client URLs to advertise to the public.# The URLs needed to be a comma-separated list.advertise-client-urls: Discovery URL used to bootstrap the cluster.discovery:# Valid values include 'exit', 'proxy'discovery-fallback: 'proxy'# HTTP proxy to use for traffic to discovery service.discovery-proxy:# DNS domain used to bootstrap initial cluster.discovery-srv:# Initial cluster configuration for bootstrapping.initial-cluster:# Initial cluster token for the etcd cluster during bootstrap.initial-cluster-token: 'etcd-cluster'# Initial cluster state ('new' or 'existing').initial-cluster-state: 'new'# Reject reconfiguration requests that would cause quorum loss.strict-reconfig-check: false# Accept etcd V2 client requestsenable-v2: true# Enable runtime profiling data via HTTP serverenable-pprof: true# Valid values include 'on', 'readonly', 'off'proxy: 'off'# Time (in milliseconds) an endpoint will be held in a failed state.proxy-failure-wait: 5000# Time (in milliseconds) of the endpoints refresh interval.proxy-refresh-interval: 30000# Time (in milliseconds) for a dial to timeout.proxy-dial-timeout: 1000# Time (in milliseconds) for a write to timeout.proxy-write-timeout: 5000# Time (in milliseconds) for a read to timeout.proxy-read-timeout: 0client-transport-security: # Path to the client server TLS cert file. cert-file: # Path to the client server TLS key file. key-file: # Enable client cert authentication. client-cert-auth: false # Path to the client server TLS trusted CA cert file. trusted-ca-file: # Client TLS using generated certificates auto-tls: falsepeer-transport-security: # Path to the peer server TLS cert file. cert-file: # Path to the peer server TLS key file. key-file: # Enable peer client cert authentication. client-cert-auth: false # Path to the peer server TLS trusted CA cert file. trusted-ca-file: # Peer TLS using generated certificates. auto-tls: false# Enable debug-level logging for etcd.debug: falselogger: zap# Specify 'stdout' or 'stderr' to skip journald logging even when running under systemd.log-outputs: [stderr]# Force to create a new one member cluster.force-new-cluster: falseauto-compaction-mode: periodicauto-compaction-retention: "1" 三个etcd机器都创建 /etc/etcd 目录，准备存储etcd配置信息 #三个master执行mkdir -p /etc/etcd vi /etc/etcd/etcd.yaml # 我们的yamlname: 'etcd-master3' #每个机器可以写自己的域名,不能重复data-dir: /var/lib/etcdwal-dir: /var/lib/etcd/walsnapshot-count: 5000heartbeat-interval: 100election-timeout: 1000quota-backend-bytes: 0listen-peer-urls: ' # 本机ip+2380端口，代表和集群通信listen-client-urls: '#改为自己的max-snapshots: 3max-wals: 5cors:initial-advertise-peer-urls: '#自己的ipadvertise-client-urls: ' #自己的ipdiscovery:discovery-fallback: 'proxy'discovery-proxy:discovery-srv:initial-cluster: 'etcd-master1=#这里不一样initial-cluster-token: 'etcd-k8s-cluster'initial-cluster-state: 'new'strict-reconfig-check: falseenable-v2: trueenable-pprof: trueproxy: 'off'proxy-failure-wait: 5000proxy-refresh-interval: 30000proxy-dial-timeout: 1000proxy-write-timeout: 5000proxy-read-timeout: 0client-transport-security: cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/ca.pem' auto-tls: truepeer-transport-security: cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' peer-client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/ca.pem' auto-tls: truedebug: falselog-package-levels:log-outputs: [default]force-new-cluster: false 三台机器的etcd做成service，开机启动 vi /usr/lib/systemd/system/etcd.service[Unit]Description=Etcd ServiceDocumentation=--config-file=/etc/etcd/etcd.yamlRestart=on-failureRestartSec=10LimitNOFILE=65536[Install]WantedBy=multi-user.targetAlias=etcd3.service # 加载&开机启动systemctl daemon-reloadsystemctl enable --now etcd# 启动有问题,使用 journalctl -u 服务名排查journalctl -u etcd 测试etcd访问 # 查看etcd集群状态etcdctl --endpoints="192.168.0.10:2379,192.168.0.11:2379,192.168.0.12:2379" --cacert=/etc/kubernetes/pki/etcd/ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem endpoint status --write-out=table# 以后测试命令export ETCDCTL_API=3HOST_1=192.168.0.10HOST_2=192.168.0.11HOST_3=192.168.0.12ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379## 导出环境变量，方便测试，参照ETCDCTL_DIAL_TIMEOUT=3sexport ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.pemexport ETCDCTL_CERT=/etc/kubernetes/pki/etcd/etcd.pemexport ETCDCTL_KEY=/etc/kubernetes/pki/etcd/etcd-key.pemexport ETCDCTL_ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379# 自动用环境变量定义的证书位置etcdctl member list --write-out=table#如果没有环境变量就需要如下方式调用etcdctl --endpoints=$ENDPOINTS --cacert=/etc/kubernetes/pki/etcd/ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem member list --write-out=table## 更多etcdctl命令，找到changelog对应版本 # 下载k8s包wget 把kubernetes把复制给master所有节点for i in k8s-master1 k8s-master2 k8s-master3 k8s-node1 k8s-node2 k8s-node3;do scp kubernetes-server-* root@$i:/root/;done #所有master节点解压kubelet，kubectl等到 /usr/local/bin。tar -xvf kubernetes-server-linux-amd64.tar.gz --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}#master需要全部组件，node节点只需要 /usr/local/bin kubelet、kube-proxy 3、apiserver 证书生成 3.1、apiserver-csr.json //10.96.0. 为service网段。可以自定义 如： 66.66.0.1// 192.168.0.250： 是我准备的负载均衡器地址（负载均衡可以自己搭建，也可以购买云厂商lb。）{ "CN": "kube-apiserver", "hosts": [ "10.96.0.1", "127.0.0.1", "192.168.0.250", "192.168.0.10", "192.168.0.11", "192.168.0.12", "192.168.0.13", "192.168.0.14", "192.168.0.15", "192.168.0.16", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "Kubernetes", "OU": "Kubernetes" } ]} 3.2、生成apiserver证书 # 192.168.0.是k8s service的网段，如果说需要更改k8s service网段，那就需要更改192.168.0.1，# 如果不是高可用集群，10.103.236.236为Master01的IP#先生成CA机构vi ca-csr.json{ "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "Kubernetes", "OU": "Kubernetes" } ], "ca": { "expiry": "87600h" }}cfssl gencert -initca ca-csr.json | cfssljson -bare ca -cfssl gencert -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem -config=/etc/kubernetes/pki/ca-config.json -profile=kubernetes apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver 4、front-proxy证书生成 官方文档：配置聚合层 | Kubernetes 注意：front-proxy不建议用新的CA机构签发证书，可能导致通过他代理的组件如metrics-server权限不可用。如果用新的，api-server配置添加 --requestheader-allowed-names=front-proxy-client 4.1、front-proxy-ca-csr.json front-proxy根ca vi front-proxy-ca-csr.json{ "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }} #front-proxy 根ca生成cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca 4.2、front-proxy-client证书 vi front-proxy-client-csr.json #准备申请client客户端 { "CN": "front-proxy-client", "key": { "algo": "rsa", "size": 2048 }} #生成front-proxy-client 证书cfssl gencert -ca=/etc/kubernetes/pki/front-proxy-ca.pem -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem -config=ca-config.json -profile=kubernetes front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client#忽略警告，毕竟我们不是给网站生成的 5、controller-manage证书生成与配置 5.1、controller-manager-csr.json vi controller-manager-csr.json { "CN": "system:kube-controller-manager", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "system:kube-controller-manager", "OU": "Kubernetes" } ]} 5.2、生成证书 cfssl gencert \ -ca=/etc/kubernetes/pki/ca.pem \ -ca-key=/etc/kubernetes/pki/ca-key.pem \ -config=ca-config.json \ -profile=kubernetes \ controller-manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager 5.3、生成配置 # 注意，如果不是高可用集群，192.168.0.250:6443改为master01的地址，6443为apiserver的默认端口# set-cluster：设置一个集群项，kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=\ --kubeconfig=/etc/kubernetes/controller-manager.conf# 设置一个环境项，一个上下文kubectl config set-context system:kube-controller-manager@kubernetes \ --cluster=kubernetes \ --user=system:kube-controller-manager \ --kubeconfig=/etc/kubernetes/controller-manager.conf# set-credentials 设置一个用户项kubectl config set-credentials system:kube-controller-manager \ --client-certificate=/etc/kubernetes/pki/controller-manager.pem \ --client-key=/etc/kubernetes/pki/controller-manager-key.pem \ --embed-certs=true \ --kubeconfig=/etc/kubernetes/controller-manager.conf# 使用某个环境当做默认环境kubectl config use-context system:kube-controller-manager@kubernetes \ --kubeconfig=/etc/kubernetes/controller-manager.conf # 后来也用来自动批复kubelet证书 6、scheduler证书生成与配置 6.1、scheduler-csr.json vi scheduler-csr.json { "CN": "system:kube-scheduler", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "system:kube-scheduler", "OU": "Kubernetes" } ]} 6.2、签发证书 cfssl gencert \ -ca=/etc/kubernetes/pki/ca.pem \ -ca-key=/etc/kubernetes/pki/ca-key.pem \ -config=/etc/kubernetes/pki/ca-config.json \ -profile=kubernetes \ scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler 6.3、生成配置 # 注意，如果不是高可用集群，192.168.0.250:6443 改为master01的地址，6443是api-server默认端口kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=\ --kubeconfig=/etc/kubernetes/scheduler.confkubectl config set-credentials system:kube-scheduler \ --client-certificate=/etc/kubernetes/pki/scheduler.pem \ --client-key=/etc/kubernetes/pki/scheduler-key.pem \ --embed-certs=true \ --kubeconfig=/etc/kubernetes/scheduler.confkubectl config set-context system:kube-scheduler@kubernetes \ --cluster=kubernetes \ --user=system:kube-scheduler \ --kubeconfig=/etc/kubernetes/scheduler.confkubectl config use-context system:kube-scheduler@kubernetes \ --kubeconfig=/etc/kubernetes/scheduler.conf#k8s集群安全操作相关 7、admin证书生成与配置 7.1、admin-csr.json vi admin-csr.json { "CN": "admin", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "system:masters", "OU": "Kubernetes" } ]} 7.2、生成证书 cfssl gencert \ -ca=/etc/kubernetes/pki/ca.pem \ -ca-key=/etc/kubernetes/pki/ca-key.pem \ -config=/etc/kubernetes/pki/ca-config.json \ -profile=kubernetes \ admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin 7.3、生成配置 # 注意，如果不是高可用集群，192.168.0.250:6443改为master01的地址，6443为apiserver的默认端口kubectl config set-cluster kubernetes \--certificate-authority=/etc/kubernetes/pki/ca.pem \--embed-certs=true \--server=\--kubeconfig=/etc/kubernetes/admin.confkubectl config set-credentials kubernetes-admin \--client-certificate=/etc/kubernetes/pki/admin.pem \--client-key=/etc/kubernetes/pki/admin-key.pem \--embed-certs=true \--kubeconfig=/etc/kubernetes/admin.confkubectl config set-context kubernetes-admin@kubernetes \--cluster=kubernetes \--user=kubernetes-admin \--kubeconfig=/etc/kubernetes/admin.confkubectl config use-context kubernetes-admin@kubernetes \--kubeconfig=/etc/kubernetes/admin.conf kubelet将使用 bootstrap 引导机制，自动颁发证书，所以我们不用配置了。要不然，1万台机器，一个万kubelet，证书配置到明年去。。。 8、ServiceAccount Key生成 k8s底层，每创建一个ServiceAccount，都会分配一个Secret，而Secret里面有秘钥，秘钥就是由我们接下来的sa生成的。所以我们提前创建出sa信息 openssl genrsa -out /etc/kubernetes/pki/sa.key 2048openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub 9、发送证书到其他节点 # 在master1上执行for NODE in k8s-master2 k8s-master3do for FILE in admin.conf controller-manager.conf scheduler.conf do scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE} donedone 六、高可用配置 高可用配置 如果你不是在创建高可用集群，则无需配置haproxy和keepalived高可用有很多可选方案 nginxhaproxykeepalived云供应商提供的负载均衡产品 云上安装注意事项 云上安装可以直接使用云上的lb，比如阿里云slb，腾讯云elb等公有云要用公有云自带的负载均衡，比如阿里云的SLB，腾讯云的ELB，用来替代haproxy和keepalived，因为公有云大部分都是不支持keepalived的。阿里云的话，kubectl控制端不能放在master节点，推荐使用腾讯云，因为阿里云的slb有回环的问题，也就是slb代理的服务器不能反向访问SLB，但是腾讯云修复了这个问题。 青云使用 创建负载均衡器，指定ip地址为我们之前的预留地址进入负载均衡器，创建监听器选择TCP，6443端口添加后端服务器地址与端口 七、组件启动 1、所有master执行 mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes#三个master节点kube-xx相关的程序都在 /usr/local/binfor NODE in k8s-master2 k8s-master3do scp -r /etc/kubernetes/* root@$NODE:/etc/kubernetes/done 接下来把master1生成的所有证书全部发给master2,master3 2、配置apiserver服务 2.1、配置 所有Master节点创建kube-apiserver.service 注意，如果不是高可用集群，192.168.0.250改为master01的地址以下文档使用的k8s service网段为10.96.0.0/16，该网段不能和宿主机的网段、Pod网段的重复特别注意：docker的网桥默认为 172.17.0.1/16。不要使用这个网段 # 每个master节点都需要执行以下内容# --advertise-address： 需要改为本master节点的ip# --service-cluster-ip-range=10.96.0.0/16： 需要改为自己规划的service网段# --etcd-servers： 改为自己etcd-server的所有地址vi /usr/lib/systemd/system/kube-apiserver.service[Unit]Description=Kubernetes API ServerDocumentation=\ --v=2 \ --logtostderr=true \ --allow-privileged=true \ --bind-address=0.0.0.0 \ --secure-port=6443 \ --insecure-port=0 \ --advertise-address=192.168.0.10 \ --service-cluster-ip-range=10.96.0.0/16 \ --service-node-port-range=30000-32767 \ --etcd-servers=\ --etcd-cafile=/etc/kubernetes/pki/etcd/ca.pem \ --etcd-certfile=/etc/kubernetes/pki/etcd/etcd.pem \ --etcd-keyfile=/etc/kubernetes/pki/etcd/etcd-key.pem \ --client-ca-file=/etc/kubernetes/pki/ca.pem \ --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \ --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \ --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \ --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/pki/sa.pub \ --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \ --service-account-issuer=\ --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \ --authorization-mode=Node,RBAC \ --enable-bootstrap-token-auth=true \ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \ --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \ --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \ --requestheader-allowed-names=aggregator,front-proxy-client \ --requestheader-group-headers=X-Remote-Group \ --requestheader-extra-headers-prefix=X-Remote-Extra- \ --requestheader-username-headers=X-Remote-User # --token-auth-file=/etc/kubernetes/token.csvRestart=on-failureRestartSec=10sLimitNOFILE=65535[Install]WantedBy=multi-user.target 2.2、启动apiserver服务 systemctl daemon-reload && systemctl enable --now kube-apiserver#查看状态systemctl status kube-apiserver 3、配置controller-manager服务 3.1、配置 所有Master节点配置kube-controller-manager.service 文档使用的k8s Pod网段为196.16.0.0/16，该网段不能和宿主机的网段、k8s Service网段的重复，请按需修改;特别注意：docker的网桥默认为 172.17.0.1/16。不要使用这个网 # 所有节点执行vi /usr/lib/systemd/system/kube-controller-manager.service## --cluster-cidr=196.16.0.0/16 ： 为Pod的网段。修改成自己想规划的网段[Unit]Description=Kubernetes Controller ManagerDocumentation=\ --v=2 \ --logtostderr=true \ --address=127.0.0.1 \ --root-ca-file=/etc/kubernetes/pki/ca.pem \ --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \ --service-account-private-key-file=/etc/kubernetes/pki/sa.key \ --kubeconfig=/etc/kubernetes/controller-manager.conf \ --leader-elect=true \ --use-service-account-credentials=true \ --node-monitor-grace-period=40s \ --node-monitor-period=5s \ --pod-eviction-timeout=2m0s \ --controllers=*,bootstrapsigner,tokencleaner \ --allocate-node-cidrs=true \ --cluster-cidr=196.16.0.0/16 \ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \ --node-cidr-mask-size=24 Restart=alwaysRestartSec=10s[Install]WantedBy=multi-user.target 3.2、启动 # 所有master节点执行systemctl daemon-reloadsystemctl daemon-reload && systemctl enable --now kube-controller-managersystemctl status kube-controller-manager 4、配置scheduler 4.1、配置 所有Master节点配置kube-scheduler.service vi /usr/lib/systemd/system/kube-scheduler.service [Unit]Description=Kubernetes SchedulerDocumentation=\ --v=2 \ --logtostderr=true \ --address=127.0.0.1 \ --leader-elect=true \ --kubeconfig=/etc/kubernetes/scheduler.confRestart=alwaysRestartSec=10s[Install]WantedBy=multi-user.target 4.2、启动 systemctl daemon-reloadsystemctl daemon-reload && systemctl enable --now kube-schedulersystemctl status kube-scheduler 八、TLS与引导启动原理 1、master1配置bootstrap 注意，如果不是高可用集群，192.168.0.250:6443改为master1的地址，6443为apiserver的默认端口 #准备一个随机token。但是我们只需要16个字符head -c 16 /dev/urandom | od -An -t x | tr -d ' '# 值如下： 737b177d9823531a433e368fcdb16f5f# 生成16个字符的head -c 8 /dev/urandom | od -An -t x | tr -d ' '# d683399b7a553977 #设置集群kubectl config set-cluster kubernetes \--certificate-authority=/etc/kubernetes/pki/ca.pem \--embed-certs=true \--server=\--kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf#设置秘钥kubectl config set-credentials tls-bootstrap-token-user \--token=l6fy8c.d683399b7a553977 \--kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf #设置上下文kubectl config set-context tls-bootstrap-token-user@kubernetes \--cluster=kubernetes \--user=tls-bootstrap-token-user \--kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf#使用设置kubectl config use-context tls-bootstrap-token-user@kubernetes \--kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf 2、master1设置kubectl执行权限 kubectl 能不能操作集群是看 /root/.kube 下有没有config文件，而config就是我们之前生成的admin.conf，具有操作权限的 # 只在master1生成，因为生产集群，我们只能让一台机器具有操作集群的权限，这样好控制mkdir -p /root/.kube ;cp /etc/kubernetes/admin.conf /root/.kube/config #验证kubectl get nodes# 应该在网络里面开放负载均衡器的6443端口;默认应该不要配置的[root@k8s-master1 ~]# kubectl get nodesNo resources found#说明已经可以连接apiserver并获取资源 3、创建集群引导权限文件 # master准备这个文件 vi /etc/kubernetes/bootstrap.secret.yamlapiVersion: v1kind: Secretmetadata: name: bootstrap-token-l6fy8c namespace: kube-systemtype: bootstrap.kubernetes.io/tokenstringData: description: "The default bootstrap token generated by 'kubelet '." token-id: l6fy8c token-secret: d683399b7a553977 usage-bootstrap-authentication: "true" usage-bootstrap-signing: "true" auth-extra-groups: system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress ---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: kubelet-bootstraproleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:node-bootstrappersubjects:- apiGroup: rbac.authorization.k8s.io kind: Group name: system:bootstrappers:default-node-token---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: node-autoapprove-bootstraproleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:nodeclientsubjects:- apiGroup: rbac.authorization.k8s.io kind: Group name: system:bootstrappers:default-node-token---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: node-autoapprove-certificate-rotationroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclientsubjects:- apiGroup: rbac.authorization.k8s.io kind: Group name: system:nodes---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:kube-apiserver-to-kubeletrules: - apiGroups: - "" resources: - nodes/proxy - nodes/stats - nodes/log - nodes/spec - nodes/metrics verbs: - "*"---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: system:kube-apiserver namespace: ""roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kube-apiserver-to-kubeletsubjects: - apiGroup: rbac.authorization.k8s.io kind: User name: kube-apiserver # 应用此文件资源内容kubectl create -f /etc/kubernetes/bootstrap.secret.yaml 九、引导Node节点启动 所有节点的kubelet需要我们引导启动 1、发送核心证书到节点 master1节点把核心证书发送到其他节点 cd /etc/kubernetes/ #查看所有信息#执行复制所有令牌操作for NODE in k8s-master2 k8s-master3 k8s-node1 k8s-node2; do ssh $NODE mkdir -p /etc/kubernetes/pki/etcd for FILE in ca.pem etcd.pem etcd-key.pem; do scp /etc/kubernetes/pki/etcd/$FILE $NODE:/etc/kubernetes/pki/etcd/ done for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.conf; do scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE} done done 2、所有节点配置kubelet # 所有节点创建相关目录mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/## 所有node节点必须有 kubelet kube-proxyfor NODE in k8s-master2 k8s-master3 k8s-node3 k8s-node1 k8s-node2; do scp -r /etc/kubernetes/* root@$NODE:/etc/kubernetes/ done 2.1、创建kubelet.service #所有节点，配置kubelet服务vi /usr/lib/systemd/system/kubelet.service[Unit]Description=Kubernetes KubeletDocumentation=所有节点配置kubelet service配置文件vi /etc/systemd/system/kubelet.service.d/10-kubelet.conf[Service]Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/pause:3.4.1"Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "ExecStart=ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS 2.2、创建kubelet-conf.yml文件 #所有节点，配置kubelet-conf文件vi /etc/kubernetes/kubelet-conf.yml# clusterDNS 为service网络的第10个ip值,改成自己的。如：10.96.0.10 apiVersion: kubelet.config.k8s.io/v1beta1kind: KubeletConfigurationaddress: 0.0.0.0port: 10250readOnlyPort: 10255authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.pemauthorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30scgroupDriver: systemdcgroupsPerQOS: trueclusterDNS:- 10.96.0.10clusterDomain: cluster.localcontainerLogMaxFiles: 5containerLogMaxSize: 10MicontentType: application/vnd.kubernetes.protobufcpuCFSQuota: truecpuManagerPolicy: nonecpuManagerReconcilePeriod: 10senableControllerAttachDetach: trueenableDebuggingHandlers: trueenforceNodeAllocatable:- podseventBurst: 10eventRecordQPS: 5evictionHard: imagefs.available: 15% memory.available: 100Mi nodefs.available: 10% nodefs.inodesFree: 5%evictionPressureTransitionPeriod: 5m0s #缩小相应的配置failSwapOn: truefileCheckFrequency: 20shairpinMode: promiscuous-bridgehealthzBindAddress: 127.0.0.1healthzPort: 1024820simageGCHighThresholdPercent: 85imageGCLowThresholdPercent: 80imageMinimumGCAge: 2m0siptablesDropBit: 15iptablesMasqueradeBit: 14kubeAPIBurst: 10kubeAPIQPS: 5makeIPTablesUtilChains: truemaxOpenFiles: 1000000maxPods: 110nodeStatusUpdateFrequency: 10soomScoreAdj: -999podPidsLimit: -1registryBurst: 10registryPullQPS: 5resolvConf: /etc/resolv.confrotateCertificates: trueruntimeRequestTimeout: 2m0sserializeImagePulls: truestaticPodPath: /etc/kubernetes/manifestsstreamingConnectionIdleTimeout: 4h0m0ssyncFrequency: 1m0svolumeStatsAggPeriod: 1m0s 2.3、所有节点启动kubelet systemctl daemon-reload && systemctl enable --now kubeletsystemctl status kubelet 会提示 "Unable to update cni config"。接下来配置cni网络即可 3、kube-proxy配置 注意，如果不是高可用集群，192.168.0.250:6443改为master1的地址，6443改为apiserver的默认端口 3.1、生成kube-proxy.conf 以下操作在master1执行 #创建kube-proxy的sakubectl -n kube-system create serviceaccount kube-proxy#创建角色绑定kubectl create clusterrolebinding system:kube-proxy \--clusterrole system:node-proxier \--serviceaccount kube-system:kube-proxy#导出变量，方便后面使用SECRET=$(kubectl -n kube-system get sa/kube-proxy --output=jsonpath='{.secrets[0].name}')JWT_TOKEN=$(kubectl -n kube-system get secret/$SECRET --output=jsonpath='{.data.token}' | base64 -d)PKI_DIR=/etc/kubernetes/pkiK8S_DIR=/etc/kubernetes# 生成kube-proxy配置# --server: 指定自己的apiserver地址或者lb地址kubectl config set-cluster kubernetes \--certificate-authority=/etc/kubernetes/pki/ca.pem \--embed-certs=true \--server=\--kubeconfig=${K8S_DIR}/kube-proxy.conf# kube-proxy秘钥设置kubectl config set-credentials kubernetes \--token=${JWT_TOKEN} \--kubeconfig=/etc/kubernetes/kube-proxy.confkubectl config set-context kubernetes \--cluster=kubernetes \--user=kubernetes \--kubeconfig=/etc/kubernetes/kube-proxy.confkubectl config use-context kubernetes \--kubeconfig=/etc/kubernetes/kube-proxy.conf #把生成的 kube-proxy.conf 传给每个节点for NODE in k8s-master2 k8s-master3 k8s-node1 k8s-node2 k8s-node3; do scp /etc/kubernetes/kube-proxy.conf $NODE:/etc/kubernetes/ done 3.2、配置kube-proxy.service # 所有节点配置 kube-proxy.service 服务，一会儿设置为开机启动vi /usr/lib/systemd/system/kube-proxy.service[Unit]Description=Kubernetes Kube ProxyDocumentation=\ --config=/etc/kubernetes/kube-proxy.yaml \ --v=2Restart=alwaysRestartSec=10s[Install]WantedBy=multi-user.target 3.3、准备kube-proxy.yaml 一定注意修改自己的Pod网段范围 # 所有机器执行vi /etc/kubernetes/kube-proxy.yaml apiVersion: kubeproxy.config.k8s.io/v1alpha1bindAddress: 0.0.0.0clientConnection: acceptContentTypes: "" burst: 10 contentType: application/vnd.kubernetes.protobuf kubeconfig: /etc/kubernetes/kube-proxy.conf #kube-proxy引导文件 qps: 5clusterCIDR: 196.16.0.0/16 #修改为自己的Pod-CIDRconfigSyncPeriod: 15m0sconntrack: max: null maxPerCore: 32768 min: 131072 tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0senableProfiling: falsehealthzBindAddress: 0.0.0.0:10256hostnameOverride: ""iptables: masqueradeAll: false masqueradeBit: 14 minSyncPeriod: 0s syncPeriod: 30sipvs: masqueradeAll: true minSyncPeriod: 5s scheduler: "rr" syncPeriod: 30skind: KubeProxyConfigurationmetricsBindAddress: 127.0.0.1:10249mode: "ipvs"nodePortAddresses: nulloomScoreAdj: -999portRange: ""udpIdleTimeout: 250ms 3.4、启动kube-proxy 所有节点启动 systemctl daemon-reload && systemctl enable --now kube-proxysystemctl status kube-proxy 十、部署calico 可以参照calico私有云部署指南 # 下载官网calicocurl -o calico.yaml## 把这个镜像修改成国内镜像# 修改一些我们自定义的. 修改etcd集群地址sed -i 's#etcd_endpoints: ""calico.yaml# etcd的证书内容，需要base64编码设置到yaml中ETCD_CA=`cat /etc/kubernetes/pki/etcd/ca.pem | base64 -w 0 `ETCD_CERT=`cat /etc/kubernetes/pki/etcd/etcd.pem | base64 -w 0 `ETCD_KEY=`cat /etc/kubernetes/pki/etcd/etcd-key.pem | base64 -w 0 `# 替换etcd中的证书base64编码后的内容sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico.yaml#打开 etcd_ca 等默认设置（calico启动后自己生成）。sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico.yaml# 修改自己的Pod网段 196.16.0.0/16POD_SUBNET="196.16.0.0/16"sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@# value: "192.168.0.0/16"@ value: '"${POD_SUBNET}"'@g' calico.yaml# 一定确定自己是否修改好了#确认calico是否修改好grep "CALICO_IPV4POOL_CIDR" calico.yaml -A 1 # 应用calico配置kubectl apply -f calico.yaml 十一、部署coreDNS git clone deployment/kubernetes#10.96.0.10 改为 service 网段的 第 10 个ip./deploy.sh -s -i 10.96.0.10 | kubectl apply -f - 十二、给机器打上role标签 kubectl label node k8s-master1 node-role.kubernetes.io/master=''kubectl label node k8s-master2 node-role.kubernetes.io/master=''kubectl label node k8s-master3 node-role.kubernetes.io/master=''kubectl taints node k8s-master1 十三、集群验证 验证Pod网络可访问性 同名称空间，不同名称空间可以使用 ip 互相访问跨机器部署的Pod也可以互相访问 验证Service网络可访问性 集群机器使用serviceIp可以负载均衡访问pod内部可以访问service域名 serviceName.namespacepod可以访问跨名称空间的service # 部署以下内容进行测试apiVersion: apps/v1kind: Deploymentmetadata: name: nginx-01 namespace: default labels: app: nginx-01spec: selector: matchLabels: app: nginx-01 replicas: 1 template: metadata: labels: app: nginx-01 spec: containers: - name: nginx-01 image: nginx---apiVersion: v1kind: Servicemetadata: name: nginx-svc namespace: defaultspec: selector: app: nginx-01 type: ClusterIP ports: - name: nginx-svc port: 80 targetPort: 80 protocol: TCP---apiVersion: v1kind: Namespacemetadata: name: hellospec: {}---apiVersion: apps/v1kind: Deploymentmetadata: name: nginx-hello namespace: hello labels: app: nginx-hellospec: selector: matchLabels: app: nginx-hello replicas: 1 template: metadata: labels: app: nginx-hello spec: containers: - name: nginx-hello image: nginx---apiVersion: v1kind: Servicemetadata: name: nginx-svc-hello namespace: hellospec: selector: app: nginx-hello type: ClusterIP ports: - name: nginx-svc-hello port: 80 targetPort: 80 protocol: TCP # 给两个master标识为workerkubectl label node k8s-node3 node-role.kubernetes.io/worker=''kubectl label node k8s-master3 node-role.kubernetes.io/worker=''kubectl label node k8s-node1 node-role.kubernetes.io/worker=''kubectl label node k8s-node2 node-role.kubernetes.io/worker=''# 给master1打上污点。二进制部署的集群，默认master是没有污点的，可以任意调度。我们最好给一个master打上污点，保证master最小可用kubectl label node k8s-master3 node-role.kubernetes.io/master=''kubectl taint nodes k8s-master1 node-role.kubernetes.io/master=:NoSchedule
版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。
相关文章
发表评论
暂时没有评论，来抢沙发吧~