Kubernetes技术详解-从理论到实践-(13)-集群-Cluster
·
1 Cluster集群简介
集群可以理解为多台机器通过网络组成“一台更大、更可靠、更易扩展的计算机”，对外提供统一服务。Kubernetes集群把服务器抽象成“资源池”，统一运行容器，是一种非常重要的云原生技术。
我们在环境搭建一章，已经介绍了K3s的三种工作模式：
- 单控制平面模式
- 高可用控制平面模式
- 混合模式
本章将详细介绍单控制平面模式和高可用控制平面模式。
在Kubernetes集群中，每台机器的角色可以是控制平面节点(Control Panel Node)，也可以是工作节点(Work Node)。
如果有一台控制平面节点，一台或多台工作节点，那这个集群就工作在单控制平面模式。
如果有大于一台控制平面节点，一台或多台工作节点，那这个集群就工作在高可用控制平面模式。
如果只有一台机器，这台机器既是控制平面节点，又是工作节点，那这个集群就工作在混合模式。
我们前面讲的章节都是就这个混合模式，接下来我们将介绍单控制平面模式和高可用控制平面模式。
注：控制平面节点和工作节点的概念是K8s官方文档推荐的，在工作中，我们也称第一个控制平面节点为master、其他控制平面节点为slave1、slave2…，工作节点为worker1、worker2…。
2 单控制平面模式
2.1 示例：单控制平面模式
准备三台虚拟机，hostname分别为master worker1 work2，其中master安装成控制平面节点， worker1和worker2节点分别安装成工作节点。单控制平面模式下，也可以只有master和worker1，没有worker2。install-k3s.sh文件K3s可执行文件、K3s所需镜像文件在本文配套资料包的k8s-test/1-setup/install-k3s/目录。
配套资料包地址：https://download.csdn.net/download/field1003/91965690
以下是示例步骤：
(1) master虚拟机安装和查看主节点
【本步骤在master虚拟机执行】
# 安装主节点
[root@master install-k3s]# ./install-k3s.sh master
Loaded image: rancher/klipper-helm:v0.8.4-build20240523
Loaded image: rancher/klipper-lb:v0.4.7
Loaded image: rancher/local-path-provisioner:v0.0.27
Loaded image: rancher/mirrored-coredns-coredns:1.10.1
Loaded image: rancher/mirrored-library-busybox:1.36.1
Loaded image: rancher/mirrored-library-traefik:2.10.7
Loaded image: rancher/mirrored-metrics-server:v0.7.0
Loaded image: rancher/mirrored-pause:3.6
setup_env: server --docker --disable traefik --disable metrics-server --service-node-port-range=0-39999 --cluster-cidr 10.42.0.0/16 --service -cidr 10.43.0.0/16 master
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
token: 3ba0d4e7c478ec4ff9fc1ae0320ee235
# 查看主节点状态
[root@master install-k3s]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 2m24s v1.29.6+k3s1
(2) worker1虚拟机安装工作节点
【本步骤在worker1虚拟机执行】
安装worker1工作节点。
worker1为节点名，192.168.88.132为控制平面节点IP地址，最后一个参数为控制平面节点的token，在安装控制平面节点成功后，最后一行输出就是这个token。
[root@worker1 install-k3s]# ./install-k3s.sh worker1 192.168.88.132 3ba0d4e7c478ec4ff9fc1ae0320ee235
Loaded image: rancher/klipper-helm:v0.8.4-build20240523
Loaded image: rancher/klipper-lb:v0.4.7
Loaded image: rancher/local-path-provisioner:v0.0.27
Loaded image: rancher/mirrored-coredns-coredns:1.10.1
Loaded image: rancher/mirrored-library-busybox:1.36.1
Loaded image: rancher/mirrored-library-traefik:2.10.7
Loaded image: rancher/mirrored-metrics-server:v0.7.0
Loaded image: rancher/mirrored-pause:3.6
setup_env: agent --docker worker1 192.168.88.132 3ba0d4e7c478ec4ff9fc1ae0320ee235
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO] systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO] systemd: Starting k3s-agent
(3) worker2虚拟机安装工作节点
【本步骤在worker2虚拟机执行】
# 安装worker2工作节点。
[root@worker2 install-k3s]# ./install-k3s.sh worker2 192.168.88.132 3ba0d4e7c478ec4ff9fc1ae0320ee235
Loaded image: rancher/klipper-helm:v0.8.4-build20240523
Loaded image: rancher/klipper-lb:v0.4.7
Loaded image: rancher/local-path-provisioner:v0.0.27
Loaded image: rancher/mirrored-coredns-coredns:1.10.1
Loaded image: rancher/mirrored-library-busybox:1.36.1
Loaded image: rancher/mirrored-library-traefik:2.10.7
Loaded image: rancher/mirrored-metrics-server:v0.7.0
Loaded image: rancher/mirrored-pause:3.6
setup_env: agent --docker worker2 192.168.88.132 3ba0d4e7c478ec4ff9fc1ae0320ee235
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO] systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO] systemd: Starting k3s-agent
(4) 查看集群节点状态
【本步骤可以在任意节点(master、worker1、worker2)执行】
# 查看节点状态
[root@master 13-cluster]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 3m23s v1.29.6+k3s1
worker1 Ready <none> 90s v1.29.6+k3s1
worker2 Ready <none> 14s v1.29.6+k3s1
(5) 部署Deployment
【本步骤在master节点执行】
# 编写deployment资源清单文件
[root@master 13-cluster]# cat nginx-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deploy
labels:
app: nginx-dm
spec:
selector:
matchLabels:
app: nginx-dm
replicas: 3
template:
metadata:
labels:
app: nginx-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
tolerations:
- key: "node-type"
operator: "Equal"
value: "dev"
effect: "NoExecute"
# 创建资源对象
[root@master 13-cluster]# kubectl apply -f nginx-deploy.yaml
deployment.apps/nginx-deploy created
# 查看Pod资源对象，通过Deployment创建了三个Pod。可以看到NODE字段，Pod运行在三个不同的工作节点上(k3s master那台设备为混合模式，既运行控制平面节点又运行工作节点)。
[root@master 13-cluster]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-68dccd9bb6-fkcgg 1/1 Running 0 5s 10.42.1.2 worker1 <none> <none>
nginx-deploy-68dccd9bb6-fmjd2 1/1 Running 0 5s 10.42.2.2 worker2 <none> <none>
nginx-deploy-68dccd9bb6-w5jzb 1/1 Running 0 5s 10.42.0.4 master <none> <none>
(6) 模拟工作节点故障
【本步骤在worker1虚拟机执行】
# 使用nmcli命令down掉网络，模拟节点下线。
[root@worker1 install-k3s]for conn in $(nmcli -t -f NAME connection show --active); do nmcli connection down "$conn"; done
# 如需恢复网络连接，执行nmcli connection up <连接名> 或systemctl restart NetworkManager命令重启NetworkManager服务即可
# 执行完此命令后，ssh连接将被断开。
[root@worker1 install-k3s]# for conn in $(nmcli -t -f NAME connection show --active); do nmcli connection down "$conn"; done
(7) 查看集群状态
【本步骤在master虚拟机执行】
当有节点故障后，K8s将自动检测故障节点 -> 标记NotReady -> 超时驱逐Pod -> 重新调度到健康节点 -> 重建副本并重新挂载卷，整个过程无需人工介入，这就是K8s的节点生命周期管理(Node Lifecycle Management)机制。
K8s自动检测故障节点，与以下三个参数有关：node-monitor-period：控制管理器检查节点状态的间隔，默认值为5秒node-monitor-grace-period：节点无响应后标记为不可用的宽限期，默认值为40秒node-status-update-frequency：kubelet向API Server上报节点状态的频率，默认值为10秒
K8s超时驱逐Pod，以下参数有关：pod-eviction-timeout：节点被认为是不可用后，开始驱逐Pod的等待时间，默认值为5分钟default-not-ready-toleration-seconds和default-unreachable-toleration-seconds：这两个参数控制Pod在节点不可用时的容忍时间，默认都是300秒
# 查看节点状态，此时worker1节点仍处于Ready状态，因为尚未到节点状态检测时间。
[root@master 13-cluster]# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master Ready control-plane,master 5m2s v1.29.6+k3s1 192.168.88.132 <none> openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
worker1 Ready <none> 3m9s v1.29.6+k3s1 192.168.88.134 <none> openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
worker2 Ready <none> 113s v1.29.6+k3s1 192.168.88.135 <none> openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
# 查看pod状态，pod尚未被驱逐。
[root@master 13-cluster]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-68dccd9bb6-fkcgg 1/1 Running 0 41s 10.42.1.2 worker1 <none> <none>
nginx-deploy-68dccd9bb6-fmjd2 1/1 Running 0 41s 10.42.2.2 worker2 <none> <none>
nginx-deploy-68dccd9bb6-w5jzb 1/1 Running 0 41s 10.42.0.4 master <none> <none>
# 过一段时间再查看
# 故障节点自动检测：故障节点已经处于NotReady状态
[root@master 13-cluster]# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master Ready control-plane,master 5m52s v1.29.6+k3s1 192.168.88.132 <none> openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
worker1 NotReady <none> 3m59s v1.29.6+k3s1 192.168.88.134 <none> openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
worker2 Ready <none> 2m43s v1.29.6+k3s1 192.168.88.135 <none> openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
# 超时驱逐Pod：pod已经处于Terminating状态。当worker1再次加入到集群时，该pod将会被删除。因为deploy要求三个pod副本，因此k3s创建一个新的pod，并调度到worker2上。
[root@master 13-cluster]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-68dccd9bb6-fkcgg 1/1 Terminating 0 8m26s 10.42.1.2 worker1 <none> <none>
nginx-deploy-68dccd9bb6-fmjd2 1/1 Running 0 8m26s 10.42.2.2 worker2 <none> <none>
nginx-deploy-68dccd9bb6-vzrfz 1/1 Running 0 116s 10.42.2.3 worker2 <none> <none>
nginx-deploy-68dccd9bb6-w5jzb 1/1 Running 0 8m26s 10.42.0.4 master <none> <none>
(8) 节点卸载
控制平面节点卸载
控制平面卸载，集群不能再提供服务，工作节点也不再工作。
【本卸载步骤在master节点执行】[root@master install-k3s]# /usr/local/bin/k3s-uninstall.sh
工作节点卸载
工作节点卸载，并不会影响集群工作，故障节点上的Pod将被转移到正常节点上。
【本卸载步骤在worker节点执行】[root@worker1 install-k3s]# /usr/local/bin/k3s-agent-uninstall.sh
3 高可用控制平面模式
在高可用控制平面模式下，至少需要两个控制平面节点，但要达到真正的高可用，至少需要三个控制平面节点。
3.1 示例：两个控制平面节点的高可用模式
下面先看两个控制平面节点情况。
准备两台虚拟机，一个hostname为master，一个hostname为slave，我们计划master为第一个安装的控制平面节点，slave加入master，组成双控制平面节点的集群。由于master和slave已经各自带了一个工作节点，所以这里就不再额外准备工作节点。
(1) 初始化第一个控制平面节点
【本步骤在master虚拟机执行】
# 初始化第一个控制平面节点，必须带--cluster-int参数，第一个IP为本节点的IP地址，后面的134为计划加入本集群的节点IP地址。如果后面有多个控制平面节点加入本集群，可以写到134后面。
[root@master install-k3s]# ./install-k3s.sh master --cluster-init 192.168.88.132 192.168.88.134
Loaded image: rancher/klipper-helm:v0.8.4-build20240523
Loaded image: rancher/klipper-lb:v0.4.7
Loaded image: rancher/local-path-provisioner:v0.0.27
Loaded image: rancher/mirrored-coredns-coredns:1.10.1
Loaded image: rancher/mirrored-library-busybox:1.36.1
Loaded image: rancher/mirrored-library-traefik:2.10.7
Loaded image: rancher/mirrored-metrics-server:v0.7.0
Loaded image: rancher/mirrored-pause:3.6
setup_env: server --docker --disable traefik --disable metrics-server --service-node-port-range=0-39999 --cluster-cidr 10.42.0.0/16 --service-cidr 10.43.0.0/16 --node-ip 192.168.88.132 --node-external-ip 192.168.88.132 --tls-san 192.168.88.132 --tls-san 192.168.88.134 master --cluster-init 192.168.88.132 192.168.88.134
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
token: K101088a583fc6673bb402f7ca84cc69ce99880c743560cae7e2c665dff6d7a439e::server:5be9ce7bd5556aefb00df658b6c6add6
# 安装成功，前面最后一行为node-token，后续slave节点加入本节点时会用到。
[root@master install-k3s]# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master Ready control-plane,etcd,master 20s v1.29.6+k3s1 192.168.88.132 192.168.88.132 openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
[root@master install-k3s]# kubectl get pod -o wide
No resources found in default namespace.
(2) 将第二个控制平面节点加入集群
【本步骤在slave虚拟机执行】
# slave为待加入集群的节点的名称，192.168.88.132为刚才创建成功的控制平面节点IP，后面为token，--node-ip为本虚拟机的IP地址，--tls-tan和证书相关，第一个节点初始化时有几个IP，这里就需要几个tls-tan，保持一致才能安装成功。
[root@slave install-k3s]# ./install-k3s.sh slave 192.168.88.132 K101088a583fc6673bb402f7ca84cc69ce99880c743560cae7e2c665dff6d7a439e::server:5be9ce7bd5556aefb00df658b6c6add6 --node-ip 192.168.88.134 --tls-san 192.168.88.132 --tls-san 192.168.88.134
Loaded image: rancher/klipper-helm:v0.8.4-build20240523
Loaded image: rancher/klipper-lb:v0.4.7
Loaded image: rancher/local-path-provisioner:v0.0.27
Loaded image: rancher/mirrored-coredns-coredns:1.10.1
Loaded image: rancher/mirrored-library-busybox:1.36.1
Loaded image: rancher/mirrored-library-traefik:2.10.7
Loaded image: rancher/mirrored-metrics-server:v0.7.0
Loaded image: rancher/mirrored-pause:3.6
setup_env flags: --docker slave 192.168.88.132 K101088a583fc6673bb402f7ca84cc69ce99880c743560cae7e2c665dff6d7a439e::server:5be9ce7bd5556aefb00df658b6c6add6 --node-ip 192.168.88.134 --tls-san 192.168.88.132 --tls-san 192.168.88.134
setup_env token for server
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
(3) slave节点查看集群
【本步骤在slave虚拟机执行】
# 安装完成，在本机可以看到有两个节点。
[root@slave install-k3s]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,etcd,master 87s v1.29.6+k3s1
slave Ready control-plane,etcd,master 8s v1.29.6+k3s1
# 参数-o wide为更详细的节点信息，132为之前创建的节点，134为刚加入的节点。
[root@slave install-k3s]# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master Ready control-plane,etcd,master 90s v1.29.6+k3s1 192.168.88.132 192.168.88.132 openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
slave Ready control-plane,etcd,master 11s v1.29.6+k3s1 192.168.88.134 <none> openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
(4) master节点查看集群
【本步骤在master虚拟机执行】
可以看到master上也可以
[root@master install-k3s]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,etcd,master 97s v1.29.6+k3s1
slave Ready control-plane,etcd,master 18s v1.29.6+k3s1
(5) 在集群内创建Deployment资源
【本步骤可以在master或slave虚拟机执行】
注：因为master和slave两个节点都是混合节点，都具有控制平面和工作节点功能，所以节点将会调度到这两台设备上。正常情况下，控制平面节点和工作节点功能分开，Pod不会调度到控制平面节点上。
# 准备创建deploy资源，副本数为3。由于这两个节点可以运行pod，因此预期deploy的三个副本会被调度到这两台虚拟机上。
# 编写Deployment资源清单文件
[root@master 13-cluster]# cat nginx-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deploy
labels:
app: nginx-dm
spec:
selector:
matchLabels:
app: nginx-dm
replicas: 3
template:
metadata:
labels:
app: nginx-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
tolerations:
- key: "node-type"
operator: "Equal"
value: "dev"
effect: "NoExecute"
# 创建Deployment
[root@master 13-cluster]# kubectl apply -f nginx-deploy.yaml
deployment.apps/nginx-deploy created
# 查看Deployment。可以看到这三个pod有1个调度到master，两个调度到slave。
[root@master 13-cluster]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-68dccd9bb6-h56gh 1/1 Running 0 7s 10.42.0.6 master <none> <none>
nginx-deploy-68dccd9bb6-qsx2s 1/1 Running 0 7s 10.42.1.7 slave <none> <none>
nginx-deploy-68dccd9bb6-swvmg 1/1 Running 0 7s 10.42.1.8 slave <none> <none>
(6) slave节点查看集群资源对象
【本步骤在slave虚拟机执行】
# 查看pod，可以看到和master上相同的结果。
[root@slave install-k3s]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-68dccd9bb6-h56gh 1/1 Running 0 17s 10.42.0.6 master <none> <none>
nginx-deploy-68dccd9bb6-qsx2s 1/1 Running 0 17s 10.42.1.7 slave <none> <none>
nginx-deploy-68dccd9bb6-swvmg 1/1 Running 0 17s 10.42.1.8 slave <none> <none>
从上面的示例可以看到，这两台虚拟机上的控制平面节点都可以正常工作，那是不是意味着有一台出现下线或出现故障后，另一台可以继续运行呢？
我们模拟slave虚拟机故障，然后查看master虚拟机是否还可以正常工作。
切换到slave虚拟机，使用前面介绍的nmcli命令，断开slave与master的连接，模拟slave节点故障。
执行如下语句，执行完之后，网络down掉，ssh远程连接也会断掉，如果要恢复，需要进入虚拟机里面执行nmcli connection up <连接名>或systemctl restart NetworkManager。[root@slave2 install-k3s]# for conn in $(nmcli -t -f NAME connection show --active); do nmcli connection down "$conn"; done
上述操作完成后，切换到master虚拟机，执行kubectl查看node情况
[root@master 13-cluster]# kubectl get node
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
可以看到，当slave虚拟机下线后，原来的master虚拟机也无法提供服务。其原因为：
我们这两个节点上都部署了etcd服务，etcd为防止分布式系统的脑裂现象，采用了Raft一致性算法，该算法要求多数派存活，即quorum=所有节点/2 + 1，因此，集群节点数至少为3，才能实现高可用。
即使在有三个节点情况下，最多也只能有一个节点故障，另外两台才能正常工作。如果集群节点数为5，那么quorum=5/2+1即3，那么即使有两个节点出现故障，k8s仍可提供服务。
想继续深入了解分布式一致性算法，可以搜索拜占庭将军问题。
3.2 示例：三个控制平面节点的高可用模式
下面看三个控制平面节点情况。
准备三台虚拟机，一个hostname为master，一个hostname为slave，一个hostname为slave2，我们计划master为第一个安装的控制平面节点，slave和slave2加入master，组成多控制平面节点的集群，演示高可用特性。
以下是操作步骤：
(1) 环境清理
由于上个例子，master在初始化时，只规划了192.168.88.132 192.168.88.134两个节点，这时如果有新的节点加入，将会报tls错误，我们这里先卸载master和slave上的k3s。
【以下命令在master虚拟机执行】
# 执行k3s-uninstall.sh，卸载控制平面节点
[root@master install-k3s]# /usr/local/bin/k3s-uninstall.sh
【以下命令在slave虚拟机执行】
# 执行k3s-uninstall.sh，卸载控制平面节点
[root@slave install-k3s]# /usr/local/bin/k3s-uninstall.sh
(2) 初始化第一个控制平面节点
【本步骤在master虚拟机执行】
# 初始化第一个节点。本次计划由三台虚拟机组成集群，因此初始化时把这三个IP地址都带上。
[root@master install-k3s]# ./install-k3s.sh master1 --cluster-init 192.168.88.132 192.168.88.134 192.168.88.135
Loaded image: rancher/klipper-helm:v0.8.4-build20240523
Loaded image: rancher/klipper-lb:v0.4.7
Loaded image: rancher/local-path-provisioner:v0.0.27
Loaded image: rancher/mirrored-coredns-coredns:1.10.1
Loaded image: rancher/mirrored-library-busybox:1.36.1
Loaded image: rancher/mirrored-library-traefik:2.10.7
Loaded image: rancher/mirrored-metrics-server:v0.7.0
Loaded image: rancher/mirrored-pause:3.6
setup_env: server --docker --disable traefik --disable metrics-server --service-node-port-range=0-39999 --cluster-cidr 10.42.0.0/16 --service-cidr 10.43.0.0/16 --node-ip 192.168.88.132 --node-external-ip 192.168.88.132 --tls-san 192.168.88.132 --tls-san 192.168.88.134 --tls-san 192.168.88.135 master1 --cluster-init 192.168.88.132 192.168.88.134 192.168.88.135
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
token: K105a6e82650a29c8d4dbe801c1bfbf7301b91c44cb20b0e28b47a01b023ed709b5::server:aa4248001eff6684eb47ac248013928d
# 安装成功，可以查看到node信息
[root@master install-k3s]# kubectl get node
NAME STATUS ROLES AGE VERSION
master1 Ready control-plane,etcd,master 6s v1.29.6+k3s1
[root@master install-k3s]# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master1 Ready control-plane,etcd,master 10s v1.29.6+k3s1 192.168.88.132 192.168.88.132 openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
(3) 将slave控制平面节点加入集群
【本步骤在slave虚拟机执行】
# 在slave上安装k3s
[root@slave install-k3s]# ./install-k3s.sh slave 192.168.88.132 K105a6e82650a29c8d4dbe801c1bfbf7301b91c44cb20b0e28b47a01b023ed709b5::server:aa4248001eff6684eb47ac248013928d --node-ip 192.168.88.134 --tls-san 192.168.88.132 --tls-san 192.168.88.134 --tls-san 192.168.88.135
Loaded image: rancher/klipper-helm:v0.8.4-build20240523
Loaded image: rancher/klipper-lb:v0.4.7
Loaded image: rancher/local-path-provisioner:v0.0.27
Loaded image: rancher/mirrored-coredns-coredns:1.10.1
Loaded image: rancher/mirrored-library-busybox:1.36.1
Loaded image: rancher/mirrored-library-traefik:2.10.7
Loaded image: rancher/mirrored-metrics-server:v0.7.0
Loaded image: rancher/mirrored-pause:3.6
setup_env flags: --docker slave 192.168.88.132 K105a6e82650a29c8d4dbe801c1bfbf7301b91c44cb20b0e28b47a01b023ed709b5::server:aa4248001eff6684eb47ac248013928d --node-ip 192.168.88.134 --tls-san 192.168.88.132 --tls-san 192.168.88.134 --tls-san 192.168.88.135
setup_env token for server
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
# 安装成功，查看node，有两个
[root@slave install-k3s]# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master1 Ready control-plane,etcd,master 70s v1.29.6+k3s1 192.168.88.132 192.168.88.132 openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
slave Ready control-plane,etcd,master 9s v1.29.6+k3s1 192.168.88.134 <none> openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
(4) 将slave2控制平面节点加入集群
【本步骤在slave2虚拟机执行】
# 和slave加入节点时有两处不同：第一个参数为node-name，节点名，不能和之前相同，这里为slave2；--node-ip参数也不同，这里需要是slave2虚拟机的IP地址
[root@slave2 install-k3s]# ./install-k3s.sh slave2 192.168.88.132 K105a6e82650a29c8d4dbe801c1bfbf7301b91c44cb20b0e28b47a01b023ed709b5::server:aa4248001eff6684eb47ac248013928d --node-ip 192.168.88.135 --tls-san 192.168.88.132 --tls-san 192.168.88.134 --tls-san 192.168.88.135
Loaded image: rancher/klipper-helm:v0.8.4-build20240523
Loaded image: rancher/klipper-lb:v0.4.7
Loaded image: rancher/local-path-provisioner:v0.0.27
Loaded image: rancher/mirrored-coredns-coredns:1.10.1
Loaded image: rancher/mirrored-library-busybox:1.36.1
Loaded image: rancher/mirrored-library-traefik:2.10.7
Loaded image: rancher/mirrored-metrics-server:v0.7.0
Loaded image: rancher/mirrored-pause:3.6
setup_env flags: --docker slave2 192.168.88.132 K105a6e82650a29c8d4dbe801c1bfbf7301b91c44cb20b0e28b47a01b023ed709b5::server:aa4248001eff6684eb47ac248013928d --node-ip 192.168.88.135 --tls-san 192.168.88.132 --tls-san 192.168.88.134 --tls-san 192.168.88.135
setup_env token for server
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
# 可以看到三个节点都已经就绪，IP地址分别为132 134 135
[root@slave2 install-k3s]# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master1 Ready control-plane,etcd,master 2m6s v1.29.6+k3s1 192.168.88.132 192.168.88.132 openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
slave Ready control-plane,etcd,master 65s v1.29.6+k3s1 192.168.88.134 <none> openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
slave2 Ready control-plane,etcd,master 8s v1.29.6+k3s1 192.168.88.135 <none> openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
(5) 在其他节点查看集群节点情况
# 【本命令在slave虚拟机执行】
# slave可以查看到三个节点信息
[root@slave install-k3s]# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master1 Ready control-plane,etcd,master 2m17s v1.29.6+k3s1 192.168.88.132 192.168.88.132 openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
slave Ready control-plane,etcd,master 76s v1.29.6+k3s1 192.168.88.134 <none> openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
slave2 Ready control-plane,etcd,master 19s v1.29.6+k3s1 192.168.88.135 <none> openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
# 【本命令在master虚拟机执行】
# 可以查看到相同的信息，集群内有三个控制平面节点
[root@master install-k3s]# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master1 Ready control-plane,etcd,master 3m23s v1.29.6+k3s1 192.168.88.132 192.168.88.132 openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
slave Ready control-plane,etcd,master 2m22s v1.29.6+k3s1 192.168.88.134 <none> openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
slave2 Ready control-plane,etcd,master 85s v1.29.6+k3s1 192.168.88.135 <none> openEuler 22.03 LTS 5.10.0-60.18.0.50.oe2203.x86_64 docker://20.10.0
(6) 创建资源对象
【本步骤可在任意虚拟机上执行】
# 准备创建包含三个副本的deploy资源，配置文件如下：
[root@master 13-cluster]# cat nginx-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deploy
labels:
app: nginx-dm
spec:
selector:
matchLabels:
app: nginx-dm
replicas: 3
template:
metadata:
labels:
app: nginx-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
tolerations:
- key: "node-type"
operator: "Equal"
value: "dev"
effect: "NoExecute"
# 创建Deployment
[root@master 13-cluster]# kubectl apply -f nginx-deploy.yaml
deployment.apps/nginx-deploy created
# 可以看到，三个Pod分别被调度到了三个节点上。
[root@master 6-service]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-68dccd9bb6-kxlww 1/1 Running 0 7s 10.42.1.6 slave <none> <none>
nginx-deploy-68dccd9bb6-llqt2 1/1 Running 0 7s 10.42.2.3 slave2 <none> <none>
nginx-deploy-68dccd9bb6-wblpt 1/1 Running 0 7s 10.42.0.6 master1 <none> <none>
(7) 模拟其中一个控制平面节点故障
【本步骤在slave虚拟机执行】
这里我们模拟slave节点故障，使用nmcli使其下线，查看集群是否还可以继续提供服务。
执行如下命令之后，slave虚拟机将断开与master、slave2的连接。[root@slave install-k3s]# for conn in $(nmcli -t -f NAME connection show --active); do nmcli connection down "$conn"; done
(8) master节点查看集群信息
【本步骤在master虚拟机执行】
# 可以看到master仍可以提供服务，因为quorum为1，目前集群仍有2个节点，符合要求。此时因为尚未到节点状态检测时间，slave仍处于Ready状态。
[root@master 6-service]# kubectl get node
NAME STATUS ROLES AGE VERSION
master1 Ready control-plane,etcd,master 4m55s v1.29.6+k3s1
slave Ready control-plane,etcd,master 3m54s v1.29.6+k3s1
slave2 Ready control-plane,etcd,master 2m57s v1.29.6+k3s1
# Pod此时状态尚未改变。
[root@master 13-cluster]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-68dccd9bb6-kxlww 1/1 Running 0 98s 10.42.1.6 slave <none> <none>
nginx-deploy-68dccd9bb6-llqt2 1/1 Running 0 98s 10.42.2.3 slave2 <none> <none>
nginx-deploy-68dccd9bb6-wblpt 1/1 Running 0 98s 10.42.0.6 master1 <none> <none>
# 过大约1分钟后，slave节点处于NotReady状态
[root@master 13-cluster]# kubectl get node
NAME STATUS ROLES AGE VERSION
master1 Ready control-plane,etcd,master 5m29s v1.29.6+k3s1
slave NotReady control-plane,etcd,master 4m28s v1.29.6+k3s1
slave2 Ready control-plane,etcd,master 3m31s v1.29.6+k3s1
# 过大约5分钟后，Pod处于Terminating状态，新创建的Pod被调度到slave2上。当slave上线后，原Pod将被删除。
[root@master 13-cluster]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-68dccd9bb6-g8gtl 1/1 Running 0 20s 10.42.2.4 slave2 <none> <none>
nginx-deploy-68dccd9bb6-kxlww 1/1 Terminating 0 6m48s 10.42.1.6 slave <none> <none>
nginx-deploy-68dccd9bb6-llqt2 1/1 Running 0 6m48s 10.42.2.3 slave2 <none> <none>
nginx-deploy-68dccd9bb6-wblpt 1/1 Running 0 6m48s 10.42.0.6 master1 <none> <none>
(9) slave2节点查看集群信息
【本步骤在slave2虚拟机执行】
# 在slave2上查看Pod状态，可以看到和master上有相同的结果。
[root@slavebb install-k3s]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-68dccd9bb6-g8gtl 1/1 Running 0 20s 10.42.2.4 slave2 <none> <none>
nginx-deploy-68dccd9bb6-kxlww 1/1 Terminating 0 6m59s 10.42.1.6 slave <none> <none>
nginx-deploy-68dccd9bb6-llqt2 1/1 Running 0 6m59s 10.42.2.3 slave2 <none> <none>
nginx-deploy-68dccd9bb6-wblpt 1/1 Running 0 6m59s 10.42.0.6 master1 <none> <none>
上述示例说明，三个节点组成的集群，如果有一个节点出现故障，其他两个节点仍可以提供完整的服务。
在slave节点故障后，其仍然存在于集群的etcd中，但Kubernetes会把它的状态标记为NotReady，因为kubelet失联，无法汇报状态。在它上面的Pod不会被立即驱逐，而是继续保留在集群记录里，直到Node控制器触发驱逐。
Deployment控制器检测到Running状态的副本数不足(从3 -> 2)，会立即在健康节点(如本示例的slave2节点)上新建一个Pod副本，以确保Deployment的期望副本数为3。这是Kubernetes的基于污点的驱逐(Taint-Based Eviction)机制，属于节点故障自动恢复(Self-Healing)的一部分，由Node Lifecycle Controller和Deployment Controller协同完成。
码道开发者社区，聚焦华为云码道 CodeArts 代码智能体，沉淀 Agent、Skill、鸿蒙开发实战内容，供开发者查阅资料、交流技术、分享工程实践
更多推荐
13
0- 0
已为社区贡献14条内容
所有评论(0)