Kubernetes技术详解-从理论到实践-(17)-调度-Scheduling
1 Scheduling调度简介
Kubernetes调度就是把待运行的Pod分配到最合适的节点上的过程。
2 Scheduling调度过程
Kubernetes的调度过程主要包括：过滤器筛掉不可用节点、打分器给剩余节点排序及把Pod绑定到最高分节点。
过滤阶段常见策略为：
PodFitsResources：CPU/内存/磁盘足够
PodFitsHost：指定节点名匹配
PodFitsHostPorts：端口不冲突
MatchNodeSelector：label 满足 nodeSelector
TolerationTaints：能容忍节点污点
VolumeBinding：PV 拓扑匹配
打分阶段的主要依据为：
LeastRequestedPriority：空闲资源越多分越高
BalancedResourceAllocation：CPU/内存使用率最均衡
ImageLocalityPriority：已存在镜像层得分高
NodeAffinityPriority：亲和性权重
InterPodAffinityPriority：Pod 亲和/反亲和权重
接下来，我们主要讲解三类调度方式。
硬编码：nodeName调度和nodeSelector调度
亲和性：节点亲合性、Pod亲和性和Pod反亲和性
污点和容忍度：污点、容忍度
3 硬编码
硬编码调度是把Pod调度到指定节点的最简单手段，主要包括nodeName和nodeSelector两种。
3.1 nodeName调度
原理：直接在 Pod 里写死 spec.nodeName: <节点名称>。
流程：kube-scheduler 完全跳过；kubelet 看到本节点名匹配就启动容器。
优点：最快，无调度延迟。
缺点：节点不存在或资源不足时 Pod 无法启动；节点改名即失效。
在Pod yaml中添加nodeName字段即可将Pod调度到指定Node节点。
spec:
nodeName: worker2
3.1.1 示例：nodeName调度
(1) 资源清理
[root@master 17-scheduling]# kubectl get pod,deploy
No resources found in default namespace.
(2) 查看集群节点
有三个节点，我们想把pod调度到worker2节点
[root@master 17-scheduling]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 125m v1.29.6+k3s1
worker1 Ready <none> 125m v1.29.6+k3s1
worker2 Ready <none> 124m v1.29.6+k3s1
(3) 编写nodeName调度的资源清单文件
编写nodeName调度的Deployment资源对象文件，只有1个Pod副本。
如果想让pod调度到指定的节点，只需要在containers同级别添加一行nodeName，pod将被调度到执行的node节点上
[root@master 17-scheduling]# cat nginx-nodename.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deploy
labels:
app: nginx-dm
spec:
selector:
matchLabels:
app: nginx-dm
replicas: 1
template:
metadata:
labels:
app: nginx-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
nodeName: worker2 # 将Pod调度到worker2节点
(4) 创建资源对象
[root@master 17-scheduling]# kubectl apply -f nginx-nodename.yaml
deployment.apps/nginx-deploy created
(5) 查看调度情况
# 可以看到pod已经被调度到worker2
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c967d7ff9-mmrvz 1/1 Running 0 6s 10.42.2.65 worker2 <none> <none>
3.2 nodeSelector调度
原理：给节点打标签，例如disk=ssd；在Pod里写nodeSelector: {disk: ssd}。
流程：仍经过scheduler，过滤阶段去掉不满足标签的节点，再正常打分/绑定。
优点：节点漂移、扩缩容更灵活；可组合多个标签。
缺点：需要提前给节点维护标签。
给节点打标签，然后在Pod yaml中添加nodeSelector字段选择Node标签，即可将Pod调度到指定Node节点。
给节点打标签
kubectl label node worker1 disk=ssd
添加nodeSelector字段
# Pod 声明
spec:
nodeSelector:
disk: ssd
3.2.1 示例：nodeSelector调度
通过键值对匹配节点的标签，将pod调度到符合条件的节点上。
为节点添加和删除标签的操作如下：
为worker1节点添加单个标签，标签名为node-role=worker1
kubectl label node worker1 node-role=worker1
为worker1节点添加多个标签
kubectl label node worker1 node-role=worker1 disktype=ssd
删除节点的单个标签
kubectl label node worker1 node-role-
删除一个节点的多个标签
kubectl label node worker1 node-role- disktype-
查看节点标签
kubectl get node worker1 --show-labels
下面是nodeSelector调度示例操作步骤。
(1) 清理环境
[root@master 17-scheduling]# kubectl get pod,deploy
No resources found in default namespace.
(2) 节点打标签
# 查看节点的标签
[root@master 17-scheduling]# kubectl get node --show-labels
NAME STATUS ROLES AGE VERSION LABELS
master Ready control-plane,master 155m v1.29.6+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/master=true,node.kubernetes.io/instance-type=k3s
worker1 Ready <none> 154m v1.29.6+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker1,kubernetes.io/os=linux,node.kubernetes.io/instance-type=k3s
worker2 Ready <none> 153m v1.29.6+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker2,kubernetes.io/os=linux,node.kubernetes.io/instance-type=k3s
# 为worker1节点添加标签disktype=ssd
[root@master 17-scheduling]# kubectl label node worker1 disktype=ssd
node/worker1 labeled
# 再次查看标签，可以看到worker1已经包含disktype=ssd的标签
[root@master 17-scheduling]# kubectl get node --show-labels
NAME STATUS ROLES AGE VERSION LABELS
master Ready control-plane,master 156m v1.29.6+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/master=true,node.kubernetes.io/instance-type=k3s
worker1 Ready <none> 155m v1.29.6+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,disktype=ssd,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker1,kubernetes.io/os=linux,node.kubernetes.io/instance-type=k3s
worker2 Ready <none> 154m v1.29.6+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker2,kubernetes.io/os=linux,node.kubernetes.io/instance-type=k3s
(3) 编写具有nodeSelector调度方式的Deployment资源清单文件
nodeSelector调度方式和nodeName相似，在containers同级目录下添加nodeSelector字段，字段内容为要选择的node标签。
[root@master 17-scheduling]# cat nginx-nodeselector.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deploy
labels:
app: nginx-dm
spec:
selector:
matchLabels:
app: nginx-dm
replicas: 1
template:
metadata:
labels:
app: nginx-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
nodeSelector: # 选择标签为disktype=ssd的Node
disktype: ssd
(4)创建Deployment
[root@master 17-scheduling]# kubectl apply -f nginx-nodeselector.yaml
deployment.apps/nginx-deploy created
(5) 查看调度情况
可以看到pod已经被调度到worker1上。
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-bc886696f-5tzk6 1/1 Running 0 5s 10.42.1.49 worker1 <none> <none>
4 亲和性
Kubernetes的亲和性（Affinity）是一组调度规则，用来告诉调度器哪些Pod应该（或必须）跟哪些节点/其他Pod放在一起或分开。相比写死nodeName或 nodeSelector，Affinity提供更丰富的表达式、软硬策略、拓扑域概念，让调度更灵活、更弹性。
4.1 亲和性三大类别
| 类别 | 作用对象 | 典型用途 |
|---|---|---|
| NodeAffinity | 节点标签 | 把Pod调度到“满足某些标签”的节点 |
| PodAffinity | 其他Pod标签 | 把Pod与“某些Pod”放在同一拓扑域（如同机架、同可用区） |
| PodAntiAffinity | 其他Pod标签 | 把Pod与“某些Pod”分散开（高可用、避免热点） |
每种类型又分为Required（硬约束）和Preferred（软偏好）。
4.2 硬约束和软偏好
硬约束与软偏好是 Kubernetes 亲和性（Affinity） 里的两种策略强度。
| 维度 | 硬约束 （Required） | 软偏好 （Preferred） |
|---|---|---|
| 英文关键字 | requiredDuringSchedulingIgnoredDuringExecution | preferredDuringSchedulingIgnoredDuringExecution |
| 语义 | 必须满足——找不到符合条件的节点就一直Pending | 尽量满足——先挑符合的节点，没有也能调度（退而求其次） |
| 调度失败结果 | Pod 处于Pending，直到出现匹配节点 | Pod 仍可被调度到不匹配节点，仅打分低 |
| 使用场景 | 强制合规、硬件要求、许可证限制 | 性能优化、能耗偏好、就近访问 |
硬约束举例：
Pod必须落在SSD磁盘节点，否则不调度。若集群无disk-type=ssd的节点，Pod一直Pending。
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: disk-type
operator: In
values: ["ssd"]
软偏好举例：
优先选SSD，没有也行。如果找不到SSD节点，仍可调度到普通磁盘节点，仅打分低。
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100 # 权重越高越优先
preference:
matchExpressions:
- key: disk-type
operator: In
values: ["ssd"]
4.3 NodeAffinity节点亲和性
节点亲和性是指根据节点的标签来控制pod的调度，将pod调度到满足特定条件的节点上。
节点亲和性有强制亲和性(requiredDuringSchedulingIgnoredDuringExecution)和偏好亲和性(preferredDuringSchedulingIgnoredDuringExecution)两种。
强制亲和性是硬性要求，硬约束，必须有满足条件的节点，pod才会被调度，否则无法启动。
偏好亲和性是软性要求，软偏好，调度器尽量将pod调度到满足条件的节点上，但如果没有这样的节点，pod仍然可以在不满足条件的节点上启动。
4.3.1 示例：节点强制亲和性
以下示例为节点强制亲和性演示，当有特定条件节点时Pod将被调度到该节点，当没有特定条件的节点时Pod将一直Pending。
(1) 环境清理
删除上个例子中为worker1添加的标签，删除deploy资源
[root@master 17-scheduling]# kubectl label node worker1 disktype-
node/worker1 unlabeled
[root@master 17-scheduling]# kubectl delete deploy nginx-deploy
deployment.apps "nginx-deploy" deleted
[root@master 17-scheduling]# kubectl get pod,deploy
No resources found in default namespace.
(2) 节点打标签
# 为worker1添加一个node-role=worker1的标签
[root@master 17-scheduling]# kubectl label node worker1 node-role=worker1
node/worker1 labeled
# 查看标签，可以看到worker1的LABELS包含刚才添加的标签
[root@master 17-scheduling]# kubectl get node --show-labels
NAME STATUS ROLES AGE VERSION LABELS
master Ready control-plane,master 175m v1.29.6+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=true,node-role.kubernetes.io/master=true,node.kubernetes.io/instance-type=k3s
worker1 Ready <none> 174m v1.29.6+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker1,kubernetes.io/os=linux,node-role=worker1,node.kubernetes.io/instance-type=k3s
worker2 Ready <none> 173m v1.29.6+k3s1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=k3s,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker2,kubernetes.io/os=linux,node.kubernetes.io/instance-type=k3s
(3) 编写具有节点亲和性功能的Deployment资源清单文件
节点亲和性，affinity字段，本规则为节点亲和性nodeAffinity，requiredDuringSchedulingIgnoredDuringExecution说明这是一个强制亲和性规则，匹配规则为key(node-role)有worker1值的这种节点。
[root@master 17-scheduling]# cat nginx-nodeaffinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deploy
labels:
app: nginx-dm
spec:
selector:
matchLabels:
app: nginx-dm
replicas: 1
template:
metadata:
labels:
app: nginx-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role
operator: In
values: ["worker1"]
(4) 创建Deployment资源
[root@master 17-scheduling]# kubectl apply -f nginx-nodeaffinity.yaml
deployment.apps/nginx-deploy created
(5) 查看Pod调度情况
可以看到Pod被调度到了符合节点强制亲和性的worker1节点上。
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-574f6599cb-4g5jr 1/1 Running 0 6s 10.42.1.50 worker1 <none> <none>
[root@master 17-scheduling]# kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deploy 1/1 1 1 19s
(6) 删除Deployment，为后面步骤做准备
[root@master 17-scheduling]# kubectl delete deploy nginx-deploy
deployment.apps "nginx-deploy" deleted
[root@master 17-scheduling]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 176m v1.29.6+k3s1
worker1 Ready <none> 175m v1.29.6+k3s1
worker2 Ready <none> 174m v1.29.6+k3s1
(7) 将Deployment资源修改为调度到无对应标签的Node上
修改nginx-nodeaffinity.yaml文件，将values中的worker1改为woker3，我们没有为任何节点添加过node-type=worker3的标签，pod应该不会正常启动。
[root@master 17-scheduling]# vim nginx-nodeaffinity.yaml
[root@master 17-scheduling]# cat nginx-nodeaffinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deploy
labels:
app: nginx-dm
spec:
selector:
matchLabels:
app: nginx-dm
replicas: 1
template:
metadata:
labels:
app: nginx-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role
operator: In
values: ["worker3"]
(8) 再次创建Deployment
[root@master 17-scheduling]# kubectl apply -f nginx-nodeaffinity.yaml
deployment.apps/nginx-deploy created
(9) 查看Pod调度状态
# 可以看到Pod处于Pending状态
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-8d76fc9cd-9ctnp 0/1 Pending 0 7s <none> <none> <none> <none>
# 查看Pod详情，可以看到其Pending状态为找不到符合要求的Node
[root@master 17-scheduling]# kubectl describe pod nginx-deploy-8d76fc9cd-9ctnp
Name: nginx-deploy-8d76fc9cd-9ctnp
(。。。部分内容略。。。)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 18s default-scheduler 0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
4.4 PodAffinity Pod亲和性
Pod亲和性允许根据其他Pod的标签来控制Pod的调度。可以指定Pod只能调度到符合条件的Pod相同的节点上。
和Node亲和性相似，Pod亲和性也有强制亲和性(requiredDuringSchedulingIgnoredDuringExecution)与偏好亲和性(preferredDuringSchedulingIgnoredDuringExecution)两种，下面的例子演示Pod强制亲和性。
4.4.1 示例：Pod强制亲和性
(1) 查看集群状态
[root@master 17-scheduling]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 3h16m v1.29.6+k3s1
worker1 Ready <none> 3h15m v1.29.6+k3s1
worker2 Ready <none> 3h14m v1.29.6+k3s1
[root@master 17-scheduling]# kubectl get pod,deploy
No resources found in default namespace.
(2) 编写一个普通的Deployment资源文件
先创建一个标签为app=nginx-dm的pod，后面创建pod亲和性规则，调度到和带app=nginx-dm标签的pod相同的节点上。
[root@master 17-scheduling]# cat nginx-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deploy
labels:
app: nginx-dm
spec:
selector:
matchLabels:
app: nginx-dm
replicas: 1
template:
metadata:
labels:
app: nginx-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
(3) 创建Deployment
[root@master 17-scheduling]# kubectl apply -f nginx-deploy.yaml
deployment.apps/nginx-deploy created
(4) 查看Pod调度情况
由于没有任何规则，本Pod由K3s自动调度，可以看到Pod被调度到worker1节点，通过kubectl --show-labels参数可以查看Pod标签。
# Pod被调度到worker1上
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-mzmvn 1/1 Running 0 4s 10.42.1.51 worker1 <none> <none>
# kubectl --show-labels参数可以查看Pod的标签
[root@master 17-scheduling]# kubectl get pod -o wide --show-labels
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS
nginx-deploy-7c7bbf4cc9-mzmvn 1/1 Running 0 16s 10.42.1.51 worker1 <none> <none> app=nginx-dm,pod-template-hash=7c7bbf4cc9
(5) 创建带Pod亲和性规则的Deployment
创建带Pod亲和性规则的Deployment，期望调度到标签为app=nginx-dm的Pod所在节点上。
[root@master 17-scheduling]# cat nginx-podaffinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-pa-deploy
labels:
app: nginx-pa-dm
spec:
replicas: 1
selector:
matchLabels:
app: nginx-pa-dm
template:
metadata:
labels:
app: nginx-pa-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values: ["nginx-dm"]
topologyKey: kubernetes.io/hostname
(6) 创建带Pod亲和性规则的Deployment
[root@master 17-scheduling]# kubectl apply -f nginx-podaffinity.yaml
deployment.apps/nginx-pa-deploy created
(7) 查看Pod调度情况
可以看到已经被调度到与标签为app=nginx-dm的pod所在节点上。
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-mzmvn 1/1 Running 0 63s 10.42.1.51 worker1 <none> <none>
nginx-pa-deploy-94c5cbd48-pj5mw 1/1 Running 0 8s 10.42.1.52 worker1 <none> <none>
(8) 删除刚才的Deployment资源
删除刚才的nginx-pa-deploy资源，为后面Pod硬约束亲和性验证做准备
[root@master 17-scheduling]# kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deploy 1/1 1 1 109s
nginx-pa-deploy 1/1 1 1 54s
[root@master 17-scheduling]# kubectl delete deploy nginx-pa-deploy
deployment.apps “nginx-pa-deploy” deleted
(9) 修改nginx-podaffinity.yaml资源清单文件
修改文件，改为与标签为app in nginx-dm1的Pod调度到同一个节点，当然，系统不存在这种标签的Pod，期望结果为Pod一直处于Pending状态。
修改pod亲和性规则，修改后的内容见下
[root@master 17-scheduling]# vim nginx-podaffinity.yaml
[root@master 17-scheduling]# cat nginx-podaffinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-pa-deploy
labels:
app: nginx-pa-dm
spec:
replicas: 1
selector:
matchLabels:
app: nginx-pa-dm
template:
metadata:
labels:
app: nginx-pa-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values: ["nginx-dm1"] # 这里由nginx-dm改为了nginx-dm1
topologyKey: kubernetes.io/hostname
(10) 再次创建Deployment资源对象
[root@master 17-scheduling]# kubectl apply -f nginx-podaffinity.yaml
deployment.apps/nginx-pa-deploy created
(11) 查看Pod调度情况
由于集群中不存在标签为app in nginx-dm1的Pod，因此本Pod无法找到对应的Node，无法调度，将一直处于Pending状态。通过kubectl describe可以查看Pod失败详情。
# Pod处于Pending状态
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-mzmvn 1/1 Running 0 2m49s 10.42.1.51 worker1 <none> <none>
nginx-pa-deploy-85f6d847ff-9vdk8 0/1 Pending 0 6s <none> <none> <none> <none>
# 查看pod详情，发现是由于不满足pod亲和性规则导致
[root@master 17-scheduling]# kubectl describe pod nginx-pa-deploy-85f6d847ff-9vdk8
Name: nginx-pa-deploy-85f6d847ff-9vdk8
(中间略。。。)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 13s default-scheduler 0/3 nodes are available: 3 node(s) didn't match pod affinity rules. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
4.5 PodAntiAffinity Pod反亲和性
与Pod亲和性相反，Pod反亲和性可以根据其他Pod的标签，避免将Pod调度到与有这些标签的Pod所在节点上。
和Node亲和性、Pod亲和性相似，Pod反亲和性也有强制反亲和性(requiredDuringSchedulingIgnoredDuringExecution)与偏好反亲和性(preferredDuringSchedulingIgnoredDuringExecution)两种，下面的例子演示Pod强制反亲和性。
4.5.1 示例：Pod强制反亲和性
(1) 先清理之前示例创建的资源
[root@master 17-scheduling]# kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deploy 1/1 1 1 19m
nginx-pa-deploy 0/1 1 0 16m
[root@master 17-scheduling]# kubectl delete deploy nginx-deploy nginx-pa-deploy
deployment.apps "nginx-deploy" deleted
deployment.apps "nginx-pa-deploy" deleted
[root@master 17-scheduling]# kubectl get pod,deploy
No resources found in default namespace.
[root@master 17-scheduling]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 3h36m v1.29.6+k3s1
worker1 Ready <none> 3h35m v1.29.6+k3s1
worker2 Ready <none> 3h34m v1.29.6+k3s1
(2) 创建一个普通的Deployment
创建一个带有app=nginx-dm标签的Deployment，包含1个Pod副本。
[root@master 17-scheduling]# cat nginx-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deploy
labels:
app: nginx-dm
spec:
selector:
matchLabels:
app: nginx-dm
replicas: 1
template:
metadata:
labels:
app: nginx-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
(3) 创建Deployment资源
[root@master 17-scheduling]# kubectl apply -f nginx-deploy.yaml
deployment.apps/nginx-deploy created
(4) 查看Pod调度情况
Pod创建成功，可以看到被调度到了worker2上。
这里的Pod调度和亲和性、反亲和性无关，因为Deployment资源清单文件还没有和亲和性、反亲和性有关的描述。
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-5r2zs 1/1 Running 0 4s 10.42.2.66 worker2 <none> <none>
(5) 编写具有反亲和性规则的Deployment资源清单文件
Pod反亲和性与Pod亲和性规则非常相似，下面仅把nginx-podaffinity.yaml的spec.template.spec.affinity.podAffinity改为了podAntiAffinity。
[root@master 17-scheduling]# cat nginx-podantiaffinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-paa-deploy
labels:
app: nginx-paa-dm
spec:
replicas: 1
selector:
matchLabels:
app: nginx-paa-dm
template:
metadata:
labels:
app: nginx-paa-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
affinity:
podAntiAffinity: # pod亲和性例子中，这个字段为podAffnitify
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values: ["nginx-dm"]
topologyKey: kubernetes.io/hostname
(6) 创建具有反亲和性规则的Deployment
[root@master 17-scheduling]# kubectl apply -f nginx-podantiaffinity.yaml
deployment.apps/nginx-paa-deploy created
(7) 查看Pod调度情况
可以看到新创建的Pod被调度到了与标签为app=nginx-dm的Pod不同的节点上，这就是Pod反亲和性的作用。
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-5r2zs 1/1 Running 0 45s 10.42.2.66 worker2 <none> <none>
nginx-paa-deploy-f4d6c8585-fvslw 1/1 Running 0 4s 10.42.1.53 worker1 <none> <none>
(8) 如果我们把app=nginx-dm的Deployment副本改为2，Pod将调度到两个节点上。由于本集群有三个工作节点，再次运行具有反亲和性规则的Deployment，将只能调度到不运行app=nginx-dm的节点上。这里不再演示。
4.6 亲和性使用场景
节点亲和性(Node Affinity)
资源隔离：将某些节点专门用于运行特定类型的工作负载，例如GPU负载或数据库，设置节点亲和性的pod才会被调度到该节点。
高可用性：可以将多个实例分布在不同的节点上， 以提高容错能力。
区域亲和性：可以通过设置节点亲和性将pod调度到特定的区域或机架上。
Pod 亲和性(Pod Affinity)
减少网络延迟：可以设置pod亲和性，让相关的pod调度到同一节点，如缓存和数据库pod设置亲和性，可以调度到同一节点。
共享资源：将需要资源共享的Pod放在同一节点，可提高性能。
Pod反亲和性(Pod Anti-Affinity)
高可用性：设置反亲和性，将多个实例分布在不同的节点，以提高容错能力。
避免资源竞争：将资源密集型的pod分散到不同的node，减少资源竞争。
5 污点(Taints)和容忍度(Tolerations)
污点和容忍度是用于控制Pod调度的机制，允许将某些节点标记为不受欢迎，并指定哪些Pod可以容忍这些标记。
为节点打了污点，则只有配置了容忍度的Pod才会调度到该节点，其他Pod不会被调度到该节点。
5.1 污点(Taints)
污点是节点的属性，用于标记节点，使得默认情况下没有容忍度的pod无法调度到该节点。
污点是键值对数据，其格式为key=value:effect，这是k3s中的声明式授权规则的一种表示方式，常用于定义访问控制策略。
其中，key=value表示授权规则的条件，key是键，value是值，effect表示应该执行什么操作。
5.1.1 污点effect的三种取值和作用
NoSchedule：硬排斥，Pod不能被调度到该节点上，但已经运行在该节点上的Pod不会被驱逐。
PreferNoSchedule：软排斥，调度器会尽量避免将Pod调度到该节点上，但不是强制的。
NoExecute：Pod不能被调度到该节点上，而且已经运行在该节点上的Pod也会被驱逐。
5.1.2 污点操作
为节点添加单个污点
kubectl taint node master node-type=dev:NoExecute
为节点添加多个污点
kubectl taint node master node-type1=dev1:NoExecute
kubectl taint node master node-type2=dev2:NoExecute
为节点删除单个污点
kubectl taint node master node-type1=dev1:NoExecute-
为节点删除多个污点
kubectl taint node master node-type1- node-type2-
查看节点的污点，master节点没有污点，worker1节点有两个污点
[root@master ~]# kubectl describe node master
Name: master
(。。。部分内容略。。。)
Taints: <none>
(。。。部分内容略。。。)
[root@master ~]# kubectl describe node worker1
Name: worker1
(。。。部分内容略。。。)
Taints: node-type1=dev1:NoExecute
node-type2=dev2:NoExecute
(。。。部分内容略。。。)
5.2 容忍度(Tolerations)
容忍度是Pod的属性，用于声明Pod可以容忍哪些污点。如果Pod的容忍度与节点的污点匹配，Pod就可以被调度到该节点上。
5.2.1 示例：污点的驱逐效果
以下示例演示集群中的三个节点，各自运行着1个Pod，对其中一个Node打污点，该Node上的Pod根据污点规则，被驱逐到其他节点。
(1) 查看集群信息
# 集群中有三个节点
[root@master 5-deployment]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 5h7m v1.29.6+k3s1
worker1 Ready <none> 5h6m v1.29.6+k3s1
worker2 Ready <none> 5h5m v1.29.6+k3s1
# 目前还没有Deployment
[root@master 5-deployment]# kubectl get deploy
No resources found in default namespace.
(2) 创建Deployment资源清单文件
创建一个具有3个副本的Deployment资源清单文件。
[root@master 5-deployment]# cat nginx-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deploy
labels:
app: nginx-dm
spec:
selector:
matchLabels:
app: nginx-dm
replicas: 3
template:
metadata:
labels:
app: nginx-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
(3) 创建Deployment资源
# 创建资源
[root@master 5-deployment]# kubectl apply -f nginx-deploy.yaml
deployment.apps/nginx-deploy created
# 创建成功后，可以看到三个pod分别被调度到master、worker1和worker2上。
[root@master 5-deployment]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-8h4mc 1/1 Running 0 5s 10.42.1.56 worker1 <none> <none>
nginx-deploy-7c7bbf4cc9-kvvmd 1/1 Running 0 5s 10.42.0.6 master <none> <none>
nginx-deploy-7c7bbf4cc9-p78zr 1/1 Running 0 5s 10.42.2.67 worker2 <none> <none>
(4) 为节点添加污点
为节点master添加污点，效果为NoExecute，Pod不能调度到该节点上，且已经运行在该节点上的Pod也会被驱逐。
[root@master 5-deployment]# kubectl taint node master node-type=dev:NoExecute
node/master tainted
(5) 查看驱逐效果
可以看到master节点上已经没有运行的pod，worker2节点新增了一个pod。
[root@master 5-deployment]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-8h4mc 1/1 Running 0 30s 10.42.1.56 worker1 <none> <none>
nginx-deploy-7c7bbf4cc9-p78zr 1/1 Running 0 30s 10.42.2.67 worker2 <none> <none>
nginx-deploy-7c7bbf4cc9-zf666 1/1 Running 0 3s 10.42.2.68 worker2 <none> <none>
5.3 综合示例：污点和容忍度
(1) 查看集群情况
# 集群中有三个节点
[root@master 17-scheduling]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 5h26m v1.29.6+k3s1
worker1 Ready <none> 5h25m v1.29.6+k3s1
worker2 Ready <none> 5h25m v1.29.6+k3s1
# master节点有一个污点
[root@master 17-scheduling]# kubectl describe node master
Name: master
(。。。部分内容略。。。)
Taints: node-type=dev:NoExecute
(。。。部分内容略。。。)
# 目前有三个没有设置容忍度规则的Pod，都未被调度到master节点上
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-8h4mc 1/1 Running 0 19m 10.42.1.56 worker1 <none> <none>
nginx-deploy-7c7bbf4cc9-p78zr 1/1 Running 0 19m 10.42.2.67 worker2 <none> <none>
nginx-deploy-7c7bbf4cc9-zf666 1/1 Running 0 18m 10.42.2.68 worker2 <none> <none>
(2) 编写带Pod容忍度规则的Deployment资源清单文件
创建一个带Pod容忍度规则的Deployment，该Deployment有3个副本。容忍度规则为匹配node-type=dev:NoExecute污点的节点。
tolerationSeconds表示容忍时间，超过这个时间，将继续根据容忍度规则匹配污点，如果能匹配上，则继续在该节点上运行Pod，直到超时后重新检查，如果容忍度不能匹配上该节点，Pod将被驱逐。
[root@master 17-scheduling]# cat nginx-tolerations.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-tol-deploy
labels:
app: nginx-tol-dm
spec:
selector:
matchLabels:
app: nginx-tol-dm
replicas: 3
template:
metadata:
labels:
app: nginx-tol-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
tolerations:
- key: "node-type"
operator: "Equal"
value: "dev"
effect: "NoExecute"
tolerationSeconds: 60
(3) 创建带Pod容忍度规则的Deployment
[root@master 17-scheduling]# kubectl apply -f nginx-tolerations.yaml
deployment.apps/nginx-tol-deploy created
(4) 查看Pod调度情况
创建的2个pod，分别调度到了2个节点，master/worker1/worker2。master设置了污点，Pod设置了容忍度，两者能匹配上，所以Pod才会调度到master节点。
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-8h4mc 1/1 Running 0 19m 10.42.1.56 worker1 <none> <none>
nginx-deploy-7c7bbf4cc9-p78zr 1/1 Running 0 19m 10.42.2.67 worker2 <none> <none>
nginx-deploy-7c7bbf4cc9-zf666 1/1 Running 0 19m 10.42.2.68 worker2 <none> <none>
nginx-tol-deploy-748ff4b455-cxnbf 1/1 Running 0 6s 10.42.1.59 worker1 <none> <none>
nginx-tol-deploy-748ff4b455-p6rvd 1/1 Running 0 6s 10.42.2.69 worker2 <none> <none>
nginx-tol-deploy-748ff4b455-tz8s2 1/1 Running 0 6s 10.42.0.7 master <none> <none>
(5) 查看容忍时间tolerationSeconds效果
# Pod在master节点上运行了60s，到达tolerationSeconds设置的时间，因此节点上的Pod先被停止运行，再继续进行容忍度和污点的匹配，如果能匹配上，则重新创建新的Pod，运行在本节点上。
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-8h4mc 1/1 Running 0 20m 10.42.1.56 worker1 <none> <none>
nginx-deploy-7c7bbf4cc9-p78zr 1/1 Running 0 20m 10.42.2.67 worker2 <none> <none>
nginx-deploy-7c7bbf4cc9-zf666 1/1 Running 0 20m 10.42.2.68 worker2 <none> <none>
nginx-tol-deploy-748ff4b455-2kkjr 0/1 ContainerCreating 0 0s <none> master <none> <none>
nginx-tol-deploy-748ff4b455-cxnbf 1/1 Running 0 60s 10.42.1.59 worker1 <none> <none>
nginx-tol-deploy-748ff4b455-p6rvd 1/1 Running 0 60s 10.42.2.69 worker2 <none> <none>
nginx-tol-deploy-748ff4b455-tz8s2 1/1 Terminating 0 60s 10.42.0.7 master <none> <none>
# master节点的污点没有任何变化，因此新创建的Pod仍可以在本节点运行。
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-8h4mc 1/1 Running 0 20m 10.42.1.56 worker1 <none> <none>
nginx-deploy-7c7bbf4cc9-p78zr 1/1 Running 0 20m 10.42.2.67 worker2 <none> <none>
nginx-deploy-7c7bbf4cc9-zf666 1/1 Running 0 20m 10.42.2.68 worker2 <none> <none>
nginx-tol-deploy-748ff4b455-2kkjr 1/1 Running 0 4s 10.42.0.8 master <none> <none>
nginx-tol-deploy-748ff4b455-cxnbf 1/1 Running 0 64s 10.42.1.59 worker1 <none> <none>
nginx-tol-deploy-748ff4b455-p6rvd 1/1 Running 0 64s 10.42.2.69 worker2 <none> <none>
# 下一个60s，继续删除master节点上的Pod，再次进行容忍度检查。
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-8h4mc 1/1 Running 0 21m 10.42.1.56 worker1 <none> <none>
nginx-deploy-7c7bbf4cc9-p78zr 1/1 Running 0 21m 10.42.2.67 worker2 <none> <none>
nginx-deploy-7c7bbf4cc9-zf666 1/1 Running 0 21m 10.42.2.68 worker2 <none> <none>
nginx-tol-deploy-748ff4b455-5nffz 0/1 ContainerCreating 0 1s <none> master <none> <none>
nginx-tol-deploy-748ff4b455-cxnbf 1/1 Running 0 2m1s 10.42.1.59 worker1 <none> <none>
nginx-tol-deploy-748ff4b455-p6rvd 1/1 Running 0 2m1s 10.42.2.69 worker2 <none> <none>
# Pod又被调度到了master上
# 如果想让Pod在到达tolerationSeconds时间后不被调度到master上，可以动态调整Pod的容忍度：
# kubectl patch pod pod-name -p '{"spec":{"tolerations":[]}}'
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-8h4mc 1/1 Running 0 21m 10.42.1.56 worker1 <none> <none>
nginx-deploy-7c7bbf4cc9-p78zr 1/1 Running 0 21m 10.42.2.67 worker2 <none> <none>
nginx-deploy-7c7bbf4cc9-zf666 1/1 Running 0 21m 10.42.2.68 worker2 <none> <none>
nginx-tol-deploy-748ff4b455-5nffz 1/1 Running 0 3s 10.42.0.9 master <none> <none>
nginx-tol-deploy-748ff4b455-cxnbf 1/1 Running 0 2m3s 10.42.1.59 worker1 <none> <none>
nginx-tol-deploy-748ff4b455-p6rvd 1/1 Running 0 2m3s 10.42.2.69 worker2 <none> <none>
(6) 改变master节点的污点情况
为master节点新增一个污点
[root@master 17-scheduling]# kubectl taint node master node-locate=area1:NoExecute
node/master tainted
(7) 再次查看Pod调度情况
由于Pod的容忍度无法匹配所有的污点，Pod不会被调度到master节点运行，已经运行的Pod也被驱逐到了其他节点。
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-8h4mc 1/1 Running 0 21m 10.42.1.56 worker1 <none> <none>
nginx-deploy-7c7bbf4cc9-p78zr 1/1 Running 0 21m 10.42.2.67 worker2 <none> <none>
nginx-deploy-7c7bbf4cc9-zf666 1/1 Running 0 21m 10.42.2.68 worker2 <none> <none>
nginx-tol-deploy-748ff4b455-cxnbf 1/1 Running 0 2m9s 10.42.1.59 worker1 <none> <none>
nginx-tol-deploy-748ff4b455-hldfr 1/1 Running 0 6s 10.42.2.70 worker2 <none> <none>
nginx-tol-deploy-748ff4b455-p6rvd 1/1 Running 0 2m9s 10.42.2.69 worker2 <none> <none>
(8) 调整Pod容忍度
如果想让Pod重新调度到master节点，可以修改Pod的容忍度。
Pod容忍度可以动态调整，也可以修改yaml文件调整。
动态调整使用kubectl语句：
kubectl patch pod my-pod -p '{"spec":{"tolerations":[{"key":"node-locate","operator":"Equal","value":"area1","effect":"NoExecute"}]}}'
下面为静态调整，即修改yaml文件：
[root@master 17-scheduling]# cp nginx-tolerations.yaml nginx-tolerations2.yaml
[root@master 17-scheduling]# vim nginx-tolerations2.yaml
[root@master 17-scheduling]# cat nginx-tolerations2.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-tol2-deploy
labels:
app: nginx-tol2-dm
spec:
selector:
matchLabels:
app: nginx-tol2-dm
replicas: 3
template:
metadata:
labels:
app: nginx-tol2-dm
spec:
containers:
- name: nginx
image: nginx:1.27.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
tolerations:
- key: "node-type"
operator: "Equal"
value: "dev"
effect: "NoExecute"
tolerationSeconds: 60
- key: "node-locate" #添加一个新的容忍度规则
operator: "Equal"
value: "area1"
effect: "NoExecute"
(9) 更新Pod配置
[root@master 17-scheduling]# kubectl apply -f nginx-tolerations2.yaml
deployment.apps/nginx-tol2-deploy created
(10) 再次查看Pod调度情况
可以看到，master上又有Pod被调度过来。
如果Node设置了多个污点，则Pod也需要设置多个容忍度规则。只有所有的污点都被容忍，Pod才会被调度到节点上。
[root@master 17-scheduling]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploy-7c7bbf4cc9-8h4mc 1/1 Running 0 46m 10.42.1.56 worker1 <none> <none>
nginx-deploy-7c7bbf4cc9-p78zr 1/1 Running 0 46m 10.42.2.67 worker2 <none> <none>
nginx-deploy-7c7bbf4cc9-zf666 1/1 Running 0 45m 10.42.2.68 worker2 <none> <none>
nginx-tol-deploy-748ff4b455-cxnbf 1/1 Running 0 26m 10.42.1.59 worker1 <none> <none>
nginx-tol-deploy-748ff4b455-hldfr 1/1 Running 0 7m51s 10.42.2.70 worker2 <none> <none>
nginx-tol-deploy-748ff4b455-p6rvd 1/1 Running 0 26m 10.42.2.69 worker2 <none> <none>
nginx-tol2-deploy-8dcc4488d-2ckmd 1/1 Running 0 5s 10.42.0.26 master <none> <none>
nginx-tol2-deploy-8dcc4488d-7h6cx 1/1 Running 0 5s 10.42.2.71 worker2 <none> <none>
nginx-tol2-deploy-8dcc4488d-x6zz9 1/1 Running 0 5s 10.42.1.60 worker1 <none> <none>
5.4 污点和容忍度使用场景
污点(Taints)和容忍度(Tolerations)在Kubernetes中非常灵活，适用于多种场景。以下举一些常见的应用场景：
- 资源预留
当希望某些节点专门用于运行特定的系统Pod或关键应用时，可以使用污点来防止其他Pod被调度到这些节点上。
如：为运行数据库的节点添加污点，然后在数据库pod添加容忍度，只有数据库相关的pod才会被调度到这些节点上。 - 节点隔离
当希望将某些类型的工作负载隔离到特定的节点上时，可以使用污点和容忍度来实现。
如：为运行GPU的节点添加污点，只有设置容忍度的pod才会被调度到GPU节点上。 - 自动驱逐
当节点资源不足时，希望能自动驱逐某些Pod，就可以使用带有NoExecute效果的污点。
如：希望在节点内存压力大时自动驱逐低优先级的Pod，为节点添加标签memory-pressure=true:NoExecute，低优先级的pod将被驱逐。 - 维护模式
当需要对节点进行维护时，可以使用污点来驱逐节点上的Pod。 - 多租户环境
在多租户环境中，可以使用污点和容忍度来隔离不同租户的Pod。
码道开发者社区，聚焦华为云码道 CodeArts 代码智能体，沉淀 Agent、Skill、鸿蒙开发实战内容，供开发者查阅资料、交流技术、分享工程实践
更多推荐
18
0- 0
已为社区贡献14条内容
所有评论(0)