Kubernetes使用NFS作为StorageClass(记录)
·
由于我Kubernetes试验环境中使用local-path本地盘作为StorageClass不能解决多个节点共享使用的问题,因此准备使用nfs共享盘作为默认SC,在我的一台k8s机群节点上挂一块盘并做成nfs,然后作为k8s集群的默认存储使用,记录一下操作步骤。
1) 在节点上准备磁盘与挂载点
假设新磁盘是 /dev/sdb,挂载到 /data/nfs
# 1. 分区(也可直接整盘用),示例用 gdisk/fdisk 自定
sudo fdisk /dev/sdb
Welcome to fdisk (util-linux 2.37.4).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x6ef5b4db.
Command (m for help): p
Disk /dev/sdb: 300 GiB, 322122547200 bytes, 629145600 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x6ef5b4db
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-629145599, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-629145599, default 629145599):
Created a new partition 1 of type 'Linux' and of size 300 GiB.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
[root@k8s-master-1 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 300G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 299G 0 part
├─rl_192-root 253:0 0 70G 0 lvm /
├─rl_192-swap 253:1 0 3.9G 0 lvm
└─rl_192-home 253:2 0 225.1G 0 lvm /home
sdb 8:16 0 300G 0 disk
└─sdb1 8:17 0 300G 0 part
[root@k8s-master-1 ~]#
# 2. 建文件系统(xfs 例子)
sudo mkfs.xfs /dev/sdb1
# 3. 创建挂载目录
sudo mkdir -p /data/nfs
# 4. 挂载
sudo mount /dev/sdb1 /data/nfs
# 5. 开机自动挂载:写入 /etc/fstab
echo '/dev/sdb1 /data/nfs xfs defaults 0 2' | sudo tee -a /etc/fstab
2) 安装并配置 NFS 服务
sudo yum install -y nfs-utils. #需要做所有node节点上执行
sudo systemctl enable --now nfs-server
导出目录(设为所有集群节点可读写;192.168.170.0/24 是集群node节点网段):
sudo chmod -R 0777 /data/nfs # 简化权限;生产可收紧
# /etc/exports
echo '/data/nfs 192.168.170.0/24(rw,sync,no_subtree_check,no_root_squash)' | sudo tee -a /etc/exports
# 生效
sudo exportfs -rav
防火墙开放(如开启了 firewalld):
# firewalld
sudo firewall-cmd --permanent --add-service=nfs
sudo firewall-cmd --permanent --add-service=mountd
sudo firewall-cmd --permanent --add-service=rpc-bind
sudo firewall-cmd --reload
验证 NFS 是否可见:
showmount -e 192.168.170.123
# 应能看到 /data/nfs 被导出
SELinux(如是启用 enforcing):
sudo setsebool -P nfs_export_all_rw 1
如需要,给目录打合适上下文:sudo chcon -Rt nfs_t /data/nfs
3) 动态绑定nfs类型存储:nfs-subdir-external-provisioner
如果希望 自动创建子目录 + 动态 PVC,可以在集群里安装 NFS 动态供给器。
(1) 安装(Helm)----(失败!找不到源,改为直接手动yaml):
helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
helm repo update
helm install nfs-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
--set nfs.server=192.168.170.123 \
--set nfs.path=/data/nfs \
--set storageClass.defaultClass=true \
--set storageClass.name=nfs-rwx
直接手动执行yaml—(成功!):
kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Namespace
metadata:
name: nfs-provisioner
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-subdir-external-provisioner
namespace: nfs-provisioner
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: nfs-subdir-external-provisioner-runner
rules:
- apiGroups: [""]
resources: ["nodes","persistentvolumes","persistentvolumeclaims","events"]
verbs: ["get","list","watch","create","delete","update","patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["endpoints","services"]
verbs: ["get","create","list","watch","update","patch"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get","list","watch","create","update","patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: run-nfs-subdir-external-provisioner
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nfs-subdir-external-provisioner-runner
subjects:
- kind: ServiceAccount
name: nfs-subdir-external-provisioner
namespace: nfs-provisioner
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-subdir-external-provisioner
namespace: nfs-provisioner
spec:
replicas: 1
selector:
matchLabels:
app: nfs-subdir-external-provisioner
template:
metadata:
labels:
app: nfs-subdir-external-provisioner
spec:
serviceAccountName: nfs-subdir-external-provisioner
containers:
- name: nfs-subdir-external-provisioner
image: registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
env:
- name: PROVISIONER_NAME
value: k8s-sigs.io/nfs-subdir-external-provisioner
- name: NFS_SERVER
value: "192.168.170.123" # ← 换成你的 NFS 服务器 IP
- name: NFS_PATH
value: "/data/nfs" # ← 换成你的导出目录
- name: ENABLE_LEADER_ELECTION
value: "true"
- name: PATH_PATTERN
value: "${.PVC.namespace}/${.PVC.name}"
- name: ARCHIVE_ON_DELETE
value: "true"
volumeMounts:
- name: nfs
mountPath: /persistentvolumes
volumes:
- name: nfs
nfs:
server: 192.168.170.123 # ← 同上
path: /data/nfs # ← 同上
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-rwx
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
reclaimPolicy: Delete
allowVolumeExpansion: true
parameters:
pathPattern: "${.PVC.namespace}/${.PVC.name}"
archiveOnDelete: "true"
mountOptions:
- vers=3
YAML
(2) 应用使用时只需创建 PVC(无需手工 PV):
#pvc-dyn.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-dynamic
spec:
accessModes: ["ReadWriteMany"]
storageClassName:
resources:
requests:
storage: 10Gi
执行并检查验证:
[root@k8s-master-1 nfs]# kubectl apply -f pvc-dyn.yml
persistentvolumeclaim/pvc-dynamic created
[root@k8s-master-1 nfs]# k get pvc |grep dyn
pvc-dynamic Bound pvc-c412fae9-ef56-433a-87ac-95586ee4ccc1 10Gi RWX nfs-rwx <unset> 12s
[root@k8s-master-1 nfs]# k get pv|grep dyn
pvc-c412fae9-ef56-433a-87ac-95586ee4ccc1 10Gi RWX Delete Bound default/pvc-dynamic nfs-rwx <unset> 23s
[root@k8s-master-1 nfs]#
成功!
更多推荐



所有评论(0)