k8s 配置nfs-client-provisioner

參考鏈接: K8S 實(shí)戰(zhàn)（六）| 配置 NFS 動(dòng)態(tài)卷提供持久化存儲(chǔ)
更多詳情，公眾號(hào): ZisFinal

1、環(huán)境

• kubelet version: Kubernetes v1.22.0
• nfs: nfs-utils-1.3.0-0.68.el7.2.x86_64

2、前言

本節(jié)中 K8S 使用 NFS 遠(yuǎn)程存儲(chǔ)，為托管的 pod 提供了動(dòng)態(tài)存儲(chǔ)服務(wù)，pod 創(chuàng)建者無(wú)需關(guān)心數(shù)據(jù)以何種方式存在哪里，只需要提出需要多大空間的申請(qǐng)即可。

總體流程是：

1. 創(chuàng)建 NFS 服務(wù)器。
2. 創(chuàng)建 Service Account。用來(lái)管控 NFS provisioner 在k8s集群中運(yùn)行的權(quán)限。
3. 創(chuàng)建 StorageClass。負(fù)責(zé)創(chuàng)建 PVC 并調(diào)用 NFS provisioner 進(jìn)行預(yù)定的工作，并關(guān)聯(lián) PV 和 PVC。
4. 創(chuàng)建 NFS provisioner。有兩個(gè)功能,一個(gè)是在NFS共享目錄下創(chuàng)建掛載點(diǎn)(volume),二是建立 PV 并將 PV 與 NFS 掛載點(diǎn)建立關(guān)聯(lián)。

三、nfs 安裝

NFS 服務(wù)器安裝

yum install nfs-utils -y

# 啟動(dòng)服務(wù)
# 注意先后順序，先啟動(dòng)rpcbind，再啟動(dòng)nfs-server
systemctl start rpcbind
systemctl start nfs

# 開(kāi)機(jī)啟動(dòng)
systemctl enable rpcbind
systemctl enable nfs

# 創(chuàng)建共享目錄
mkdir -p /data/nfs

# 修改共享目錄權(quán)限
chmod -R 777 /data/nfs

# 修改配置文件
vim /etc/exports

# 添加共享目錄
/data/nfs *(rw,sync,no_root_squash,no_all_squash)

# 重啟服務(wù)
systemctl restart rpcbind
systemctl restart nfs

# 查看服務(wù)狀態(tài)
systemctl status rpcbind
systemctl status nfs

# 查看共享目錄
showmount -e nfs_server_ip

# 查看rpc服務(wù)
rpcinfo -p nfs_server_ip

四、nfs-privisoner 部署安裝

4.1 創(chuàng)建RBAC授權(quán)

nfs-rbac.yaml

---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: nfs-client-provisioner
  namesapce: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: kube-system
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

4.2 創(chuàng)建 Storageclass

nfs-storage.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-storage
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"  #---設(shè)置為默認(rèn)的storageclass
provisioner: nfs-client  #---動(dòng)態(tài)卷分配者名稱，必須和上面創(chuàng)建的"PROVISIONER_NAME"變量中設(shè)置的Name一致
parameters:
  archiveOnDelete: "true"  #---設(shè)置為"false"時(shí)刪除PVC不會(huì)保留數(shù)據(jù),"true"則保留數(shù)據(jù)
mountOptions:
  - hard        #指定為硬掛載方式
  - nfsvers=4   #指定NFS版本，這個(gè)需要根據(jù) NFS Server 版本號(hào)設(shè)置

4.3 創(chuàng)建nfs-client-provisioner自動(dòng)配置程序，以便自動(dòng)創(chuàng)建持久卷(PV)

• 自動(dòng)創(chuàng)建的 PV 以 namespace?namespace?{pvcName}-${pvName} 的命名格式創(chuàng)建在 NFS 上
• 當(dāng)這個(gè) PV 被回收后會(huì)以 archieved-namespace?namespace?{pvcName}-${pvName} 的命名格式存在 NFS 服務(wù)器上

nfs-provisioner-deploy.yaml

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-client-provisioner
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nfs-client-provisioner
  strategy:
    type: Recreate  #---設(shè)置升級(jí)策略為刪除再創(chuàng)建(默認(rèn)為滾動(dòng)更新)
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: nfs-client  #---nfs-provisioner的名稱，以后設(shè)置的storageclass要和這個(gè)保持一致
            - name: NFS_SERVER
              value: 192.168.1.13  #---NFS服務(wù)器地址，和 valumes 保持一致
            - name: NFS_PATH
              value: /data/nfs  #---NFS服務(wù)器目錄，和 valumes 保持一致
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.1.13  #---NFS服務(wù)器地址
            path: /data/nfs #---NFS服務(wù)器目錄

4.4 創(chuàng)建測(cè)試pvc

test-claim.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: nfs-storage
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi

4.5 部署安裝

# 創(chuàng)建sa
kubectl create -f nfs-rbac.yaml
# 創(chuàng)建 sc
kubectl create -f nfs-storage.yaml
# 創(chuàng)建nfs-privisioner-client
kubectl create -f nfs-provisioner-deploy.yaml
# 創(chuàng)建測(cè)試pvc 
kubectl create -f test-claim.yaml

創(chuàng)建完成后，可執(zhí)行一下命令進(jìn)行查看

查看sa

kubectl get sa -A | grep nfs-client
# output
# kube-system            nfs-client-provisioner               1         7d6h

查看sc

kubectl get sc
# output
# nfs-storage (default)   nfs-client    Delete          Immediate           false                  5h55m

查看pvc

kubectl get pvc -A | grep test-claim
# output
# default     test-claim                     Bound    pvc-0e32355e-12ca-4171-8b7c-d935a4aba080   10Gi       RWX            nfs-storage    7s

注: 如果test-claim一直出于pending狀態(tài)，需要查看nfs-client pod日志

五、問(wèn)題處理

5.1 pull image quay.io/external_storage/nfs-client-provisioner:latest timeout

文中提到的鏡像，因?yàn)槟承┰驘o(wú)法下載，所以需要通過(guò)科學(xué)上網(wǎng)方法，將鏡像下載到本地，然后在nfs-client-provisioner部署的節(jié)點(diǎn)上加載進(jìn)去即可

如果還是不能下載，通過(guò)下面方式下載即可

鏈接: https://pan.baidu.com/s/1qdRsI28AqVDxBTbY0PdQVA?pwd=y5f7 提取碼: y5f7

加載方式

如果是底層采用的是docker運(yùn)行方式，則直接執(zhí)行以下命令加載即可

docker load -i nfs-client-provisioner.tar

如果采用的是containerd運(yùn)行方式，則直接執(zhí)行以下命令加載即可

ctr -n=k8s.io image import nfs-client-provisioner.tar

5.2 selfLink was empty, can’t make reference

kubernetes 1.20之后，已經(jīng)移除了selfLink的支持，所以部署后，當(dāng)創(chuàng)建pvc的時(shí)候，nfs-provisioner 會(huì)報(bào)如下錯(cuò)誤

官方參考鏈接: issues_25

provision "default/test-claim" class "nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference

解決辦法

按照官方給定的解決辦法，在kube-apiserver啟動(dòng)參數(shù)上，加如下參數(shù)即可

--feature-gates=RemoveSelfLink=false

因?yàn)椴捎玫氖嵌M(jìn)制部署，所以直接修改kube-apiserver.service文件啟動(dòng)參數(shù)，在后面直接加上即可

完整kube-apiserver.service如下

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
      --v=2  \
      --logtostderr=true  \
      --allow-privileged=true  \
      --bind-address=0.0.0.0  \
      --secure-port=6443  \
      --insecure-port=0  \
      --advertise-address=192.168.1.7 \
      --service-cluster-ip-range=192.168.0.0/16  \
      --service-node-port-range=30000-32767  \
      --etcd-servers=https://192.168.1.7:2379,https://192.168.1.8:2379,https://192.168.1.9:2379 \
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \
      --enable-bootstrap-token-auth=true  \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \
      --requestheader-allowed-names=aggregator  \
      --requestheader-group-headers=X-Remote-Group  \
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \
      --requestheader-username-headers=X-Remote-User \
      --feature-gates=RemoveSelfLink=false # 最后添加這一行即可

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

然后在所有master節(jié)點(diǎn)，重啟kube-apiserver（因?yàn)樾薷牧藄ervice,重啟之前需要reload一下）

systemctl daemon-reload 
systemctl restart kube-apiserver

5.3 mkdir /peresistentVolume/* permission denied

出現(xiàn)這個(gè)問(wèn)題，一般是nfs服務(wù)器文件權(quán)限導(dǎo)致的，因?yàn)閚fs服務(wù)器，在操作的時(shí)候，用到的是用戶和用戶組都是nfsnobody

我nfs的掛載路徑為/data/nfs,所以將此目錄的用戶和用戶組全部修改為nfsnobody即可

chown -R nfsnobody:nfsnobody /data/nfs

執(zhí)行上述命令，即可解決報(bào)錯(cuò)問(wèn)題

5.4 mount nfs: mounting * failed, reason given by server: No such file or directory

出現(xiàn)這類問(wèn)題，是在pvc聲明的時(shí)候，引用了舊版的格式，舊版格式如下

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
spec:
  storageClassName: nfs-storage # 聲明的時(shí)候，采用了舊版的聲明，已經(jīng)不需要在spec下面添加了，更改到了annotations
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi

需要改成如下格式文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-585852.html

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: nfs-storage
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi

到了這里，關(guān)于k8s 配置nfs-client-provisioner的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！