K8s — Elasticsearch dynamic persistance volume with nfs — simple

5 min readMar 20, 2021

Create NFS with Ubuntu 20.04 (/mnt/nfs)

#apt -y install nfs-kernel-server
#chown -R nobody:nogroup /mnt/nfs/

Add below to /etc/exports

/mnt/nfs *(rw,sync,no_subtree_check,no_root_squash,no_all_squash,insecure)

#{path to export} {client ip to allow * mean any} {options…}

sudo systemctl restart nfs-kernel-server

exportfs -v

Try mount from client

#sudo apt install nfs-common
#mount -t nfs {nfs server ip}:{nfs path} {local path}

a.txt on nfs server

Prepare K8s

I use K8s version 1.18.15
if you want newer version for K8s and would like to mount nfs please check this article https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner/issues/25

First you need to create ServiceAccount,Clusterrolebinding,Rolebinding follow yaml below.

kind: ServiceAccount
apiVersion: v1
metadata:
  name: nfs-pod-provisioner-sa
---
kind: ClusterRole # Role of kubernetes
apiVersion: rbac.authorization.k8s.io/v1 # auth API
metadata:
  name: nfs-provisioner-clusterRole
rules:
  - apiGroups: [""] # rules on persistentvolumes
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-provisioner-rolebinding
subjects:
  - kind: ServiceAccount
    name: nfs-pod-provisioner-sa # defined on top of file
    namespace: demo # NAMESPCAE
roleRef: # binding cluster role to service account
  kind: ClusterRole
  name: nfs-provisioner-clusterRole # name defined in clusterRole
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-pod-provisioner-otherRoles
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-pod-provisioner-otherRoles
subjects:
  - kind: ServiceAccount
    name: nfs-pod-provisioner-sa # same as top of the file
    # replace with namespace where provisioner is deployed
    namespace: demo # NAMESPACE
roleRef:
  kind: Role
  name: nfs-pod-provisioner-otherRoles
  apiGroup: rbac.authorization.k8s.io

apply yml for namespace you want to create nfs provisioner

#kubectl apply -f {file.yml} -n {namespace}

example apply nfs serviceaccount at namespace demo

Create storageclass yml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-storageclass # IMPORTANT pvc needs to mention this name
provisioner: nfs-test # name can be anything
parameters:
  archiveOnDelete: "false"

then apply storageclass to k8s — storage class not specify namespace

#kubectl apply -f {file.yml}

example storageclass

Create deployment file for nfs pod provisioner with nfs-client-provisioner:latest

https://quay.io/repository/external_storage/nfs-client-provisioner?tag=latest&tab=tags

kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-pod-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-pod-provisioner
  template:
    metadata:
      labels:
        app: nfs-pod-provisioner
    spec:
      serviceAccountName: nfs-pod-provisioner-sa # SERVICE ACCOUNT THAT CREATED AT FIRST
      containers:
        - name: nfs-pod-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-provisioner-v
              mountPath: /persistentvolumes # DO NOT CHANGE!!
          env:
            - name: PROVISIONER_NAME # DO NOT CHANGE!!
              value: nfs-test # SAME NAME AT STORAGECLASS DEFINE
            - name: NFS_SERVER # DO NOT CHANGE!!
              value: 172.16.22.155 # NFS SERVER IP
            - name: NFS_PATH # DO NOT CHANGE!!
              value: /mnt/nfs # NFS PATH AT NFS SERVER
      volumes:
       - name: nfs-provisioner-v # same as volumemouts name
         nfs:
           server: 172.16.22.155  # NFS SERVER IP
           path: /mnt/nfs

apply deployment file for create pod nfs provisioner

#kubectl apply -f {file.yml} -n demo

Check the pod can mount nfs

#kubectl exec -it {nfs_pod_name} sh -n demo

you will fine a.txt at /persistentvolumes/ on nfs pod that created.

Test Dynamic provisioning for Elasticsearch

We wil use helm to install elasticsearch (just very simple)
you can use this helm command to show values can edit from helm chart

#helm show values elastic/elasticsearch

Create elasticsearch yml for persistance volume claim
here is my es_demo.yml

---
clusterName: "elasticsearch"
nodeGroup: "master"# The service that non master groups will try to connect to when joining the cluster
# This should be set to clusterName + "-" + nodeGroup for your master group
masterService: ""# Elasticsearch roles that will be applied to this nodeGroup
# These will be set as environment variables. E.g. node.master=true
roles:
  master: "true"
  ingest: "true"
  data: "true"
  remote_cluster_client: "true"
  ml: "true"replicas: 1
minimumMasterNodes: 1volumeClaimTemplate:
  storageClassName: "nfs-storageclass" # SAME NAME AS THE STORAGECLASS
  accessModes:
    - ReadWriteMany #  must be the same as PersistentVolume
  resources:
    requests:
      storage: 200Giservice:
  labels: {}
  labelsHeadless: {}
  type: NodePort
  nodePort: "31122"
  annotations: {}
  httpPortName: http
  transportPortName: transport
  loadBalancerIP: ""
  loadBalancerSourceRanges: []
  externalTrafficPolicy: ""

Then helm install elasticsearch

#helm install {name} --values es_demo.yml elastic/elasticsearch -n demo

you can see log for provisioner pod by following command

#kubectl logs -f {pod name} -n demo

result should create directory on NFS server for elasticsearch

DONE~