Share volumes across pods and zones on EKS

Article

Steve Lubars · Sep 22 10m read

#AWS #Cloud #Kubernetes #Mirroring #InterSystems IRIS #InterSystems Kubernetes Operator (IKO)

Background

For a variety of reasons, users may wish to mount a persistent volume on two or more pods spanning multiple availability zones. One such use case is to make data stored outside of IRIS available to both mirror members in case of failover.

Unfortunately the built-in storage classes in most Kubernetes implementations (whether cloud or on-prem) do not provide this capability:

Does not support access mode "ReadWriteMany"
Does not support being mounted on more than one pod at a time
Does not support access across availability zones

However, some Kubernetes add-ons (both provider and third-party) do provide this capability. The one we'll be looking at in this article is Amazon Elastic File System (EFS).

Overview

In this article we will:

Create a Kubernetes cluster on Amazon EKS (Elastic Kubernetes Service)
Use EFS to create a persistent volume of type ReadWriteMany
Use IKO to deploy an IRIS failover mirror spanning two availability zones
Mount the persistent volume on both mirror members
Demonstrate that both mirror members have read/write access to the volume

Steps

The following steps were all carried out using AWS CloudShell. Please note that InterSystems is not responsible for any costs incurred in the following examples.

We will be using region "us-east-2" and availability zones "us-east-2b" and "us-east-2c".

Create Kubernetes Cluster

export AWS_REGION=us-east-2 export CLUSTER=sample eksctl create cluster \ --name $CLUSTER \ --region $AWS_REGION \ --zones us-east-2b,us-east-2c \ --node-type m5.2xlarge \ --nodes 3

Configure EBS and EFS

export AWS_ID=$(aws sts get-caller-identity --query Account --output text)
export EBS_ROLE=AmazonEKS_EBS_CSI_DriverRole_$CLUSTER
export EFS_ROLE=AmazonEKS_EFS_CSI_DriverRole_$CLUSTER

eksctl utils associate-iam-oidc-provider \
  --cluster $CLUSTER  \
  --region $AWS_REGION \
  --approve

aws eks create-addon \
  --addon-name aws-ebs-csi-driver \
  --cluster-name $CLUSTER \
  --region $AWS_REGION \
  --service-account-role-arn arn:aws:iam::${AWS_ID}:role/${EBS_ROLE} \
  --configuration-values '{"defaultStorageClass":{"enabled":true}}'

eksctl create addon \
  --name aws-efs-csi-driver \
  --cluster $CLUSTER \
  --region=$AWS_REGION \
  --service-account-role-arn arn:aws:iam::$AWS_ID:role/$EFS_ROLE \
  --force

eksctl create addon \
  --name=eks-pod-identity-agent \
  --cluster=$CLUSTER

export ADDONS=$(aws eks list-addons --cluster-name $CLUSTER --query addons[] --output text)
for ADDON in $ADDONS; do
  eksctl update addon \
    --name $ADDON \
    --cluster $CLUSTER \
    --region $AWS_REGION
  done

eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster $CLUSTER \
  --region $AWS_REGION \
  --role-name $EBS_ROLE \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve \
  --override-existing-serviceaccounts

eksctl create iamserviceaccount \
  --name efs-csi-controller-sa \
  --namespace kube-system \
  --cluster $CLUSTER \
  --region $AWS_REGION \
  --role-name $EFS_ROLE \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEFSCSIDriverPolicy \
  --approve \
  --override-existing-serviceaccounts

Configure Security and Ingress

We create a Security Group and configure ingress to EFS port 2049 (NFS):

export VPC_ID=$(aws eks describe-cluster --name $CLUSTER --query "cluster.resourcesVpcConfig.vpcId" --output text)

export SG=$(aws ec2 create-security-group \
              --description efs-sample-sg \
              --group-name efs-sg \
              --vpc-id $VPC_ID \
              --query "GroupId" \
              --output text)

export VPC_CIDR=$(aws ec2 describe-vpcs --vpc-ids $VPC_ID --query "Vpcs[].CidrBlock" --output text)

aws ec2 authorize-security-group-ingress \
  --group-id $SG \
  --protocol tcp \
  --port 2049 \
  --cidr $VPC_CIDR

Create a File System

The File System routes traffic from the PersistentVolume in each zone to the shared file store.

export FS_ID=$(aws efs create-file-system \
                  --region $AWS_REGION \
                  --performance-mode generalPurpose \
                  --query 'FileSystemId' \
                  --output text)

Each File System needs an Access Point. We set user and group to 51773 ("irisowner") and provide access to the entire volume ("/"). Note that changing ownership requires root access by EFS ("Uid=0,Gid=0"):

export ACCESS_POINT=$(aws efs create-access-point \
                        --file-system-id $FS_ID \
                        --root-directory "Path=/,CreationInfo={OwnerUid=51773,OwnerGid=51773,Permissions=777}" \
                        --posix-user "Uid=0,Gid=0" \
                        --tags Key=Name,Value=east-users \
                        --query "AccessPointId" \
                        --output text)

Each File System also needs a Mount Target in the subnet of each availability zone. Each Mount Target has an IP address that routes to the local PersistentVolume:

export SUBNET_IDS=$(aws eks describe-cluster --name $CLUSTER --query "cluster.resourcesVpcConfig.subnetIds" --output text)
for SUBNET_ID in $SUBNET_IDS; do
  aws efs create-mount-target \
    --file-system-id $FS_ID \
    --subnet-id $SUBNET_ID \
    --security-group $SG
done

Create a StorageClass

Add the following to a file named "efs-sc.yaml":

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com

Now create the storage class:

kubectl apply -f efs-sc.yaml

Create a PersistentVolume

Determine the Volume Handle for the File System:

echo $FS_ID::$ACCESS_POINT
fs-0e67f9ac9a3ba51cd::fsap-02c3ed5dc9233394f    // <-- example only, do not use

Add the following to a file named "efs-pv.yaml". Replace the volumeHandle field below with your own:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv
spec:
  capacity:
    storage: 5Gi
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-0e67f9ac9a3ba51cd::fsap-02c3ed5dc9233394f
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  volumeMode: Filesystem

Now create the persistent volume:

kubectl apply -f efs-pv.yaml

Create a PersistentVolumeClaim

Add the following to a file named "efs-pvc.yaml":

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi

Now create the persistent volume claim:

kubectl apply -f efs-pvc.yaml

Install IKO

Install and run IKO:

helm install sample iris_operator_amd-3.8.42.100/chart/iris-operator

See IKO documentation for additional information on how to download and configure IKO.

Create an IrisCluster

Add the following to a file named iris-efs-demo.yaml:

apiVersion: intersystems.com/v1beta1
kind: IrisCluster
metadata:
  name: sample
spec:
  storageClassName: iris-ssd-storageclass
  licenseKeySecret:
    name: iris-key-secret
  imagePullSecrets:
    - name: dockerhub-secret
  volumes:
  - name: efs-volume
    persistentVolumeClaim:
      claimName: efs-pvc
  topology:
    data:
      image: containers.intersystems.com/intersystems/iris:2025.2
      preferredZones: ["us-east-2a","us-east-2b"]
      mirrored: true
      volumeMounts:
      - name: efs-volume
        mountPath: "/mnt/nfs"

Notes:

The mirror spans both availability zones in our cluster
See IKO documentation for information on how to configure an IrisCluster

Now create the IrisCluster:

kubectl apply -f iris-efs-demo.yaml

Soon after that you should see the IrisCluster is up and running:

$ kubectl get pod,pv,pvc
NAME                 READY  STATUS   RESTARTS  AGE
pod/sample-data-0-0  1/1    Running  0         9m34s
pod/sample-data-0-1  1/1    Running  0         91s

NAME              CAPACITY  ACCESS MODES  STATUS   CLAIM                      STORAGECLASS
pvc-bbdb986fba54   5Gi       RWX           Bound    efs-pvc                    efs-sc
pvc-9f5cce1010a3   4Gi       RWO           Bound    iris-data-sample-data-0-0  iris-ssd-storageclass
pvc-5e27165fbe5b   4Gi       RWO           Bound    iris-data-sample-data-0-1  iris-ssd-storageclass

NAME                      STATUS  VOLUME            CAPACITY  ACCESS MODES  STORAGECLASS            
efs-pvc                    Bound   pvc-bbdb986fba54  5Gi       RWX           efs-sc
iris-data-sample-data-0-0  Bound   pvc-9f5cce1010a3  4Gi       RWO           iris-ssd-storageclass
iris-data-sample-data-0-1  Bound   pvc-5e27165fbe5b  4Gi       RWO           iris-ssd-storageclass

We can also (by joining the output of "kubectl get pod" with "kubectl get node") see that the mirror members reside in different availability zones:

sample-data-0-0 ip-192-168-18-38.us-east-2.compute.internal   us-east-2b
sample-data-0-1 ip-192-168-52-17.us-east-2.compute.internal   us-east-2c

Test the shared volume

We can create files on the shared volume on each pod:

kubectl exec sample-data-0-0 -- touch /mnt/nfs/primary.txt
kubectl exec sample-data-0-1 -- touch /mnt/nfs/backup.txt

And then observe that files are visible from both pods:

$ kubectl exec sample-data-0-0 -- ls /mnt/nfs
primary.txt
backup.txt
$ kubectl exec sample-data-0-1 -- ls /mnt/nfs
primary.txt
backup.txt

Cleanup

Delete IrisCluster deployment

kubectl delete -f iris-efs-demo.yaml --ignore-not-found
helm uninstall sample --ignore-not-found

Delete Persistent Volumes

kubectl delete pvc efs-pvc iris-data-sample-data-0-0 iris-data-sample-data-0-1 --ignore-not-found

Note that deleting PersistentVolumeClaim triggers deletion of the corresponding PersistentVolume.

Delete EFS resources

export ACCESS_POINTS=$(aws efs describe-access-points --file-system-id $FS_ID --query "AccessPoints[].AccessPointId" --output text)
for ACCESS_POINT in $ACCESS_POINTS; do
    aws efs delete-access-point \
        --access-point-id $ACCESS_POINT
done

export MOUNT_TARGETS=$(aws efs describe-mount-targets --file-system-id $FS_ID --query "MountTargets[].MountTargetId" --output text)
for MOUNT_TARGET in $MOUNT_TARGETS; do
    aws efs delete-mount-target \
        --mount-target-id $MOUNT_TARGET
done

aws efs delete-file-system --file-system-id $FS_ID

Delete more resources

aws iam detach-role-policy \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --role-name $EBS_ROLE

aws iam delete-role \
    --role-name $EBS_ROLE

aws iam detach-role-policy \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEFSCSIDriverPolicy \
  --role-name $EFS_ROLE

aws iam delete-role \
    --role-name $EFS_ROLE

ADDONS=$(eksctl get addon --cluster $CLUSTER --region $AWS_REGION --output json | grep Name | cut -d '"' -f4 | xargs echo)
for ADDON in $ADDONS; do
  aws eks delete-addon \
    --addon-name $ADDON \
    --cluster-name $CLUSTER
  done

aws ec2 revoke-security-group-ingress \
  --group-id $SG

aws ec2 delete-security-group \
  --group-id $SG

Delete Kubernetes Cluster

eksctl delete cluster --name $CLUSTER

Conclusion

We demonstrated how Amazon EFS can be used to mount read/write volumes on pods residing in different availability zones. Several other solutions are available both for AWS and for other cloud providers. As you can see, their configuration can be highly esoteric and vendor-specific, but once working can be reliable and effective.