Share volumes across pods and zones on GKE

Article

Steve Lubars · Sep 9 8m read

#Cloud #Google Cloud Platform (GCP) #Kubernetes #Mirroring #InterSystems IRIS

Background

For a variety of reasons, users may wish to mount a persistent volume on two or more pods spanning multiple availability zones. One such use case is to make data stored outside of IRIS available to both mirror members in case of failover.

Unfortunately the built-in storage classes in most Kubernetes implementations (whether cloud or on-prem) do not provide this capability:

Does not support access mode "ReadWriteMany"
Does not support being mounted on more than one pod at a time
Does not support access across availability zones

However, some Kubernetes add-ons (both provider and third-party) do provide this capability. The one we'll be looking at in this article is Google FIlestore.

Overview

In this article we will:

Create a Kubernetes cluster on GKE (Google Kubernetes Engine)
Use Google Filestore to create a persistent volume of type ReadWriteMany
Use IKO to deploy an IRIS failover mirror spanning two availability zones
Mount the persistent volume on both mirror members
Demonstrate that both mirror members have read/write access to the volume

Steps

The following steps were all carried out using Google Cloud Shell. Please note that InterSystems is not responsible for any costs incurred in the following examples.

We will be using region "us-east1" and availability zones "us-east1-b" and "us-east1-c". Replace project "gke-filestore-demo" with your own.

Create a Network

Google Filestore requires creation of a custom network and subnet:

gcloud compute networks create vpc-multi-region --subnet-mode=custom

Create a Subnet

gcloud compute networks subnets create subnet-us-east1 \
  --project=gke-filestore-demo \
  --region=us-east1 \
  --network=vpc-multi-region \
  --range=172.16.1.0/24 \
  --secondary-range=pods=10.0.0.0/16,services=192.168.1.0/24

Enable Google Services

gcloud services enable file.googleapis.com servicenetworking.googleapis.com

Create Addresses for VPC Peering

gcloud compute addresses create google-service-range \
    --project=gke-filestore-demo \
    --global \
    --purpose=VPC_PEERING \
    --prefix-length=20 \
    --description="Peering range for Google managed services" \
    --network=vpc-multi-region

Create VPC Peering

gcloud services vpc-peerings connect \
    --project=gke-filestore-demo \
    --service=servicenetworking.googleapis.com \
    --ranges=google-service-range \
    --network=vpc-multi-region

Create Kubernetes Cluster

gcloud container clusters create sample-cluster \ --project=gke-filestore-demo \ --region us-east1 \ --node-locations us-east1-b,us-east1-c \ --machine-type t2d-standard-2 --num-nodes=3 \ --release-channel rapid \ --enable-ip-alias \ --network=vpc-multi-region \ --subnetwork=subnet-us-east1 \ --cluster-secondary-range-name=pods \ --services-secondary-range-name=services \ --shielded-secure-boot \ --shielded-integrity-monitoring \ --enable-autoscaling \ --addons=GcpFilestoreCsiDrive

Create a StorageClass

Google provides a number of storage classes, but none of them will work with our custom network, so we must create our own. Add the following to a file named gke-filestore-sc.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gke-filestore-sc
provisioner: filestore.csi.storage.gke.io
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
  tier: ENTERPRISE
  network: vpc-multi-region
allowedTopologies:
- matchLabelExpressions:
  - key: topology.gke.io/zone
    values:
    - us-east1

Note that the value under key "toppology.gke.io/zone" is a region, not a zone.

Now create the storage class:

kubectl apply -f gke-filestore-sc.yaml

Create a PersistentVolumeClaim

Add the following to a file named gke-filestore-pvc.yaml:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: gke-filestore-pvc
spec:
  accessModes:
  - ReadWriteMany
  storageClassName: gke-filestore-sc
  resources:
    requests:
      storage: 1Ti

Note that 1TB is the minimum size allowed.

Now create the persistent volume claim:

kubectl apply -f gke-filestore-pvc.yaml

Install IKO

See IKO documentation for information on how to download and configure IKO:

helm install sample iris_operator_amd-3.8.42.100/chart/iris-operator

Create an IrisCluster

Add the following to a file named iris-filestore-demo.yaml:

apiVersion: intersystems.com/v1alpha1
kind: IrisCluster
metadata:
  name: sample
spec:
  storageClassName: iris-ssd-storageclass
  licenseKeySecret:
    name: iris-key-secret
  imagePullSecrets:
    - name: dockerhub-secret
  volumes:
  - name: nfs-volume
    persistentVolumeClaim:
      claimName: gke-filestore-pvc
  topology:
    data:
      image: containers.intersystems.com/intersystems/iris:2025.2
      preferredZones: ["us-east1-b","us-east1-c"]
      mirrored: true
      volumeMounts:
      - name: nfs-volume
        mountPath: "/mnt/nfs"
      podTemplate:
        spec:
          initContainers:
          - name: nfs-init
            image: busybox
            command: ["sh","-c","/bin/chown -R 51773:51773 /mnt/nfs; /bin/chmod -R 777 /mnt/nfs"]
            securityContext:
              runAsUser: 0
              runAsGroup: 0
              runAsNonRoot: false
            volumeMounts:
            - name: nfs-volume
              mountPath: "/mnt/nfs"

Notes:

We use an init container to make the volume writable by user "irisowner"
The mirror spans both availability zones in our cluster
See IKO documentation for information on how to configure an IrisCluster

Now create the IrisCluster:

kubectl apply -f iris-filestore-demo.yaml

Note that it may take some time for the shared volume to be allocated, and you may see errors such as the following in the events log:

15s  Warning  ProvisioningFailed  PersistentVolumeClaim/nfs-pvc  failed to provision volume with StorageClass "gke-filestore-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded

A better way to confirm that allocation is underway is to look directly at the filestore instance:

$ gcloud filestore instances list
INSTANCE_NAME: pvc-6d59f5ce-7d4f-42d0-8a20-2ee3602f8d32
LOCATION: us-east1
TIER: ENTERPRISE
CAPACITY_GB: 1024
FILE_SHARE_NAME: vol1
IP_ADDRESS: 10.110.51.194
STATE: CREATING
CREATE_TIME: 2025-09-09T03:13:07

Note the status of "CREATING"; deployment won't proceed until the state reaches "READY". This may take between five and ten minutes. Soon after that you should see the IrisCluster is up and running:

$ kubectl get pod,pv,pvc
NAME                 READY  STATUS   RESTARTS  AGE
pod/sample-data-0-0  1/1    Running  0         9m34s
pod/sample-data-0-1  1/1    Running  0         91s

NAME              CAPACITY  ACCESS MODES  STATUS   CLAIM                      STORAGECLASS
pvc-bbdb986fba54   1Ti       RWX           Bound    gke-filestore-pvc          gke-filestore-sc
pvc-9f5cce1010a3   4Gi       RWO           Bound    iris-data-sample-data-0-0  iris-ssd-storageclass
pvc-5e27165fbe5b   4Gi       RWO           Bound    iris-data-sample-data-0-1  iris-ssd-storageclass

NAME                      STATUS  VOLUME            CAPACITY  ACCESS MODES  STORAGECLASS            
gke-filestore-pvc          Bound   pvc-bbdb986fba54  1Ti       RWX           gke-filestore-sc
iris-data-sample-data-0-0  Bound   pvc-9f5cce1010a3  4Gi       RWO           iris-ssd-storageclass
iris-data-sample-data-0-1  Bound   pvc-5e27165fbe5b  4Gi       RWO           iris-ssd-storageclass

We can also (by joining the output of "kubectl get pod" with "kubectl get node") see that the mirror members reside in different availability zones:

sample-data-0-0 gke-sample-cluster-default-pool-a3b1683a-7g77   us-east1-c
sample-data-0-1 gke-sample-cluster-default-pool-010e420a-dw1h   us-east1-b

Test the shared volume

We can create files on the shared volume on each pod:

kubectl exec sample-data-0-0 -- touch /mnt/nfs/primary.txt
kubectl exec sample-data-0-1 -- touch /mnt/nfs/backup.txt

And then observe that files are visible from both pods:

$ kubectl exec sample-data-0-0 -- ls /mnt/nfs
primary.txt
backup.txt
$ kubectl exec sample-data-0-1 -- ls /mnt/nfs
primary.txt
backup.txt

Cleanup

Delete IrisCluster deployment

kubectl delete -f iris-filestore-demo.yaml --ignore-not-found
helm uninstall sample --ignore-not-found

Delete Persistent Volumes

kubectl delete pvc gke-filestore-pvc iris-data-sample-data-0-0 iris-data-sample-data-0-1 --ignore-not-found

Note that deleting PersistentVolumeClaim triggers deletion of the corresponding PersistentVolume.

Delete Kubernetes Cluster

gcloud container clusters delete sample-cluster --region us-east1 --quiet

Delete Google Filestore resources

gcloud services vpc-peerings delete --network=vpc-multi-region --quiet gcloud compute addresses delete google-service-range --global --quiet gcloud services disable servicenetworking.googleapis.com file.googleapis.com --quiet gcloud compute networks subnets delete subnet-us-east1 --quiet gcloud compute networks delete vpc-multi-region --quiet

Conclusion

We demonstrated how Google Filestore can be used to mount read/write volumes on pods residing in different availability zones. Several other solutions are available both for GKE and for other cloud providers. As you can see, their configuration can be highly esoteric and vendor-specific, but once working can be reliable and effective.