Creating a cross-regional DR Async with IKO

Regardless of whether an instance of IRIS is in the cloud or not, high availability and disaster recovery are always important considerations. While IKO already allows for the use of NodeSelectors to enforce the scheduling of IRISCluster nodes across multiple zones, multi-region k8s clusters are generally not recommended or even supported in the major CSP's managed Kubernetes solutions. However, when discussing HA and DR for IRIS, we may want to have an async member in a completely separate region, or even in a different cloud provider altogether. With additional options added to IKO in the development of version 3.8 (specifically 3.8.4+), it's now possible to create this kind of multi-region HA architecture.

In this post we will:

• Create two different Kubernetes clusters in entirely separate Regions.
• Configure both Kubernetes clusters to allow for bi-directional communication
• Configure internal Loadbalancers to connect our IRISCluster nodes
• Create and mirror pair and DR Async IKO IRISCluster, one in each Kubernetes cluster
• Verify the Mirror Status of each node and prove that a simple global is accessible on the DR Async node

---

Previous Work

It should be noted that we have previously experimented with several tools for creating remote DR Async members through IKO and with K8s.

1) We've attempted cross--cluster communication schemes using Calico/Tigera and Istio. In both cases the solutions were overly complicated and did not end up meeting our success criteria 

2) We've used external LoadBalancer to expose mirror members to each other over the internet. While effective, the solution we will cover here allows for much greater security, and doesn't expose our IRIS instances to the internet at large in any capacity.

 

---

Prerequisites 

In order to replicate the following work, you will need the following tools

• AWS CLI
• EKCTL
• Helm

---

Creating and Installing Dependencies on EKS Kubernetes Clusters

It's fairly straightforward to create EKS Clusters through the EKSCTL tool.

• First create two files, cluster1.yaml and cluster2.yaml

cluster1.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: sferguso-cross1
  region: us-east-1
  version: "1.27"
kubernetesNetworkConfig:
  ipFamily: "IPv4"
  serviceIPv4CIDR: "10.100.0.0/16"
vpc:
  cidr: "192.168.0.0/16"
nodeGroups:
  - name: ng-1
    instanceType: m5.large
    desiredCapacity: 5
    volumeSize: 80

cluster2.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: sferguso-cross2
  region: us-west-2
  version: "1.27"
kubernetesNetworkConfig:
  ipFamily: "IPv4"
  serviceIPv4CIDR: "10.200.0.0/16"
vpc:
  cidr: "172.16.0.0/16"
nodeGroups:
  - name: ng-1
    instanceType: m5.large
    desiredCapacity: 5
    volumeSize: 80

• Secondly, use EKSCTL to create the clusters. (This will take some time)

eksctl create cluster -f cluster1.yaml
eksctl create cluster -f cluster2.yaml

Note in the above configurations, that the regions are us-east-1 and us-west-2. This will result in each Kubernetes cluster spanning 2 Availability Zones by default (which generally are their own independent data centers), but in entirely separate regions for the greatest possible fault tolerance.

Note as well that the CIDRS both for the kubernetesNetworkConfigs and the VPCs are all unique. This allows us to be certain exactly where traffic is originating from. At the very least, the VPC CIDRs must be different to ensure no overlap for later steps.

• Next we need to install our Storage CSI Driver.

If your AWS permissions are wide enough, its possible to simply run the following commands to install EBS onto the two k8s clusters 

eksctl utils associate-iam-oidc-provider --region=us-east-1 --cluster=sferguso-cross1 --approve
eksctl create iamserviceaccount --name ebs-csi-controller-sa --namespace kube-system --cluster sferguso-cross1 --role-name AmazonEKS_EBS_CSI_DriverRole_sferguso-cross1 --role-only --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy --approve
eksctl create addon --name aws-ebs-csi-driver --cluster sferguso-cross1 --service-account-role-arn arn:aws:iam::<Your_Account_ID>:role/AmazonEKS_EBS_CSI_DriverRole_sferguso-cross1 --force
 
eksctl utils associate-iam-oidc-provider --region=us-west-2 --cluster=sferguso-cross2 --approve
eksctl create iamserviceaccount --name ebs-csi-controller-sa --namespace kube-system --cluster sferguso-cross2 --region us-west-2 --role-name AmazonEKS_EBS_CSI_DriverRole_sferguso-cross2 --role-only --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy --approve
eksctl create addon --name aws-ebs-csi-driver --cluster sferguso-cross2 --region=us-west-2 --service-account-role-arn arn:aws:iam::<Your_Account_ID>:role/AmazonEKS_EBS_CSI_DriverRole_sferguso-cross2 --force

If you do not have IAM permissions for these commands (true for all ISC AWS accounts), you can create the IAM Roles by hand as well. This requires you to make two new Roles (AmazonEKS_EBS_CSI_DriverRole_sferguso-cross1 and AmazonEKS_EBS_CSI_DriverRole_sferguso-cross2) in the IAM Console from the AWS Managed AmazonEBSCSIDriverPolicy and a policy that looks like the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<Your_Account_ID>:oidc-provider/oidc.eks.<region>.amazonaws.com/id/<EKS_OIDC>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.<region>.amazonaws.com/id/<EKS_OIDC>:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa",
          "oidc.eks.<region>.amazonaws.com/id/<EKS_OIDC>:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

Note that you can find the <EKS_OIDC> in the AWS Console in the Overview tab of your EKS Cluster, and <Your_Account_ID> in the top right dropdown of the Console.

• Then we must install IKO

Follow the instructions at https://docs.intersystems.com/components/csp/docbook/DocBook.UI.Page.cls?KEY=AIKO36#AIKO36_install

Important: Before running helm install, navigate to the IKO chart values.yaml and make sure to set FQDN to false. The rest of the exercise will not work without this setting

useFQDN: false

---

Create a Peering Connection between the two EKS created VPCs

Follow instructions at https://docs.aws.amazon.com/vpc/latest/peering/create-vpc-peering-connection.html#same-account-different-region
 

Additionally follow the instructions at https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-routing.html to create routes for traffic to move through the peering connection

This should result in a ingress route for every RouteTable like the following:

${cidr_block_othercluster}

${vpc_peering_id}

 

---

Create Internal LoadBalancers 

Now that the two EKS Clusters are connected, we need to expose K8s services that will associate themselves with our IKO defined IRISClusters. These internal load balancers have DNS names that only resolve within either of the VPCs created by EKS.
 

In the Cluster you will use to create an IKO mirror pair apply the following:

mirror-svcs.yaml

apiVersion: v1
kind: Service
metadata:
  name: primary-svc
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
  type: LoadBalancer
  ports:
  - port: 1972
    protocol: TCP
    name: superserver
  - port: 2188
    protocol: TCP
    name: iscagent
  selector:
    statefulset.kubernetes.io/pod-name: crossover-data-0-0
---
apiVersion: v1
kind: Service
metadata:
  name: backup-svc
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
  type: LoadBalancer
  ports:
  - port: 1972
    protocol: TCP
    name: superserver
  - port: 2188
    protocol: TCP
    name: iscagent
  selector:
    statefulset.kubernetes.io/pod-name: crossover-data-0-1

In the Cluster you will use to create an IKO async member apply the following:

async-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: async-svc
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
  type: LoadBalancer
  ports:
  - port: 1972
    protocol: TCP
    name: superserver
  - port: 2188
    protocol: TCP
    name: iscagent
  selector:
    statefulset.kubernetes.io/pod-name: crossover-data-0-100

You can apply these services with the following command (note that I've chosen my us-east-1 cluster to contain my mirror pair, and us-west-2 to contain my async dr member)

kubectl --context <east> apply -f mirror-svcs.yaml
kubectl --context <west> apply -f async-svc.yaml

• Once These Internal LoadBalancer services come up, create ExternalName services that map cross-cluster.

In the Cluster you will use to create an IKO mirror pair apply the following:

mirror-cross-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: crossover-data-0-100
spec:
  type: ExternalName
  externalName: "a74fcb8ce93d94781ae4e0ea9d4d1c41-a8ec6876e5f7d747.elb.us-west-2.amazonaws.com" # Example Host Name of async-svc
  ports:
  - port: 1972
    name: "superserver"
  - port: 2188
    name: "iscagent"

In the Cluster you will use to create an IKO async member apply the following:

async-cross-svcs.yaml

apiVersion: v1
kind: Service
metadata:
  name: crossover-data-0-1
spec:
  type: ExternalName
  externalName: "a59d4f5a0c102479a9d48854da1f0d87-b27eeccb1044a9da.elb.us-east-1.amazonaws.com" # Example Host Name of backup-svc
  ports:
  - port: 1972
    name: "superserver"
  - port: 2188
    name: "iscagent"
---
apiVersion: v1
kind: Service
metadata:
  name: crossover-data-0-0
spec:
  type: ExternalName
  externalName: "a59d4f5a0c102479a9d48854da1f0d87-b27eeccb1044a9da.elb.us-east-1.amazonaws.com" # Example Host Name of primary-svc
  ports:
  - port: 1972
    name: "superserver"
  - port: 2188
    name: "iscagent"

Again, you can apply these with the following style command:

kubectl --context <east> apply -f mirror-cross-svc.yaml
kubectl --context <west> apply -f async-cross-svcs.yaml

Once all these service are running, we're ready to create the matching IKO IRISClusters. Note that the ExternalName services use the expected naming scheme of the IRISClusters we'll make in a second. This is why disabling FQDN is so important when we initially installed  IKO, as without that setting off, IKO will not create a routable mirroring configuration for the cluster using these services. In this way, the Primary and Failover member will route to the Async member over the ExternalName service (which references the associated Internal Loadbalancer in the other VPC) and vice-versa.

---

Create Mirrored IRISCluster and Async IRISCluster

For the following I created an AWS ECR (sferguso-cross) to push my IRIS images too. EKS automatically has pull access to any ECR in the same account. You can either do the same, or use an IKO ImagePullSecret to source IRIS images from containers.intersystems.com
 

In the Cluster for the IKO mirror pair apply the following:

iris-mirror.yaml

apiVersion: intersystems.com/v1alpha1
kind: IrisCluster
metadata:
  name: crossover
spec:
  licenseKeySecret:
    name: iris-key-secret
  configSource:
    name: my-iriscluster-config
  tls:
    common:
      secret:
        secretName: common-certs
  topology:
    data:
      image: <Your_Account_ID>.dkr.ecr.us-east-1.amazonaws.com/sferguso-cross:iris-2024.1.0L.217.0
      compatibilityVersion: "2024.1.0"
      mirrored: true

In the Cluster for the Async Member apply the following:

iris-async.yaml

apiVersion: intersystems.com/v1alpha1
kind: IrisCluster
metadata:
  name: crossover
spec:
  licenseKeySecret:
    name: iris-key-secret
  configSource:
    name: my-iriscluster-config
  tls:
    common:
      secret:
        secretName: common-certs
  topology:
    data:
      image: <Your_Account_ID>.dkr.ecr.us-east-1.amazonaws.com/sferguso-cross:iris-2024.1.0L.217.0
      compatibilityVersion: "2024.1.0"
      mirrored: true
      mirrorMap: drasync
      start: 100

And as usual, apply these to the clusters through kubectl

kubectl --context <east> apply -f iris-mirror.yaml
kubectl --context <west> apply -f iris-async.yaml

---

Verification

In the Cluster for the IKO mirror pair
 

$ kubectl exec crossover-data-0-0 -- iris qlist
IRIS^/usr/irissys^2024.1.0L.208.0^running, since Thu Jan 18 16:43:48 2024^iris.cpf^1972^0^0^ok^IRIS^Failover^Primary^/irissys/data/IRIS/^1972

$ kubectl exec crossover-data-0-1 -- iris qlist
IRIS^/usr/irissys^2024.1.0L.208.0^running, since Thu Jan 18 16:44:27 2024^iris.cpf^1972^0^0^ok^IRIS^Failover^Backup^/irissys/data/IRIS/^1972

$ kubectl exec -it crossover-data-0-0 -- iris session IRIS -UIRISCLUSTER
IRISCLUSTER>s ^result="Success!"
$ kubectl exec -it crossover-data-0-1 -- iris session IRIS -UIRISCLUSTER
IRISCLUSTER>w ^result
Success!

In the Cluster for the Async Member

$ kubectl exec crossover-data-0-100 -- iris qlist
IRIS^/usr/irissys^2024.1.0L.208.0^running, since Thu Jan 18 17:07:23 2024^iris.cpf^1972^0^0^ok^IRIS^Disaster Recovery^Connected^/irissys/data/IRIS/^1972

$ kubectl exec -it crossover-data-0-100 -- iris session IRIS -UIRISCLUSTER
IRISCLUSTER>w ^result
Success!

Huzzah! We officially have an Async DR node that is running on the opposite coast from its other mirror members.