IKO Plus: Stretched Cross Cloud, Cross Regional IrisCluster with Tailscale

Another step in this implementation path, adding cross cloud, cross regional stretched IrisCluster with Mirroring + Disaster Recovery using the Intersystems Kubernetes Operator (IKO) and Tailscale

Though trivial, Id like to go multi-cloud with the stretched IrisCluster for a couple of reasons to socialize the power of Wireguard when it supplies the network for a properly zoned IrisCluster by adding another mirror role to Amazon Web Services in the Western United States based datacenter in Oregon.

Lets stage the solution check boxes here for the solution and see if we can check them later after we make some changes to the IrisCluster topology.

Business Continuity calls for:

⬜  High Availability, Implements Mirror

⬜  Disaster Recovery Instance/Data must be 2500 miles (4000 kilometers) from the closest HA Mirror

⬜   Disaster Recovery Instance/Data must be hosted by an alternate vendor.

Let's Go

As a recap from the previous post, we already have a stretched cluster running on Google Cloud Platform.

 

Existing IrisCluster Compute

Create Compute

In the us-west-2 region on AWS, create an instance.
 

aws ec2 run-instances \
  --region us-west-2 \
  --instance-type c6i.xlarge \
  --image-id ami-03aa99ddf5498ceb9 \
  --block-device-mappings DeviceName=/dev/sda1,Ebs={VolumeSize=200} \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ikoplus-worker-aws}]'

Join Your Tailnet

Flatten your connectivity with Wireguard related technologies.

curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.tailscale-keyring.list | sudo tee /etc/apt/sources.list.d/tailscale.list
sudo apt-get update
sudo apt-get install tailscale
sudo tailscale up
tailscale ip -4

Inspect your Tailnet

Kubernetes Everbody

Done well and right with Canonical k8s.

# on control plane
sudo k8s get-join-token --worker

# on our new aws node
sudo k8s join-cluster eyJ0b2tlbiI6IiI.....

... Please allow some time for initial Kubernetes node registration, its coming over the pond so give it a break.

New Zone to the New Node

Add your new zone to the node...

kubectl label node ip-10-129-0-182.ec2.internal topology.kubernetes.io/zone=us-aws -n ikoplus

Update Stretched IrisCluster Toplogy

Two things to update here:

• add `asyncdr` to the mirrorMap
• add the new zone

Bam


Attestation

Mirror Status

Foot Guns

This did not come without a hitch,  and I was surprised as the GCP cross regional worked flawlessly outside of timesync adjustments...so I want to share a couple of things that got it over the hump here to go cross-region, cross-cloud.
 

AWS

• disabled DNS hostnames in the vpc, added host entries with tailscale ips
• tightened up polling for chronyd
• recreated the VPC to have no NAT gateway, only IGW

Instance

• had to enable ip forwarding

echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
sudo sysctl -p /etc/sysctl.d/99-tailscale.conf

Tailnet

On the tailnet I had to advertise the subnets on the boxes and accept the routes.

sudo tailscale up --advertise-routes=10.150.0.0/20 --accept-routes
sudo tailscale up --advertise-routes=10.154.0.0/20 --accept-routes
sudo tailscale up --advertise-routes=10.129.0.0/20 --accept-routes

Then in the tailnet, each machine needed an approval.

IKO

I have to work on the node automation and make sure they stick in the appropriate places after a delete and recreation of the stretched IrisCluster... by this I was comfortable with having the control plane in the midwest US,  and DR in Oregeon and HA in the UK, but after a delete and recreate the roles ended up in different places.  Similarly, I want to do something different with the arbiter as well.

We Had One Job

Lets check our Business Continuity requirements:

✅  High Availability, Implements Mirror ( Thanks IKO )

✅  Disaster Recovery Instance/Data must be 2500 miles (4000 kilometers) from the closest HA Mirror ( 2544 miles to next DC)

✅  Disaster Recovery Instance/Data must be hosted by an alternate vendor ( there are two cloud icons up there)

Onward and Upward, love the Tailnets!