eBPF - Cilium on FHIR® - A Star Wars Story

Article

Ron Sweeney · Sep 19 6m read

#FHIR #Kubernetes #Security #InterSystems IRIS for Health

Anakin Skywalker challenged the high ground and has been terribly injured on Mustafar.

He is a relatively new employee of the Galatic Empire, covered by a Large Group Planetary Plan and now has an active encounter in progress for emergent medical services in the Grand Medical Facility on Coruscant. The EMR deployed for the Galactic Health System is Powered by InterSystems FHIR Server running on Kubernetes protected by Cilium.

Let's recreate the technical landscape, to be performed in the style of Isovalent Labs...

Kind Cluster

Lets fire up a 3 node cluster, and disable the CNI so we can replace it with Cilium.

cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker
networking:
  disableDefaultCNI: true
EOF

This will provision the kind cluster, 3 nodes wide with a single control plane.

Cilium

Cilium is an open-source project that provides networking, security, and observability for containerized environments like Kubernetes clusters. It uses a Linux kernel technology called eBPF (extended Berkeley Packet Filter) to inject security, networking, and observability logic into the kernel.

In other words, wields the force.

cilium install --version v1.16.0
cilium status --wait

Hubble

Hubble is a clown suit for Cilium, providing ridiculous visibility to what powers Cilium are in play in real time.

cilium enable hubble

InterSystems FHIR Workload

InterSystems is the GOAT of interoperability, and transforms Healthcare Data like a protocol Droid.

kubectl apply -f https://raw.githubusercontent.com/sween/basenube/main/scenarios/ciliumfhir/deploy/cilium-fhir-starwars.yaml

The resulting workload has 4 deployments:

GrandMedicalFacility

Integrated Delivery Network based in Coruscant, with facilities as far as the Outer Rim, runs Epic and utilizes InterSystems I4H as a FHIR Server.

MedicalDroid FX-6

This 1.83-meter-tall droid supplied Vader with a blood transfusion and trained in cybernetic legs procedures.

MedicalDroid DD-13

Also known as the DD-13 tripedal medical droid, this droid has three legs for stability and was designed to install cybernetic implants.

MedicalDroid 2-1B

2-1B droids have hypodermic injectors and precision-crafted servogrip pincers, and can be upgraded to specialize in areas like cybernetic limb replacement, neurosurgery, and alien biology.

Since we will need it anyway for upcoming interviews, lets tell the story in true STAR (Sithuation, Task, Action, Result) methodology.

Sith-uation

Palpatine accompanied the fallen jedi to the facility, and upon arrival helped registration admit him as Darth Vader.

cat > vader.json << 'EOF'
{
  "name": [
    {
      "use": "official",
      "family": "Vader",
      "given": [
        "Darth"
      ]
    }
  ],
  "gender": "male",
  "id": "DarthVader",
  "birthDate": "1977-05-25",
  "resourceType": "Patient"
}
EOF
curl -v -X PUT \
  -H "Content-Type: application/fhir+json" \
  -d @vader.json \
  "http://coruscanthealth:52773/intersystems/fhir/r5/Patient/DarthVader"

Darth Vader is now registered, and can be seen throughout the Health System...

Galactic IT Outage

There is a problem though!

Shortly after registration, a Galactic IT Outage has occured, making the Identity Provider for the Health System unavailable. The InterSystems FHIR Resource Server is SMART enabled, and the IDP is casters up, making EMR launches impossible with the absence of the jwt token with the applicable scopes to protect the routes.

Sure as Bantha Fodder, we definitely have a problem... the care team cannot access the patient record, nothing but 401's and 403's and were not talking about your Galactic Retirement Plan.

Although the Hubble UI only provides a hint to what is going on, Inspecting the Hubble flows with Layer 7 information reveals the sithuation.

...and after adding some debugs bunny to the InterSystems FHIR endpoint, confirms it.

FHIR Debug

zn "USER"
Set ^FSLogChannel("all")=1
zn "%SYS"
Set ^%ISCLOG=5
Set ^%ISCLOG("Category","HSFHIR")=5
Set ^%ISCLOG("Category","HSFHIRServer")=5
Set ^%ISCLOG("Category","OAuth2")=5
Set ^%ISCLOG("Category","OAuth2Server")=5

zw^FSLOG
...
^FSLOG(379555)="DispatchRequest^HS.FHIRServer.Service^944|Msg|Dispatch interaction read for Patient/DarthVader|09/19/2024 10:48:20.833339AM"
^FSLOG(379556)="DispatchRequest^HS.FHIRServer.Service^944|Msg|Request Completed in .000186 secs: Patient/DarthVader|09/19/2024 10:48:20.833450AM"
^FSLOG(379557)="processRequest^HS.FHIRServer.RestHandler^944|Msg|Response Status: 401, Json: Patient|09/19/2024 10:48:20.833454AM"
...

Task

Action

With the route enforcement from SMART not applicable, lets do this our way and use Cilium to protect the endpoints while Vader gets immediate attention the Emperor demands. We will go Rogue One here on the cluster and hand off the endpoint/route protection to Cilium while the Galaxy figures itself out from the outage.

Lets institute a deny all, from everywhere, with a CiliumClusterwideNetworkPolicy, and work backwards zero trust style.

cat <<EOF | kubectl apply -n galactic-fhir -f- 
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: "denyall-coruscanthealth"
spec:
  description: "Block all the traffic (except DNS) by default"
  egress:
  - toEndpoints:
    - matchLabels:
        io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: '53'
        protocol: UDP
      rules:
        dns:
        - matchPattern: '*'
  endpointSelector:
    matchExpressions:
    - key: io.kubernetes.pod.namespace
      operator: NotIn
      values:
      - kube-system
EOF

Looking good, Cilium dropping it like its hot!

Now, lets open up the FHIR endpoint on the intersystems pod, disabling the oauth2 client.

set app = "/intersystems/fhir/r5"
Set strategy = ##class(HS.FHIRServer.API.InteractionsStrategy).GetStrategyForEndpoint(app)
// 7 = Mass Openness
Set configData.DebugMode = 7
Set configData = strategy.GetServiceConfigData()
Do strategy.SaveServiceConfigData(configData)

Lastly, lets create a CiliumNetworkPolicy to allow anybody from the org:empire, access to the route for DarthVaders record in the galactic-fhir namespace.

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l7-visibility"
spec:
  endpointSelector:
    matchLabels:
      org: empire
  egress:
  - toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": galactic-fhir
    toPorts:
    - ports:
      - port: "52773"
        protocol: TCP
      rules:
        http:
          - method: "GET"
            path: "/intersystems/fhir/r5/Patient/DarthVader"
          - method: "HEAD"
            path: "/intersystems/fhir/r5/Patient/DarthVader"
EOF

Looks like we may be able to get back to iRacing, I think we are good.

...except

Yeah, looks like the payer is getting dropped...

Policy verdict = DROPPED

Let's add another policy, allowing org:payer access to Vaders route:

cat <<EOF | kubectl apply -n galactic-fhir -f- 
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l7-visibility-payer"
spec:
  endpointSelector:
    matchLabels:
      org: payer
  egress:
  - toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": galactic-fhir
    toPorts:
    - ports:
      - port: "52773"
        protocol: TCP
      rules:
        http:
          - method: "GET"
            path: "/intersystems/fhir/r5/Patient/DarthVader"
          - method: "HEAD"
            path: "/intersystems/fhir/r5/Patient/DarthVader"
EOF

Welp, that did not quite cut it, and we can see why.

So we gave the payer a call and told them to access the "correct" patient record, and Anakin Vader gets his legs.