New post
Article
· Oct 4 9m read

eBPF: Tetragon Security for IRIS Workloads

#Kubernetes #Monitoring #Security #InterSystems IRIS

Runtime Enforcement

 

So far in the eBPF Journey applied to InterSystems Workloads, we've been pretty much read only when it comes to system calls, binary execution, and file monitoring.  But much like the Network Security Policies that were in play with the last post that enforce connectivity,  what if we can enforce system calls, file access, and processes in the same manner across an entire cluster ?

Enter, Tetragon, a flexible Kubernetes-aware security observability and runtime enforcement tool that applies policy and filtering directly with eBPF, allowing for reduced observation overhead, tracking of any process, and real-time enforcement of policies.

Enforcement when your application cant provide it.

Where it Runs

Observability and Enforcement Cluster Wide

Up and Running

The obligatory steps to get up and running if you chose to do so, performed in the style of an Isovalent Lab.

kubernetes" Icon - Download for free – IconduckCluster

Kind cluster, 3 worker nodes wide, without a default CNI.

 
kind.sh

Cilium

Install Cilium, if for nother else, a CNI.

 
cilium.sh



Runtime EnforcementTetragon

Here we install the star of our show, Tetragon,  as a daemon set.

 
tetragon.sh

 

Intersystems logo Geometric Uppercase Display Letter i logo, Custom shape Negative Space Flat Two Colors, Blue + Turquoise. IRIS Workload

Quick IRIS pod, not privileged, but easily modified to be so.  This is the pod we will be executing things on to explain some of the tracing policy behavior.

 
iris.sh

 


Tracing Policies

TracingPolicies are custom resources that make it easy to setup real-time filters for kernel events. A TracingPolicy matches and filters system calls for observability and also triggers an action on these matches.

Right out of the box though, process_exec and process_exit without having to load any tracing policies.

In one terminal, execute your ZF, in the other, examine the Tetragon events:

kubectl exec -ti -n kube-system tetragon-sw9k4 -c tetragon -- tetra getevents -o compact --pods iris-nopriv

If we take a look at the process execution in Tetragon for the following call out that prints the current working directory.



This may be obvious to you, but running ZF with the "/SHELL" argument, invokes bash, and then calls the command, where as when it is ommitted, it calls out to the binary directly.  Now we used the compact output in the above, but if you observe the events in json format, you can see how they are called differently, with the /shell option having a parent process.

ie:

            "cwd": "/usr/irissys/bin",
            "binary": "/usr/irissys/bin/irisdb",
            "arguments": "-w /home/irisowner -s /usr/irissys/mgr",
            "flags": "execve",

 

 
direct.json

 

 

 
shell.json
 

 

The JSON events get sent to the tetragon log and can be sent to a SIEM system or observability for actionable insights.

SecurityRuntime Enforcement

This may lack a little bit of imagination for a use case, but what if we wanted to forbid anybody from calling out and "catting" the license file?

For this, we need to apply a TracingPolicy, that enforces an matchAction  These policies are a little involved, but this one is the long way of saying "Hey, if you run `cat /usr/irissys/mgr/iris.key`, I am going to kill you (SIGKILL you).  
 

kubectl apply -f - <<EOF
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "iris-read-file-sigkill"
spec:
  kprobes:
  - call: "fd_install"
    syscall: false
    return: false
    args:
    - index: 0
      type: int
    - index: 1
      type: "file"
    selectors:
    - matchPIDs:
      - operator: NotIn
        followForks: true
        isNamespacePID: true
        values:
        - 1
      matchArgs:
      - index: 1
        operator: "Prefix"
        values:
        - "/usr/irissys/secrets/"
        - "/usr/irissys/mgr/iris.key"
        #- "/tmp/" # wrecks havoc
      matchActions:
      - action: FollowFD
        argFd: 0
        argName: 1
  - call: "__x64_sys_close"
    syscall: true
    args:
    - index: 0
      type: "int"
    selectors:
    - matchActions:
      - action: UnfollowFD
        argFd: 0
        argName: 0
  - call: "__x64_sys_read"
    syscall: true
    args:
    - index: 0
      type: "fd"
    - index: 1
      type: "char_buf"
      returnCopy: true
    - index: 2
      type: "size_t"
    selectors:
    - matchActions:
      - action: Sigkill

EOF

Once deployed, you should see it loaded as a TracingPolicy resource:

So lets see it enforce the policy:

The -1 tells us something is awry, and the command was unsuccessful.

But not known to the fellow brogrammer, we administratively blocked it and sent a SIGKILL to the process!

That is going to be a long call to the WRC for the unsuspecting end user (or wrc specialist).

Experiments

I found a couple that were interesting in the hundreds I stole, applied, and played around with, notable was one that gave up the system calls per binary.  If you really wanted to nerd out, you could literally block by syscall.

Another one that was mesmerizing was the file access TracingPolicy, which showed all processes accessing all the files.  

These and other polices can be found in the examples repo @ tetragon:

  • System calls
  • Process attributes
  • Command-line arguments
  • Network activity
  • File system operations
Discussion (0)1
Log in or sign up to continue