Deploy Inspektor Gadget on OpenShift 4.6

ยท 902 words ยท 5 minute read

Introduction ๐Ÿ”—

Inspektor Gadget is a collection of tools (or gadgets) to debug and inspect Kubernetes applications.

Inspektor Gadget is deployed to each node as a privileged DaemonSet. It uses in-kernel BPF helper programs to monitor events mainly related to syscalls from userspace programs in a pod. The BPF programs are run by the kernel and gather the log data. Inspektor Gadget’s userspace utilities fetch the log data from ring buffers and display it.

The architecture explains how it works under the hood, but to me it means to be able to run eBPF programs easily on Kubernetes.

Inspektor Gadget in OpenShift ๐Ÿ”—

There are a few minor details currently that ‘prevents’ to run it in OpenShift:

  • It needs to be installed in the kube-system namespace (it is not recommended to deploy things there)
  • It uses /run/ to store the gadgettracermanager socket (instead its own directory)

But mainly, it doesn’t recognize RHCOS and fails to be deployed.

Fortunately, there is a fork created by Philippe Huet that circunvent those issues and works pretty well!

In the meantime, I opened a Github issue to notify the Kinvolk folks about this so in the future it can be deployed without any forks.

NOTE: Obviously this is totally unsupported.

OCP Requirements ๐Ÿ”—

AFAIK, there is no special requirements to run it, besides being cluster-admin (well, the workers operating system is a limitation as well but we are using Philippe’s fork that allows it to run it in OCP)

In this post I’ll deploy it using crc but I’ve been able to make it work using a baremetal IPI cluster as well.

NOTE: In case you don’t know what crc is, check this blog post or the official documentation , but tl;dr.- it is a stripped version of OpenShift so you can run it in your beefy laptop (4 vCPUs, 9 GB of free memory & 35 GB of storage space is required) in Linux/Windows/OSX as it is basically a VM with a preconfigured single node OCP cluster in.

Deploy OCP (using crc) ๐Ÿ”—

A Red Hat account is required to access the user pull secret. You can create a developer account that will give you access to crc, RHEL and some other goodies in the developers.redhat.com site.

  • Download crc
  • Download the pull-secret file
  • Extract the crc binary tar Jxvf crc-linux-amd64.tar.xz
  • Optionally, copy it to your $PATH
  • Run crc setup and paste the pull-secret content when asked
  • Run crc start and wait for the VM to be up
  • Run eval $(crc oc-env) and export KUBECONFIG=~/.crc/machines/crc/kubeconfig

You are now cluster-admin of your own OCP cluster running on your laptop, pretty cool right?

Deploy Inspektor Gadget ๐Ÿ”—

oc apply -f https://raw.githubusercontent.com/clustership/inspektor-gadget/master/openshift/deployment.yaml

NOTE: There is a bug currently where the daemonset can fail with sigsegv. In that case, delete the daemonset and redeploy it again:

oc delete ds gadget -n gadget-tracing
oc apply -f https://raw.githubusercontent.com/clustership/inspektor-gadget/master/openshift/deployment.yaml

Compile the client ๐Ÿ”—

It is required to have make, git and golang already installed. RHEL8:

sudo dnf install -y make git golang

Clone the inspektor-gadget fork repository:

git clone https://github.com/clustership/inspektor-gadget.git

Build the client:

make kubectl-gadget-linux-amd64

Copy the binary to the /usr/local/bin/ directory so oc and kubectl can recognize it as a plugin:

chmod a+x ./kubectl-gadget-linux-amd64
sudo mv kubectl-gadget-linux-amd64 /usr/local/bin/kubectl-gadget
sudo restorecon /usr/local/bin/kubectl-gadget

NOTE: It is required to compile your own cli because it includes fixes to make it work with OpenShift (basically querying the inspketor-gadget pods running in the gadget-tracing namespace instead of kube-system)

Usage (get capabilities requested by a pod) ๐Ÿ”—

Deploy a demo application. Basically this one but in the gadget-tracing namespace:

cat <<EOF | oc apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: set-priority
  namespace: gadget-tracing
  labels:
    k8s-app: set-priority
spec:
  selector:
    matchLabels:
      name: set-priority
  template:
    metadata:
      labels:
        name: set-priority
    spec:
      containers:
      - name: set-priority
        image: busybox
        command: [ "sh", "-c", "while /bin/true ; do nice -n -20 echo ; sleep 5; done" ]
EOF

Then, get the pod name and the worker name where the pod is running. In this case, we will be focused on crc-tnpk6-master-0:

oc get po -o wide -n gadget-tracing

NAME                            READY   STATUS    RESTARTS   AGE   IP               NODE                 NOMINATED NODE   READINESS GATES
gadget-cf8hx                    1/1     Running   0          18h   192.168.126.11   crc-tnpk6-master-0   <none>           <none>
set-priority-5646554d9d-jcmch   1/1     Running   0          18h   10.116.0.27      crc-tnpk6-master-0   <none>           <none>

Run the kubectl-gadget capabilities command as:

kubectl-gadget capabilities --unique --verbose -p set-priority-5646554d9d-jcmch --node crc-tnpk6-master-0

or

oc gadget capabilities --verbose --unique -p set-priority-5646554d9d-jcmch --node crc-tnpk6-master-0

The output looks like:

Node numbers: 0 = crc-tnpk6-master-0
Running command: exec /opt/bcck8s/bcc-wrapper.sh --tracerid 20201203092226-55d305646fcd --gadget /usr/share/bcc/tools/capable  --namespace gadget-tracing --podname set-priority-5646554d9d-jcmch  --  --unique -v
NODE TIME      UID    PID    COMM             CAP  NAME                 AUDIT 
[ 0] 08:22:30  1000580000 16178  sh               21   CAP_SYS_ADMIN        0     
[ 0] 08:22:30  1000580000 20501  sh               21   CAP_SYS_ADMIN        0     
[ 0] 08:22:30  1000580000 20501  nice             6    CAP_SETGID           1     
[ 0] 08:22:30  1000580000 20501  nice             7    CAP_SETUID           1     
[ 0] 08:22:30  1000580000 20501  nice             23   CAP_SYS_NICE         1     
[ 0] 08:22:30  1000580000 20500  sh               21   CAP_SYS_ADMIN        0     
[ 0] 08:22:30  1000580000 20500  true             6    CAP_SETGID           1     
[ 0] 08:22:30  1000580000 20500  true             7    CAP_SETUID           1     
[ 0] 08:22:30  1000580000 20502  sh               21   CAP_SYS_ADMIN        0     
[ 0] 08:22:30  1000580000 20502  sleep            6    CAP_SETGID           1     
[ 0] 08:22:30  1000580000 20502  sleep            7    CAP_SETUID           1
^C
Terminating...
Running command: exec /opt/bcck8s/bcc-wrapper.sh --tracerid 20201123115948-c2009dae9253 --stop

References ๐Ÿ”—

Notes ๐Ÿ”—

Yes, I’ve tried to resist the temptation to put an Inspector gadget image in the post to make it look more professional… but I guess nobody reads it until completion, so here it is! :D

Inspector Gadget