Running Test Workflows in a sandboxed container runtime

Test Workflows can run code you have not written or reviewed: third-party tools, community images, or tests generated by an AI agent. In a standard container, that code shares the node's kernel with every other pod on the node, so a kernel vulnerability exploited from a test can reach the node and the workloads beside it. A sandboxed container runtime such as gVisor gives each pod its own kernel layer, so test code cannot reach the node kernel directly.

This page shows how to run Test Workflows in a sandboxed runtime by sending the runner's executions to a dedicated namespace and letting cluster admission policies assign the runtime to every pod created there. Testkube runs each workflow as standard Kubernetes pods, so the policies cover every pod a workflow creates, with no changes to the workflows themselves. The setup uses gVisor and Kyverno, in three steps:

1. Admission policies on a dedicated sandbox namespace give every pod created there the gVisor runtime class, and reject any pod without it.
2. The runner runs executions in the sandbox namespace by default.
3. A verification workflow confirms that pods start under gVisor.

Prerequisites

• A Kubernetes cluster where every node runs gVisor, with a RuntimeClass for gVisor using the handler runsc. The samples assume the RuntimeClass is named gvisor. If gVisor runs only on selected nodes, also follow gVisor on selected nodes only.
• Kyverno, which provides the MutatingPolicy and ValidatingPolicy types the samples use.
• A Testkube runner installed from the testkube-runner Helm chart. See Installing Testkube Runner with Helm Charts.

The samples on this page are written for Kyverno 1.19 and testkube-runner chart 2.14.2.

Create the sandbox namespace and admission policies

The following sample creates the sandbox namespace and two Kyverno policies. Both policies match pods created in testkube-sandbox, selected by the namespace's kubernetes.io/metadata.name label:

• The MutatingPolicy gives every pod the gVisor runtime class.
• The ValidatingPolicy rejects any pod without the gVisor runtime class, so a pod the mutation misses is rejected instead of starting on the node kernel.
• The Namespace's Pod Security baseline label makes Kubernetes reject pods that ask to mount folders from the node or use other host access, which would get around the sandbox. Keep this label.

testkube-sandbox-policies.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: testkube-sandbox
  labels:
    pod-security.kubernetes.io/enforce: baseline
---
apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata:
  name: testkube-sandbox-gvisor-mutate
spec:
  failurePolicy: Fail
  autogen:
    podControllers:
      controllers: []
  matchConstraints:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: testkube-sandbox
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods"]
  mutations:
    - patchType: JSONPatch
      jsonPatch:
        expression: >
          [
            JSONPatch{op: "add", path: "/spec/runtimeClassName", value: "gvisor"}
          ]
---
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: testkube-sandbox-gvisor-validate
spec:
  failurePolicy: Fail
  validationActions: [Deny]
  autogen:
    podControllers:
      controllers: []
  matchConstraints:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: testkube-sandbox
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods"]
  validations:
    - expression: "has(object.spec.runtimeClassName) && object.spec.runtimeClassName == 'gvisor'"
      message: "Pods in testkube-sandbox must run under the gvisor runtime class."

Apply the file:

kubectl apply -f testkube-sandbox-policies.yaml

Settings in the policies that are required

Two settings in the policies are required:

• Autogen off on both policies (autogen.podControllers.controllers: []). Without it, Kyverno also checks the Jobs Testkube creates and rejects them, so no test pod ever starts.
• failurePolicy: Fail. If Kyverno is unavailable, pods in the sandbox namespace are rejected rather than started without gVisor.

Configure the runner

The runner's execution.default.namespace setting moves executions into a namespace separate from the runner, as described in Install runner in one namespace and run executions in another. Set it to the sandbox namespace in your runner's values.yaml:

values.yaml

execution:
  default:
    namespace: testkube-sandbox

Apply the change the way you apply other changes to the runner's values.

Every Test Workflow this runner executes now runs in the sandbox namespace by default, under gVisor. A workflow can run in another namespace only if the runner allows it under execution.additionalNamespaces, and pods there run without gVisor unless that namespace has the same policies. See Considerations for production rollouts.

Verify the sandbox

You now have a sandbox namespace whose policies give every pod the gVisor runtime class, and a runner that sends its executions there by default. Check that it works by running the following workflow. It sets no job.namespace, so it reaches the sandbox namespace only through the runner configuration, and it checks the main pod, a service and two parallel workers.

sandbox-check.yaml

apiVersion: testworkflows.testkube.io/v1
kind: TestWorkflow
metadata:
  name: sandbox-check
spec:
  container:
    image: alpine:3.20
  services:
    web:
      image: nginx:1.27-alpine
      readinessProbe:
        httpGet:
          path: /
          port: 80
  steps:
    - name: Main pod runtime
      shell: |
        echo "kernel: $(uname -r)"
        dmesg | grep -qi gvisor && echo "RESULT: under gVisor" || (echo "RESULT: NOT gVisor" && exit 1)
    - name: Reach the sandboxed service
      shell: wget -qO- http://{{ services.web.0.ip }} | grep -i nginx
    - name: Parallel workers
      parallel:
        count: 2
        container:
          image: alpine:3.20
        shell: |
          echo "worker {{ index }} kernel: $(uname -r)"
          dmesg | grep -qi gvisor && echo "RESULT: worker under gVisor" || (echo "RESULT: NOT gVisor" && exit 1)

A passing run shows the following:

• Main pod and each worker: the kernel version from uname -r, which under gVisor differs from the node's, followed by RESULT: under gVisor or RESULT: worker under gVisor. Without gVisor, the step prints RESULT: NOT gVisor and fails.
• Service step: the Nginx welcome page. The step fails if the page does not arrive.

What you've accomplished

Executions on this runner now run under gVisor by default, isolated from the node kernel. From here:

• No per-workflow changes. Every workflow this runner executes that sets no job.namespace runs in the sandbox, including workflows created later.
• New images. Before you rely on the sandbox for a new image, run a workflow that uses that image on this runner.
• New nodes. Nodes you add to the cluster need gVisor too, so that sandbox pods can run on them.
• Where the setup stops. The sandbox covers execution pods this runner creates in the sandbox namespace. Considerations for production rollouts covers other runners and namespaces, network access and Kubernetes API access.

Considerations for production rollouts

gVisor on selected nodes only

The steps on this page assume every node in the cluster runs gVisor. If gVisor runs only on a dedicated set of nodes, sandbox pods must also be placed on those nodes. Otherwise a pod can land on a node without gVisor and never start, while its execution stays in running with no error.

To place them, give the gVisor nodes a label and a NoSchedule taint, and add a scheduling section to the gVisor RuntimeClass with a node selector for that label and a toleration for that taint. Kubernetes then adds both to every pod that uses the RuntimeClass, including pods whose runtime class the MutatingPolicy sets, so the sandbox policies stay as they are. Give any gVisor nodes you add later the same label and taint. RuntimeClass scheduling describes how Kubernetes combines these with a pod's own node selector and tolerations.

For example, with nodes labelled and tainted sandbox.example.com/runtime=gvisor, the RuntimeClass's scheduling section looks like this:

scheduling:
  nodeSelector:
    sandbox.example.com/runtime: gvisor
  tolerations:
    - key: sandbox.example.com/runtime
      operator: Equal
      value: gvisor
      effect: NoSchedule

Other points to consider

• Network access. gVisor does not restrict network access. A NetworkPolicy on the sandbox namespace can block destinations tests do not need, such as the cloud metadata endpoint at 169.254.169.254, as long as the cluster's network plugin enforces NetworkPolicies. Keep connections to the applications under test and to the Testkube components described in Testkube Architecture open, and run sandbox-check after adding the policy.
• Kubernetes API access. Execution pods use a ServiceAccount that can manage pods, secrets and other objects in the sandbox namespace, which Testkube needs for services and parallel steps. For runners whose workflows use neither, Disable access to Kubernetes from executions removes that access.
• Other runners and namespaces. The sandbox covers execution pods this runner creates in the sandbox namespace. Workflows on other runners, workflows sent to a namespace listed under execution.additionalNamespaces without the same policies, and the runner pod itself run without gVisor. To sandbox only some workflows, give them a dedicated runner and target it, as described in Runner Targeting.
• Compatibility. gVisor does not implement every Linux system call, so some images, such as those that run a browser, may not run. The baseline label rejects pods that request privileged mode, which Docker-in-Docker needs. Run a workflow that uses an image on this runner before you rely on it.
• Performance. gVisor adds overhead to system calls and I/O. Expect slower runs for tests that make many system calls or read and write heavily, and set execution timeouts with that in mind.
• Kyverno availability. With failurePolicy: Fail, no sandboxed execution can start while Kyverno is unavailable. Run Kyverno with more than one replica, so that losing one pod does not stop executions.
• Other runtimes and admission controllers. Testkube's part of this setup does not depend on gVisor or Kyverno, so the same approach applies to other sandboxed runtimes such as Kata Containers, and to other admission controllers that can both change and reject pods. Adapt the policy and verification steps using that tool's documentation.
• Global Template. Setting job.namespace in the runner's Global Template does not isolate executions, because a workflow's own job.namespace overrides it.

Troubleshooting

An execution stays in running without progress. The pod was most likely placed on a node without gVisor, so it never starts. Look for the event no runtime for "runsc" is configured in the sandbox namespace:

kubectl get events -n testkube-sandbox

If the event is there, the node has no gVisor. Install gVisor on it. If gVisor runs only on selected nodes, check that those nodes carry the label and taint named in the gVisor RuntimeClass's scheduling section.

Pods are rejected although the policies look correct. Check that Kyverno is running. With failurePolicy: Fail, Kubernetes rejects pods in the sandbox namespace while Kyverno is unavailable. If Kyverno is running, read the rejection message: a pod that requests a setting the baseline label forbids, such as a hostPath volume, is rejected by Pod Security, and the message names the setting.

A step fails only under gVisor. The image may use a system call that gVisor does not implement. To confirm, run the same workflow on a runner without the sandbox.