Testkube Agents

A Testkube Environment can contain any number of Agents to perform specific tasks. Agents are added to an Environment via the Dashboard (see below) and then deployed into their target cluster/namespace using either the provided CLI or Helm commands.

tip

The Agents described here are related to the Test Orchestration capabilities of Testkube, not to be confused with Testkube AI Agents, which allow you to run Agentic workloads in your Testkube Environment.

From SuperAgent to the Capability Model (Testkube 2.7.0)

As of Testkube 2.7.0, the concept of a single "superagent" (one agent holding all state when connected to the Control Plane) has been deprecated and replaced with agents that have explicit capabilities - Read More. Standalone mode (agent not connected to any Control Plane) remains unchanged and is still supported.

When upgrading a "superagent" from a pre 2.7.0 version to a 2.7.0+ version, that Agent will be automatically migrated to an Agent with all 4 capabilities enabled, its name will be set to default-agent-<environment-name>.

Agent Capabilities

A Testkube Agent can have any of the following 4 capabilities

1. Runner Agents - are deployed to any cluster/namespace where you want to execute your TestWorkflows - Read More.
2. Listener Agents - are deployed to any cluster/namespace where you want to listen for - Read More.
3. GitOps Agents - are deployed for syncing Testkube Resources into the Control Plane - Read More.
4. Webhook Agents - are deployed to from where you want to emit Webhook/CDEvents - Read More.

note

Naming - Agents vs Capabilities

Since an Agent can technically have any combination of these capabilities enabled, we will use the terms capability and agent somewhat interchangeably - i.e. a "Runner Agent" is the same as an "Agent with the Runner Capability", while an Agent with both the Listener and Webhook capabilities is both a Listener Agent and a Webhook Agent.

Runner Agents

Runner Agents execute Test Workflows in the cluster/namespace where they are deployed. You can have any number of Runner Agents in a Testkube Environment, allowing you to

1. Run the same Workflow in multiple namespaces/clusters, (possibly at the same time!).
2. Add ephemeral Runner Agents (deployed in ephemeral infrastructure) to an Environment and run your Test Workflows on them - Read More.

Use-cases for this are:

• Execute the same set of tests across production, staging, or testing environments to ensure consistent test execution.
• Execute the same set of tests in geographically dispersed environments to ensure consistent application behavior.
• Execute tests in local sandbox environments during development while having access to the centralized catalog of tests.
• Execute tests from multiple geographical locations against a (single) environment for realistic performance and e2e testing.
• Execute tests in ephemeral environments created during CI/CD pipelines for testing and deployment purposes - Read More.

Read More about how to run Workflows on Runner Agents at Running Test Workflows.

info

Runner Agents require a license - Read More.

Listener Agents

Listener Agents are created and deployed to clusters/namespaces where you want to listen for Kubernetes Events to trigger Test Triggers.

Use-cases for deploying Listener Agents:

• Listening to application changes in a cluster(s) managed by a GitOps tool to trigger tests when the cluster state is updated.
• Listening to infrastructure changes in a cluster(s) to ensure that validation tests are run whenever a cluster is updated.

Having multiple/separate Listener Agents from Runner Agents allows you to listen for changes in clusters separate from where you might want to run your tests, for example, if tests need to run from outside a cluster to validation connectivity or network performance, you could listen for events using a Listener Agent in cluster A, which would trigger the execution of Workflow on a Runner Agent deployed in cluster B.

Read more about how TestTriggers map to Listener Agents at Listener Agents with TestTriggers.

GitOps Agents

GitOps Agents are used to sync Testkube Resources from a Kubernetes namespace into the Testkube Control-Plane, which is required if you are using Testkube in a GitOps environment and want to manage your Testkube Resources in Git together with other Kubernetes Resources - Read More.

Webhook Agents

Testkube has the capability to emit different types of events for integrating with external tools:

• Webhooks
• CDEvents
• Kubernetes events

These are all emitted by a dedicated Webhook Agent which you can deploy anywhere in your infrastructure from where you want these events to be emitted.

Even if you have multiple Webhook Agents, only the first one will actually be used to emit Webhooks. This will be improved with dedicated Webhook Agent targeting in the next Testkube release.

note

For the emitting of CDEvents and Kubernetes Events, that corresponding functionality needs to be enabled further as described in the documents linked above.

Managing Agents

Agents are managed in the Agents tab under the Environment Settings:

The table has the following columns:

• Name: The name given to the agent on creation.
• Capabilities: Capability icons for the agent.
• Labels: Labels currently associated with the Runner Agent. Can be set with testkube update agent or via the runner's Helm values — see Updating Runner Agent labels and mode and Using labels for Runner Agent selection.
• Runner Mode: The Runner Mode of the agent if it is a Runner Agent. Can be changed with testkube update agent or via the runner's Helm values — see Updating Runner Agent labels and mode and Runner Agent modes.
• License: The type of license assigned to the Agent if it is a Runner Agent - Read More
• Agent ID: the Agent ID
• Version: The version of the Agent; a warning triangle will be shown if an updated version is available.
• Last seen: When the Agent was last seen.

Adding Agents to an Environment

Add a new Agent to an Environment by clicking the "Connect New Agent" button, which will open the below dialog:

As indicated, there are 3 main choices:

1. Which capabilities the Agent should have; Runner, Listener, GitOps or Webhook
2. If you want to use the Testkube CLI, Helm Chart or GitOps configuration to install the Agent
3. The commands to run to install the Agent with the selected capabilities.

Selecting "Helm" as the tool for installation will show corresponding commands:

Selecting "GitOps" as the tool for installation will show the YAML to use when auto-provisioning the Agent as part of a GitOps deployment - Read More.

Once the Agent is installed with the select capabilities and tool, it will show up in the list of Agents and is ready for use.

Managing an existing Agent

Existing agents are currently managed via the Testkube CLI, use the popup menu to the right in the table to get Agent information, Delete the Agent, or see examples of applicable CLI commands:

tip

Check out Multi-Agent CLI Overview for an overview of all Testkube CLI commands for working with Testkube Agents, or Runner Helm Chart Overview.

Agent Token Masking

In the “Organization Management” section, under the “Product Features” tab, there is an option called “Agent Token Masking”. When this toggle is enabled, the Testkube Dashboard will no longer display sensitive Runner Agent tokens in the UI.

warning

Important: This masking feature applies only to the Testkube Dashboard UI. Agent tokens or secret keys will still be visible in CLI outputs (e.g., when using testkube create runner or retrieving details for Helm chart installation as described in Installing Runner Agent with Helm Charts) and in any direct API interactions.

Updating Runner Agent labels and mode

A Runner Agent's labels and mode (--global / --group) can be managed in two ways, and both are supported simultaneously:

1. From the Control Plane — using testkube update agent <name> -l <key>=<value> or the Dashboard.
2. From the runner Deployment — by setting runner.register.labels, runner.register.global, or runner.register.groupName in the Helm values (or the equivalent RUNNER_GROUP / RUNNER_IS_GLOBAL environment variables on the runner Deployment).

When a runner restarts (for example after a Helm upgrade or rolling update of its Deployment) it reconnects to the Control Plane and may republish its registration metadata, but only for the fields it is explicitly configured to manage:

Runner-side configuration	What gets refreshed on reconnect
At least one entry in runner.register.labels	Labels are replaced with the runner's set
runner.register.labels is empty / unset	Labels are preserved (the runner does not touch them)
runner.register.global=true or runner.register.groupName set	Runner mode is replaced with the runner's setting
Neither set (Independent defaults)	Runner mode is preserved (the runner does not touch it)
Kubernetes API error reading the Deployment	Labels are preserved (warning logged); mode follows the rules above

This means CLI-managed agents keep working unchanged after upgrade: if you have not configured labels or mode in your runner's Helm values, the runner will not overwrite values you set with testkube update agent or testkube create agent --global / --group.

Switching to Helm as the source of truth

If you want the runner Deployment itself to be the source of truth for labels and/or mode, set the corresponding Helm values and run helm upgrade:

Field	Helm value	Runner env var
Labels	runner.register.labels	(label key prefix configurable via RUNNER_LABELS_PREFIX, default runner.testkube.io/)
Runner group	runner.register.groupName	RUNNER_GROUP
Global flag	runner.register.global	RUNNER_IS_GLOBAL

Once the runner pod restarts and reconnects, the Agents list in the Dashboard will reflect the new values. From that point on, any change you make through the CLI will be overwritten the next time the runner reconnects — pick a single source of truth per runner.

Demoting a Global or Grouped runner to Independent

Clearing runner.register.global / runner.register.groupName and reconnecting will not demote a runner to Independent: an empty policy snapshot is indistinguishable from "unset," so the runner does not touch the stored mode in that case (this prevents accidental downgrades during Helm value cleanups). To demote a runner explicitly, use:

$ testkube update agent <name> --runner-mode independent   # or use the Dashboard

note

Upgrading from a release before this change (kubeshop/testkube#7791) is safe for environments where labels and runner policy were managed via the CLI: those values are preserved on reconnect because the runner only refreshes fields it has been explicitly configured to publish.

Agent Key Rotation

Agent secret keys can be rotated to maintain security best practices or in response to a potential compromise. When a key is rotated, the previous key remains valid for a configurable grace period to allow in-flight requests to complete without interruption.

How It Works

• Grace period: After rotation, the old key continues to work for a configurable duration (default: 24 hours, maximum: 7 days). This ensures connected agents are not immediately disconnected.
• Single previous key: Only one previous key is retained at a time. If you rotate again before the grace period expires, the earlier previous key is discarded.
• Hash-only storage: Only the hash of the previous key is stored — the previous key value cannot be recovered from the Control Plane.

Rotating via the CLI

Use the testkube agent rotate-key command to rotate an agent's secret key:

$ testkube agent rotate-key my-agent

To specify a custom grace period:

$ testkube agent rotate-key my-agent --grace-period 4h

Rotating via the API

You can also rotate keys using the API:

DELETE /organizations/<organizationId>/agents/<agentIdOrName>/secret-key

Best Practices

• Rotate keys periodically as part of your security hygiene.
• Use the grace period to perform a rolling update of your agent deployments with the new key before the old key expires.
• After rotating, verify that all agents have reconnected with the new key before the grace period ends.

Licensing for Runner Agents

Testkube Runner Agents require a license for usage, which can be either Fixed or Floating.

• Runner Agents assigned a Fixed License can always run Workflows independently at any time.
• Runner Agents assigned a Floating license share the ability to execute Workflows concurrently; if one Runner Agent with a floating license is executing a Workflow, a second agent will queue Workflow executions until the first agent is complete. If you, for example, purchase two floating licenses and assign those to 10 agents, two of those agents will be able to execute Workflows concurrently at any give time.

Floating licenses are useful for automated and/or ephemeral use-cases where you don't know in advance how many Runner Agents you will have at any given point in time, and/or you don’t mind if your Workflow executions get queued.

note

Neither Listener, Webhook or GitOps Agents require a license, you can have as many of these Agents deployed in your infrastructure as you need.

Assigning Licenses to Runner Agents

Runner Agents are by default assigned a fixed license (as in all the examples above), use the --floating argument with Runner Agent creation commands to instead assign a floating license, for example:

# install temporary Runner Agent using a floating license
$ testkube install agent pr-12u48y34-runner --runner --create --floating

The Runner Agent will be shown with the License Type "Floating" in the list of Agents:

Runner Agent License Enforcement

The Runner Agent limit for both Fixed and Floating Licenses is counted and enforced at the organization level, i.e., across all your environments. Furthermore:

• You will only be able to create as many fixed Runner Agents as you have Fixed licenses in your Testkube plan.
• You will need to have at least one Floating license in your Testkube plan to be able to create Runner Agents with the --floating argument.

Please don't hesitate to Get in Touch if you have any questions/concerns about licensing.

Migrating old Environments

If you have an existing Environment created before the Multi-Agent functionality was introduced in Q2 2025, and that Environment already has Workflows being executed by CI/CD, CronJobs, Kubernetes Event Triggers, etc., these will continue to be executed on any Global Runner Agent connected to your Environment unless you update the corresponding triggering commands/configuration to target a specific Runner Agent, either by name, group or label - Read More.

info

Existing Environments that do not make the use of Runner Agents will continue to work as before, it is only when you start adding additional Runner Agents that you might need to adjust how your existing Workflows are triggered by external sources.