Testkube Runner Metrics

The Testkube Runner API Server exposes a /metrics endpoint that can be consumed by Prometheus, Grafana, etc. These metrics are available on Standalone Runners and on connected Runners with the execution and webhooks capabilities enabled, as outlined below - Read More.

In standalone mode, metrics reflect local executions. In connected mode, metrics reflect executions performed in the associated Environment. Runners with the execution capability, and runners with the webhooks capability, expose the execution-related metrics listed below, while the Control Plane exposes its own metrics for centralized observability - see Control Plane Metrics.

note

No metrics are produced until at least one Workflow has been executed since the last Runner API server restart.

Available Metrics

Metric	Runner Capabilities*	Description	Labels
Workflow Executions
testkube_testworkflow_executions_count	Webhooks + Execution	Total number of test workflow executions	name, result, labels, testworkflow_uri, triggered_by, tags
testkube_testworkflow_executions_duration_ms	Webhooks + Execution	Duration of test workflow executions	name, result, labels, testworkflow_uri, triggered_by, tags
testkube_testworkflow_aborts_count	Execution	Total number of test workflow aborts	result
Workflow Execution Steps
testkube_testworkflow_execution_steps_count	Webhooks + Execution	Total number of test workflow execution steps	workflow_name, step_name, status
testkube_testworkflow_execution_steps_duration_ms	Webhooks + Execution	Duration of test workflow execution steps	workflow_name, step_name, status
testkube_testworkflow_execution_steps_start_time_ms	Execution	Start time of test workflow execution steps	workflow_name, step_name, status
testkube_testworkflow_execution_steps_finish_time_ms	Execution	Finish time of test workflow execution steps	workflow_name, step_name, status
Workflow Lifecycle (CRUD)
testkube_testworkflow_creations_count	Execution	Total number of test workflows created	result
testkube_testworkflow_updates_count	Execution	Total number of test workflows updated	result
testkube_testworkflow_deletes_count	Execution	Total number of test workflows deleted	result
Workflow Templates
testkube_testworkflowtemplate_creations_count	Execution	Total number of test workflow templates created	result
testkube_testworkflowtemplate_updates_count	Execution	Total number of test workflow templates updated	result
testkube_testworkflowtemplate_deletes_count	Execution	Total number of test workflow templates deleted	result
Triggers
testkube_testtriggers_creations_count	Execution	Total number of test trigger created events	result
testkube_testtriggers_updates_count	Execution	Total number of test trigger updated events	result
testkube_testtriggers_deletes_count	Execution	Total number of test trigger deleted events	result
testkube_testtriggers_bulk_updates_count	Execution	Total number of test trigger bulk update events	result
testkube_testtriggers_bulk_deletes_count	Execution	Total number of test trigger bulk delete events	result
testkube_testtrigger_event_count	Execution	Total number of test trigger events	name, resource, eventType, causes
Webhooks
testkube_webhook_executions_count	Execution + Webhooks	Total number of webhook executions	name, eventType, result

(* = these are the capabilities required for the metric to be available and updated.)

For TestTrigger-driven executions, these metrics reflect triggers whose workflow selectors use either labelSelector.matchLabels or labelSelector.matchExpressions. This lets you filter target TestWorkflows with standard Kubernetes label selector operators such as In, NotIn, Exists, and DoesNotExist. For selector examples, see Test Triggers.

Example Prometheus Output

# HELP testkube_testworkflow_executions_count The total number of test executions
# TYPE testkube_testworkflow_executions_count counter
testkube_testworkflow_executions_count{name="test-website",result="passed",type="curl-container/test"} 1

Installation

If a Prometheus operator is not installed, please follow the steps here: https://grafana.com/docs/grafana-cloud/quickstart/prometheus_operator/.

Then, add a ServiceMonitor custom resource to your cluster which will scrape metrics from our Testkube API server:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: testkube-api-server
  labels:
    app: prometheus
spec:
  endpoints:
    - interval: 10s
      port: http
  selector:
    matchLabels:
      app.kubernetes.io/name: api-server

If you're installing Testkube manually with our Helm chart, you can pass the prometheus.enabled value to the install command.

Grafana Dashboard

To use the Grafana dashboard, import this JSON definition:

https://github.com/kubeshop/testkube/blob/main/assets/grafana-dasboard.json