Control Plane Metrics

This page consolidates Testkube Control Plane Prometheus metrics, including scheduler metrics.

In v2.7+, Control Plane metrics are enabled by default with:

• TK_CLOUD_METRICS_ENABLED=true

For context on the source-of-truth migration in v2.7, see Testkube Resource Management.

Availability and Enablement

Control Plane metrics are available in Testkube Cloud and Enterprise connected deployments and are exposed on the control plane /metrics endpoint.

• Path: /metrics on the Control Plane API service.
• TK_CLOUD_METRICS_ENABLED controls control-plane metrics emission (default true in v2.7+).
• TK_SCHEDULER_METRICS_ENABLED controls scheduler metrics emission (default true).
• TK_SCHEDULER_METRICS_INCLUDE_EXECUTION_ID enables execution-identity queue metric emission (default false).
• TK_WORKFLOW_CUSTOM_LABEL_METRICS_ENABLED enables exporting custom TestWorkflow labels as Prometheus metadata metrics (default false).
• TK_WORKFLOW_CUSTOM_LABEL_METRICS_MAX_PER_RUN caps how many custom labels are exported per workflow execution (default 10).

To enable custom workflow label metrics, set:

testkube-cloud-api:
  api:
    workflowCustomLabelMetricsEnabled: true
    workflowCustomLabelMetricsMaxPerRun: 10

Label Model (Control Plane)

• environment_name.
• Workflow execution metrics also include workflow_namespace.
• runner_id (stable), runner_name (readable).
• No dynamic maps/arrays: dropped labels, tags, testworkflow_uri, trigger causes.
• triggered_by is normalized to manual, schedule, or trigger.
• CRUD counters drop resource names to keep cardinality bounded.
• testkube_testworkflow_custom_label_info is optional and exports selected custom workflow labels as label_key / label_value.

Available Metrics

Metric	Type	Description	Labels
Scheduler
testkube_scheduler_environment_queue_limit	gauge	Configured queue limit for the environment	environment_name
testkube_scheduler_queued_executions	gauge	Number of queued executions awaiting assignment	environment_name
testkube_scheduler_queued_executions_by_workflow	gauge	Number of queued executions awaiting assignment, grouped by workflow name	environment_name, workflow_name
testkube_scheduler_queued_execution_info	gauge	Identity metric for queued executions (value is always 1); emitted only when TK_SCHEDULER_METRICS_INCLUDE_EXECUTION_ID=true	environment_name, workflow_name, execution_id, execution_name
testkube_scheduler_active_executions	gauge	Number of currently active executions	environment_name
testkube_scheduler_queue_oldest_age_seconds	gauge	Age in seconds of the oldest queued execution (0 when none)	environment_name
testkube_scheduler_runner_active_executions	gauge	Active executions attributed to each runner	environment_name, runner_id, runner_name
Workflow Executions
testkube_testworkflow_executions_total	counter	Total number of test workflow executions	environment_name, workflow_name, workflow_namespace, workflow_result, triggered_by, runner_id, runner_name
testkube_testworkflow_executions_duration_seconds	histogram	Duration of test workflow executions	environment_name, workflow_name, workflow_namespace, workflow_result, triggered_by, runner_id, runner_name
testkube_testworkflow_aborts_total	counter	Total number of test workflow aborts	environment_name, workflow_name, workflow_namespace, workflow_result, triggered_by, runner_id, runner_name
testkube_testworkflow_custom_label_info	gauge	Optional metadata metric exporting selected custom TestWorkflow labels	execution labels + label_key, label_value
Workflow Execution Steps
testkube_testworkflow_execution_steps_total	counter	Total number of test workflow execution steps	execution labels + step_name, step_status
testkube_testworkflow_execution_steps_duration_seconds	histogram	Duration of test workflow execution steps	execution labels + step_name, step_status
testkube_testworkflow_execution_steps_start_time_seconds	histogram	Start time of test workflow execution steps	execution labels + step_name, step_status
testkube_testworkflow_execution_steps_finish_time_seconds	histogram	Finish time of test workflow execution steps	execution labels + step_name, step_status
Workflow Lifecycle (CRUD)
testkube_testworkflow_creations_total	counter	Total number of test workflows created	environment_name, result
testkube_testworkflow_updates_total	counter	Total number of test workflows updated	environment_name, result
testkube_testworkflow_deletes_total	counter	Total number of test workflows deleted	environment_name, result
Workflow Templates
testkube_testworkflowtemplate_creations_total	counter	Total number of test workflow templates created	environment_name, workflow_template, result
testkube_testworkflowtemplate_updates_total	counter	Total number of test workflow templates updated	environment_name, workflow_template, result
testkube_testworkflowtemplate_deletes_total	counter	Total number of test workflow templates deleted	environment_name, workflow_template, result
Triggers
testkube_testtrigger_creations_total	counter	Total number of test trigger created events	environment_name, result
testkube_testtrigger_events_total	counter	Total number of TestTrigger-fired workflow executions	environment_name, trigger_name
testkube_testtrigger_updates_total	counter	Total number of test trigger updated events	environment_name, result
testkube_testtrigger_deletes_total	counter	Total number of test trigger deleted events	environment_name, result
testkube_testtrigger_bulk_updates_total	counter	Total number of test trigger bulk update events	environment_name, result
testkube_testtrigger_bulk_deletes_total	counter	Total number of test trigger bulk delete events	environment_name, result
Webhooks
testkube_webhook_executions_total	counter	Total number of webhook executions	environment_name, webhook_name, event_type, result
testkube_webhook_creations_total	counter	Total number of webhook create attempts	environment_name, result
testkube_webhook_updates_total	counter	Total number of webhook update attempts	environment_name, result
testkube_webhook_deletes_total	counter	Total number of webhook delete attempts	environment_name, result

Examples

# scheduler queue depth
testkube_scheduler_queued_executions{environment_name="prod-env"} 4

# queued executions by workflow
testkube_scheduler_queued_executions_by_workflow{
  environment_name="prod-env",
  workflow_name="metrics-demo"
} 2

# queued execution identity (requires TK_SCHEDULER_METRICS_INCLUDE_EXECUTION_ID=true)
testkube_scheduler_queued_execution_info{
  environment_name="prod-env",
  workflow_name="metrics-demo",
  execution_id="01904b4d-4f8a-7f98-9dd7-991820db74d8",
  execution_name="metrics-demo-42"
} 1

# workflow execution (control plane)
testkube_testworkflow_executions_total{
  environment_name="prod-env",
  workflow_name="metrics-demo",
  workflow_namespace="testkube",
  workflow_result="passed",
  triggered_by="manual",
  runner_id="tkc_run_123",
  runner_name="runner-a"
} 1

# workflow custom label metadata (requires TK_WORKFLOW_CUSTOM_LABEL_METRICS_ENABLED=true)
testkube_testworkflow_custom_label_info{
  environment_name="prod-env",
  workflow_name="metrics-demo",
  workflow_namespace="testkube",
  workflow_result="passed",
  triggered_by="manual",
  runner_id="tkc_run_123",
  runner_name="runner-a",
  label_key="team",
  label_value="platform"
} 1

# workflow step duration (seconds, histogram)
testkube_testworkflow_execution_steps_duration_seconds_bucket{
  environment_name="prod-env",
  workflow_name="metrics-demo",
  workflow_namespace="testkube",
  workflow_result="passed",
  triggered_by="manual",
  runner_id="tkc_run_123",
  runner_name="runner-a",
  step_name="build",
  step_status="passed",
  le="1"
} 0

# trigger-fired workflow execution
testkube_testtrigger_events_total{
  environment_name="prod-env",
  trigger_name="smoke-on-deploy"
} 1

# webhook execution
testkube_webhook_executions_total{
  environment_name="prod-env",
  webhook_name="notify",
  event_type="END_TESTWORKFLOW_SUCCESS",
  result="success"
} 3

# webhook create
testkube_webhook_creations_total{
  environment_name="prod-env",
  result="created"
} 1

Agent to Control-Plane Mapping

Every Agent metric has a corresponding Control Plane metric. The mapping follows a small set of consistent conventions described below.

Naming Conventions

• Counter suffix: Agent metrics ending in _count become _total on the Control Plane (e.g. testkube_testworkflow_executions_count → testkube_testworkflow_executions_total).
• Trigger name normalization: Agent trigger metrics using the plural testtriggers become singular testtrigger on the Control Plane (e.g. testkube_testtriggers_updates_count → testkube_testtrigger_updates_total).
• Duration units: Agent timing metrics in milliseconds (_ms) become histograms in seconds (_seconds) on the Control Plane for cross-replica aggregation (e.g. testkube_testworkflow_executions_duration_ms → testkube_testworkflow_executions_duration_seconds).

Label Changes

All Control Plane metrics gain environment_name for environment-level scoping. Beyond that:

• Workflow execution and step metrics gain runner_id, runner_name, and workflow_namespace for finer scoping, and rename agent labels for clarity: name → workflow_name, result → workflow_result, status → step_status.
• triggered_by is normalized to one of manual, schedule, or trigger.
• High-cardinality labels are dropped: labels, tags, testworkflow_uri, and trigger causes are not carried to the Control Plane.
• CRUD counters use only environment_name and result (workflow names are omitted to keep cardinality bounded). Template metrics additionally include workflow_template.
• Webhook metrics rename name → webhook_name and eventType → event_type.
• Optional custom workflow label metadata is exposed separately via testkube_testworkflow_custom_label_info using label_key and label_value.

Metrics Not Mapped

Agent-only legacy test and test-suite metric families (testkube_test_*, testkube_testsuite_*) do not have Control Plane equivalents.

Full Mapping Reference

Agent Metric	Control-Plane Metric
testkube_testworkflow_executions_count	testkube_testworkflow_executions_total
testkube_testworkflow_executions_duration_ms	testkube_testworkflow_executions_duration_seconds
testkube_testworkflow_aborts_count	testkube_testworkflow_aborts_total
testkube_testworkflow_execution_steps_count	testkube_testworkflow_execution_steps_total
testkube_testworkflow_execution_steps_duration_ms	testkube_testworkflow_execution_steps_duration_seconds
testkube_testworkflow_execution_steps_start_time_ms	testkube_testworkflow_execution_steps_start_time_seconds
testkube_testworkflow_execution_steps_finish_time_ms	testkube_testworkflow_execution_steps_finish_time_seconds
testkube_testworkflow_creations_count	testkube_testworkflow_creations_total
testkube_testworkflow_updates_count	testkube_testworkflow_updates_total
testkube_testworkflow_deletes_count	testkube_testworkflow_deletes_total
testkube_testworkflowtemplate_creations_count	testkube_testworkflowtemplate_creations_total
testkube_testworkflowtemplate_updates_count	testkube_testworkflowtemplate_updates_total
testkube_testworkflowtemplate_deletes_count	testkube_testworkflowtemplate_deletes_total
testkube_testtrigger_creations_count	testkube_testtrigger_creations_total
testkube_testtrigger_event_count	testkube_testtrigger_events_total
testkube_testtriggers_updates_count	testkube_testtrigger_updates_total
testkube_testtriggers_deletes_count	testkube_testtrigger_deletes_total
testkube_testtriggers_bulk_updates_count	testkube_testtrigger_bulk_updates_total
testkube_testtriggers_bulk_deletes_count	testkube_testtrigger_bulk_deletes_total
testkube_webhook_executions_count	testkube_webhook_executions_total