Test Workflows - Matrix and Sharding

info

This Workflows functionality is not available when running the Testkube Runner in Standalone Mode - Read More

It is often desirable to run a test for multiple scenarios or environments, for example to distribute load or to verify your application for different input. This is traditionally achieved by defining configuration matrixes for each scenario and running all permutations.

Furthermore, it is common to split tests into multiple executions that run in parallel ("shards") to speed up the execution.

Test Workflows have built-in mechanisms for both these cases, using both static and dynamic configurations, which can further be combined with parallelisation to achieve highly efficient test executions.

note

Sharding (or parallel execution in general) usually only makes sense for long-lasting tests. In the case of small and/or short-lasting tests, sharding will often not decrease overall execution times.

Usage

Matrix and sharding features are supported in Services (services), and both Composite (execute) and Parallel Step (parallel) operations.

Services (services)

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: example-matrix-services
spec:
  services:
    remote:
      matrix:
        browser:
          - driver: chrome
            image: selenium/standalone-chrome:4.21.0-20240517
          - driver: edge
            image: selenium/standalone-edge:4.21.0-20240517
          - driver: firefox
            image: selenium/standalone-firefox:4.21.0-20240517
      image: "{{ matrix.browser.image }}"
      description: "{{ matrix.browser.driver }}"
      readinessProbe:
        httpGet:
          path: /wd/hub/status
          port: 4444
        periodSeconds: 1
  steps:
    - shell: 'echo {{ shellquote(join(map(services.remote, "tojson(_.value)"), "\n")) }}'

Composite (execute)

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: example-matrix-test-suite
spec:
  steps:
    - execute:
        workflows:
          - name: k6-workflow-smoke
            matrix:
              target:
                - https://testkube.io
                - https://docs.testkube.io
            config:
              target: "{{ matrix.target }}"

Parallel Steps (parallel)

apiVersion: testworkflows.testkube.io/v1
kind: TestWorkflow
metadata:
  name: example-sharded-playwright
spec:
  content:
    git:
      uri: https://github.com/kubeshop/testkube
      paths:
        - test/playwright/playwright-project
  container:
    image: mcr.microsoft.com/playwright:v1.32.3-focal
    workingDir: /data/repo/test/playwright/playwright-project

  steps:
    - name: Install dependencies
      shell: "npm ci"

    - name: Run tests
      parallel:
        count: 2
        transfer:
          - from: /data/repo
        shell: "npx playwright test --shard {{ index + 1 }}/{{ count }}"

Syntax

This feature allows you to provide few properties:

• matrix to run the operation for different combinations
• count/maxCount to replicate or distribute the operation
• shards to provide the dataset to distribute among replicas

Both matrix and shards can be used together - all the sharding (shards + count/maxCount) will be replicated for each matrix combination.

Matrix

Matrix allows you to run the operation for multiple combinations. The values for each instance are accessible by matrix.<key>.

In example:

parallel:
  matrix:
    image: ["node:20", "node:21", "node:22"]
    memory: ["1Gi", "2Gi"]
  container:
    resources:
      requests:
        memory: "{{ matrix.memory }}"
  run:
    image: "{{ matrix.image }}"

Will instantiate 6 copies:

index	matrixIndex	matrix.image	matrix.memory	shardIndex
0	0	"node:20"	"1Gi"	0
1	1	"node:20"	"2Gi"	0
2	2	"node:21"	"1Gi"	0
3	3	"node:21"	"2Gi"	0
4	4	"node:22"	"1Gi"	0
5	5	"node:22"	"2Gi"	0

The matrix properties can be a static list of values, like:

matrix:
  browser: ["chrome", "firefox", "{{ config.another }}"]

or could be dynamic one, using Test Workflow's expressions:

matrix:
  files: 'glob("/data/repo/**/*.test.js")'

Sharding

Often you may want to distribute the load, to speed up the execution. To do so, you can use shards and count/maxCount properties.

• shards is a map of data to split across different instances
• count/maxCount are describing the number of instances to start
  • count defines static number of instances (always)
  • maxCount defines maximum number of instances (will be lower if there is not enough data in shards to split)

Similarly to matrix, the shards may contain a static list, or Test Workflow's expression, see examples below:

Replicas (count only)

parallel:
  count: 5
  description: "{{ index + 1 }} instance of {{ count }}"
  run:
    image: grafana/k6:latest

Static sharding (count + shards)

parallel:
  count: 2
  description: "{{ index + 1 }} instance of {{ count }}"
  shards:
    url:
      [
        "https://testkube.io",
        "https://docs.testkube.io",
        "https://app.testkube.io",
      ]
  run:
    # shard.url for 1st instance == ["https://testkube.io", "https://docs.testkube.io"]
    # shard.url for 2nd instance == ["https://app.testkube.io"]
    shell: 'echo {{ shellquote(join(shard.url, "\n")) }}'

Dynamic sharding (maxCount + shards)

parallel:
  maxCount: 5
  shards:
    # when there will be less than 5 tests found - it will be 1 instance per 1 test
    # when there will be more than 5 tests found - they will be distributed similarly to static sharding
    testFiles: 'glob("cypress/e2e/**/*.js")'
  description: '{{ join(map(shard.testFiles, "relpath(_.value, \"cypress/e2e\")"), ", ") }}'

tip

While running tests in parallel shards, each run can result in its own report or artifacts, which need to be captured and possibly merged into a single report.

Sometimes this can be handled directly by the tool (like Playwright's merge-report, see the Sharded Playwright Example), but sometimes handling that in custom ways may be needed.

Counters

Besides having the matrix.<key> and shard.<key> there are some counter variables available in Test Workflow's expressions:

• index and count - counters for total instances
• matrixIndex and matrixCount - counters for the combinations
• shardIndex and shardCount - counters for the shards

Matrix and sharding together

Sharding can be run along with matrix. In that case, for every matrix combination, we do have selected replicas/sharding. In example:

matrix:
  browser: ["chrome", "firefox"]
  memory: ["1Gi", "2Gi"]
count: 2
shards:
  url:
    [
      "https://testkube.io",
      "https://docs.testkube.io",
      "https://app.testkube.io",
    ]

Will start 8 instances:

index	matrixIndex	matrix.browser	matrix.memory	shardIndex	shard.url
0	0	"chrome"	"1Gi"	0	["https://testkube.io", "https://docs.testkube.io"]
1	0	"chrome"	"1Gi"	1	["https://app.testkube.io"]
2	1	"chrome"	"2Gi"	0	["https://testkube.io", "https://docs.testkube.io"]
3	1	"chrome"	"2Gi"	1	["https://app.testkube.io"]
4	2	"firefox"	"1Gi"	0	["https://testkube.io", "https://docs.testkube.io"]
5	2	"firefox"	"1Gi"	1	["https://app.testkube.io"]
6	3	"firefox"	"2Gi"	0	["https://testkube.io", "https://docs.testkube.io"]
7	3	"firefox"	"2Gi"	1	["https://app.testkube.io"]

Dynamic Sharding

Individual tests that are spread across shards can have varying execution times, potentially leading to a situation where some shards take much longer to execute than others. A common approach to solve this is to statically define which tests that go into which shard, based on previous execution times. While Testkube currently does not support this approach natively, an alternative approach is available that will work for any kind of sharding mechanism.

To implement dynamic sharding in a parallel step:

• Set the number of shards very high (preferably to the number of testcases, but NOT higher)
• Set the number of parallel workers to a lower value (higher than 1), which will be the actual number of parallel workers that will be used to execute the tests.

Testkube will still create individual nodes for each shard, but only the specified number of parallel workers will execute at the same time, resulting in individual long-running tests not blocking other short-running test-cases, while multiple long-running test-cases can be executed in parallel without staggering the entire execution.

Playwright Example

For example, with Playwright tests, hooking into Playwrights native sharding mechanism:

- name: Run tests
  parallel:
    description: "Shard: {{ index + 1 }}/{{ count }}"
    count: 20 # keeping this one high, to have a high number of shards - every test could be a separate shard
    parallelism: 3 # keeping this low to execute up to 3 "workers" at a time
    transfer:
      - from: /data/repo
    run:
      shell: "npx playwright test  --shard={{ index + 1 }}/{{ count }}" # sharding

tip

See the Sharded Playwright Example for a more detailed example.

Cypress Example

This example is for Cypress, where we use the shards property to split test files into shards:

- name: Run tests
  parallel:
    maxCount: 10
    parallelism: 3
    shards:
      testFiles: glob("cypress/e2e/**/*.js")
    description: '{{join(map(shard.testFiles,"relpath(_.value, \"cypress/e2e\")"),",)}}'
    transfer:
      - from: /data/repo
    run:
      args:
        - --env
        - NON_CYPRESS_ENV=NON_CYPRESS_ENV_value
        - --spec
        - '{{join(shard.testFiles,",")}}'

tip

See the Sharded Cypress Example for a more detailed example.