Test Workflows - Services

info

This Workflows functionality is not available when running the Testkube Agent in Standalone Mode - Read More

Often if your use-case is more complex, you may need additional services for the Tests you are running. Common use cases are:

• Database, i.e. MongoDB or PostgreSQL
• Workers, i.e. remote JMeter workers or Selenium Grid's remote browsers
• Service under test, i.e. your API to run E2E tests against it

Testkube allows you to run such services for the Test Workflow, communicate with them and debug smoothly.

How it works

When you define the service, the Test Workflow is creating a new pod and any other require resources for each of instances, read its status and logs, and provides its information (like IP) for use in further steps. After the service is no longer needed, it's cleaned up.

info

As the services are started in a separate pod, they don't share the file system with the Test Workflow execution. There are multiple ways to share data with them - either using one of techniques described below, or advanced Kubernetes' native ways like ReadWriteMany volumes.

Syntax

To add some services, you need to specify the services clause. It can be either directly on the spec level (to be available for the whole execution), or on specific step (to isolate it) - see Schema Reference.

tip

You may want to use services in Test Workflow Template, to reuse them for multiple tests.

YAML

apiVersion: testworkflows.testkube.io/v1
kind: TestWorkflow
metadata:
  name: example-workflow-with-mongo-service
spec:
  services:
    db:
      timeout: 5m
      image: mongo:latest
      env:
      - name: MONGO_INITDB_ROOT_USERNAME
        value: root
      - name: MONGO_INITDB_ROOT_PASSWORD
        value: p4ssw0rd
      readinessProbe:
        tcpSocket:
          port: 27017
        periodSeconds: 1
  steps:
  - name: Check if it is running
    run:
      image: mongo:latest
      shell: |
        echo Connecting to MongoDB at {{ services.db.0.ip }}
        mongosh -u root -p p4ssw0rd {{ services.db.0.ip }} --eval 'db.serverStatus().localTime'

Log Output

Connecting to the services

To connect to create services, you can simply use services.<SVC_NAME>.<INDEX>.ip expression in the place you need its address (i.e. environment variable, or shell command).

• services.db.0.ip will return string - IP of the 1st instance of the db service
• services.db.*.ip will return []string - list of IPs of all the db service instances

Basic configuration

The service allows similar fields as the run command, i.e.:

• image, env, volumeMounts, resources - to configure the container
• command, args - to specify command to run
• shell - to specify script to run (instead of command/args)
• description that may provide human-readable information for each instance separately

Fetching logs

By default we are not saving the logs for the services. If you would like to fetch the logs, you can use logs property. It takes an expression condition, so you can dynamically choose whether it should be saved or not. Often you will use:

• logs: always to always store the logs
• logs: failed to store the logs only if the Test Workflow has failed

Pod configuration

The service is started as a separate pod, so you can configure pod similarly to the Test Workflow's configuration.

Lifecycle

You can apply readinessProbe to ensure that the service will be available for the next step.

The Test Workflow won't continue until the container will be ready then. To ensure that the execution won't get stuck, you can add timeout property (like timeout: 1h30m20s), so it will fail if the service is not ready after that time.

Matrix and sharding

The services are meant to support matrix and sharding, to run multiple replicas and/or distribute the load across multiple instances. It is supported by regular matrix/sharding properties (matrix, shards, count and maxCount).

You can read more about it in the general Matrix and Sharding documentation.

Providing content

There are multiple ways to provide the files inside the services.

info

As the services are started in a separate pod, they don't share the file system with the Test Workflow execution.

Copying content inside

It is possible to copy the files from the original Test Workflow into the services. As an example, you may want to fetch the repository and install the dependencies on the original TestWorkflow, and then distribute it to the services.

To do so, you can use transfer property. It takes list of files to transfer:

• { from: "/data/repo/build" } will copy the /data/repo/build directory from execution's Pod into /data/repo/build in the service's Pod
• { from: "/data/repo/build", to: "/out" } will copy the /data/repo/build directory from execution's Pod into /out in the service's Pod
• { from: "/data/repo/build", to: "/out", "files": ["**/*.json"] } will copy only JSON files from the /data/repo/build directory from execution's Pod into /out in the service's Pod

Example

YAML

apiVersion: testworkflows.testkube.io/v1
kind: TestWorkflow
metadata:
  name: example-workflow-with-building-app-and-files-transfer
spec:
  content:
    git:
      uri: https://github.com/kubeshop/testkube-docs.git
      revision: main
  container:
    workingDir: /data/repo
    resources:
      requests:
        cpu: 1
        memory: 2Gi

  steps:
  - name: Build the application
    run:
      image: node:21
      shell: npm i && npm run build

  - name: Test the application
    services:
      server:
        timeout: 1m
        transfer:
        - from: /data/repo/build
          to: /usr/share/nginx/html
        image: nginx:1.25.4
        logs: always
        readinessProbe:
          httpGet:
            path: /
            port: 80
          periodSeconds: 1
    steps:
    - shell: wget -q -O - {{ services.server.0.ip }}

Log Output

Static content or a Git repository

Services allow to provide the content property similar to the one directly in the Test Workflow. As an example, you may provide static configuration files to the service:

YAML

apiVersion: testworkflows.testkube.io/v1
kind: TestWorkflow
metadata:
  name: example-workflow-with-nginx
spec:
  services:
    http:
      timeout: 5m
      content:
        files:
        - path: /etc/nginx/nginx.conf
          content: |
            events { worker_connections 1024; }
            http {
              server {
                listen 8888;
                location / { root /www; }
              }
            }
        - path: /www/index.html
          content: "foo-bar"
      image: nginx:1.25.4
      readinessProbe:
        httpGet:
          path: /
          port: 8888
        periodSeconds: 1
  steps:
  - shell: wget -q -O - {{ services.http.0.ip }}:8888

Log Output

Examples

JMeter with distributed Remote Workers

You can easily run JMeter with distributed remote workers, that could be even spread evenly across all the Kubernetes nodes.

The example below:

• Read JMX configuration from Git repository (spec.content.git)
• Start 5 remote workers (spec.services.slave.count)
  • Distribute them evenly across nodes (spec.services.slave.use[0] - distribute/evenly template is setting common pod.topologySpreadConstraints)
  • Reserve 1/8 CPU and 128MB memory for each instance (spec.services.slave.container.resources)
  • Wait until they will accept connection at port 1099 (spec.services.slave.readinessProbe)
• Run JMeter controller against all the remote workers (spec.services.steps[0].run)
  • It uses {{ services.slave.*.ip }} as an argument - services.slave.*.ip will return list of IPs, and they will be joined by comma (,) to convert to text

apiVersion: testworkflows.testkube.io/v1
kind: TestWorkflow
metadata:
  name: distributed-jmeter-example
spec:
  content:
    git:
      uri: https://github.com/kubeshop/testkube
      revision: main
      paths:
      - test/jmeter/executor-tests/jmeter-executor-smoke.jmx
  container:
    workingDir: /data/repo/test/jmeter/executor-tests
  services:
    slave:
      use:
      - name: distribute/evenly
      count: 5
      timeout: 30s
      image: anasoid/jmeter:5.6-plugins-21-jre
      command:
      - jmeter-server
      - -Dserver.rmi.localport=60000
      - -Dserver_port=1099
      - -Jserver.rmi.ssl.disable=true
      container:
        resources:
          requests:
            cpu: 128m
            memory: 128Mi
      readinessProbe:
        tcpSocket:
          port: 1099
        periodSeconds: 1
  steps:
  - name: Run tests
    run:
      image: anasoid/jmeter:5.6-plugins-21-jre
      shell: |
        jmeter -n \
          -X -Jserver.rmi.ssl.disable=true -Jclient.rmi.localport=7000 \
          -R {{ services.slave.*.ip }} \
          -t jmeter-executor-smoke.jmx

Selenium tests with multiple remote browsers

You can initialize multiple remote browsers, and then run tests against them in parallel.

The example below:

• Clone the test code (content)
• Start 3 instances of remote service (services.remote)
  • Each instance have different browser used (image of services.remote.matrix.browser passed to services.remote.image)
  • Each instance expose the driver name in their description (driver of services.remote.matrix.browser passed to services.remote.description)
  • Wait until the browser is ready to connect (services.remote.readinessProbe)
  • Always save the browser logs (services.remote.logs)
• Run tests in parallel for each of the browser (steps[0].parallel)
  • Run for each remote service instance (steps[0].parallel.matrix.browser)
  • Transfer the code from the repository to the parallel step (steps[0].parallel.transfer)
  • Sets the environment variables based on the service instance's description and IP (steps[0].parallel.container.env)
  • Run tests (steps[0].parallel.shell)

YAML

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: selenium-remote-browsers-example
spec:
  content:
    git:
      uri: https://github.com/cerebro1/selenium-testkube.git
      paths:
      - selenium-java
  services:
    remote:
      matrix:
        browser:
        - driver: chrome
          image: selenium/standalone-chrome:4.21.0-20240517
        - driver: edge
          image: selenium/standalone-edge:4.21.0-20240517
        - driver: firefox
          image: selenium/standalone-firefox:4.21.0-20240517
      logs: always
      image: "{{ matrix.browser.image }}"
      description: "{{ matrix.browser.driver }}"
      readinessProbe:
        httpGet:
          path: /wd/hub/status
          port: 4444
        periodSeconds: 1
  steps:
  - name: Run cross-browser tests
    parallel:
      matrix:
        browser: 'services.remote'
      transfer:
      - from: /data/repo/selenium-java
      container:
        workingDir: /data/repo/selenium-java
        image: maven:3.9.6-eclipse-temurin-22-alpine
        env:
        - name: SELENIUM_BROWSER
          value: '{{ matrix.browser.description }}'
        - name: SELENIUM_HOST
          value: '{{ matrix.browser.ip }}:4444'
      shell: mvn test

Log Output

Run database for integration tests

To test the application, you often want to check if it works well with the external components too. As an example, unit tests won't cover if there is a syntax error in SQL query, or there are deadlocks in the process, unless you will run it against actual database.

The example below:

• Start single MongoDB instance as db service (services.db)
  • Configure initial credentials to root/p4ssw0rd (services.db.env)
  • Wait until the MongoDB accept connections (services.db.readinessProbe)
• Run integration tests (steps[0].run)
  • Configure API_MONGO_DSN environment variable to point to MongoDB (steps[0].run.env[0])
  • Install local dependencies and run tests (steps[0].run.shell)

apiVersion: testworkflows.testkube.io/v1
kind: TestWorkflow
metadata:
  name: database-service-example
spec:
  content:
    git:
      uri: https://github.com/kubeshop/testkube.git
      revision: develop
  services:
    db:
      image: mongo:latest
      env:
      - name: MONGO_INITDB_ROOT_USERNAME
        value: root
      - name: MONGO_INITDB_ROOT_PASSWORD
        value: p4ssw0rd
      readinessProbe:
        tcpSocket:
          port: 27017
        periodSeconds: 1
  container:
    workingDir: /data/repo
  steps:
    - name: Run integration tests
      run:
        image: golang:1.22.3-bookworm
        env:
        - name: API_MONGO_DSN
          value: mongodb://root:p4ssw0rd@{{services.db.0.ip}}:27017
        shell: |
          apt-get update
          apt-get install -y ca-certificates libssl3 git skopeo
          go install gotest.tools/gotestsum@v1.9.0

          INTEGRATION=y gotestsum --format short-verbose -- -count 1 -run _Integration -cover ./pkg/repository/...