Skip to main content
New to Testkube? Unleash the power of cloud native testing in Kubernetes with Testkube. Get Started >

Helm Chart Installation and Usage Guide

Welcome to the Testkube Enterprise Helm chart installation and usage guide. This comprehensive guide provides step-by-step instructions for installing and utilizing the Testkube Enterprise Helm chart. Testkube Enterprise is a cutting-edge Kubernetes-native testing platform designed to optimize your testing and quality assurance processes with enterprise-grade features.

Prerequisites​

Before you proceed with the installation, please ensure that you have the following prerequisites in place:

  • Kubernetes cluster (version 1.21+)
  • Helm (version 3+)
  • cert-manager (version 1.11+) - Used for TLS certificate management.
  • NGINX Controller (version v1.8+) - Used for Ingress configuration.
  • (OPTIONAL) Prometheus Operator (version 0.49+) - used for metrics collection
  • Own a public/private domain for creating Ingress rules.
  • License Key and/or License File, if offline access is required.

NOTE While it is possible to use custom TLS certificates for the Testkube Enterprise API and Dashboard, we strongly recommend using cert-manager for easier certificate management.

Configuration​

Docker Images​

DEPRECATION NOTICE: As of November 2023, Testkube Enterprise Docker images are publicly accessible. You only need to follow the steps in this section if you wish to re-publish the images to your private Docker registry; otherwise, you may skip this section.

To begin, ensure that you have access to the Testkube Enterprise API & Dashboard Docker images. You can either request access from your Testkube representative or upload the Docker image tarball artifacts to a private Docker registry.

Next, create a secret to store your Docker registry credentials:

kubectl create secret docker-registry testkube-enterprise-registry \
  --docker-server=<your-registry-server> \
  --docker-username=<your-name>          \
  --docker-password=<your-pword>         \
  --docker-email=<your-email>            \
  --namespace=testkube-enterprise

Make sure to configure the image pull secrets in your values.yaml file:

global:
  imagePullSecrets:
    - name: testkube-enterprise-registry

License​

Select the appropriate license type for your environment.

For air-gapped & firewalled environments, we offer an option to use an Offline License for enhanced security. An Offline License consists of a License Key and License File.

If your environment has internet access, you can use an Online License, which only requires the License Key.

Online License​

If your environment has internet access, you can use an Online License, which only requires the License Key, and can be provided as a Helm parameter or Kubernetes secret.

To provide the License Key as a Helm parameter, use the following configuration:

global:
  enterpriseLicenseKey: <your license key>

To provide the License Key as a Kubernetes secret, first we need to create a secret with the required field. Run the following command to create the secret:

kubectl create secret generic testkube-enterprise-license \
  --from-literal=LICENSE_KEY=<your license key>           \
  --namespace=testkube-enterprise

And then use the following Helm chart configuration:

global:
  enterpriseLicenseSecretRef: <secret name>

Offline License​

For an Offline License, supply both the License Key and License File as either Kubernetes secrets or Helm parameters. Using secrets is safer, as it prevents exposing sensitive license information in Helm chart values.

The Kubernetes secret needs to contain 2 entries: license.lic and LICENSE_KEY. To create the secret with the License Key and License File, run the following command:

kubectl create secret generic testkube-enterprise-license \
  --from-literal=LICENSE_KEY=<your license key>            \
  --from-file=license.lic=<path-to-license-file>          \
  --namespace=testkube-enterprise

After creating the secret, use the following Helm chart configuration:

global:
  enterpriseOfflineAccess: true
  licenseFileSecret: testkube-enterprise-license

Alternatively, you can provide the License File as a Helm parameter:

global:
  licenseKey: <your license key>
  licenseFile: <your license file>

Ingress​

Testkube Enterprise requires the NGINX Controller to configure and optimize its protocols. NGINX is the sole supported Ingress Controller, and is essential for Testkube Enterprise's operation.

We highly recommend installing Testkube Enterprise with Ingress enabled. This requires a valid domain (public or private) with a valid TLS certificate. Ingresses are enabled and created by default.

To disable Ingress creation, adjust the following values accordingly. Note that you must then manually configure the API & Dashboard services to maintain accessibility:

global:
  ingress:
    enabled: false

testkube-cloud-api:
  api:
    tls:
      serveHTTPS: false

Configuration​

To ensure the reliable functioning of gRPC and Websockets protocols, Testkube Enterprise is locked in with NGINX Ingress Controller.

Below are current configurations per Ingress resource which ensure Testkube Enterprise protocols work performant and reliably. It is not recommended to change any of these settings!

gRPC Ingress annotations:

annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 8m
    nginx.ingress.kubernetes.io/client-header-timeout: "10800"
    nginx.ingress.kubernetes.io/client-body-timeout: "10800"

Websockets Ingress annotations:

annotations:
  nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
  nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"

If you want to use a different Ingress Controller, please reach out to our support team.

Domain​

Testkube Enterprise requires a domain (public or internal) under which it will expose the following services:

SubdomainService
dashboard.<your-(sub)domain>Dashboard UI
api.<your-(sub)domain>REST API
agent.(sub)<your-domain>gRPC API
websockets.(sub)<your-domain>WebSockets API
storage.(sub)<your-domain>Storage API
status.(sub)<your-domain>Status Pages API

TLS​

For best the performance, TLS should be terminated at the application level (Testkube Enterprise API) instead of NGINX/Ingress level because gRPC and Websockets protocols perform significantly better when HTTP2 protocol is used end-to-end. Note that NGINX, by default, downgrades the HTTP2 protocol to HTTP1.1 when the backend service is using an insecure port.

If cert-manager (check the Prerequisites for installation guide) is installed in your cluster, it should be configured to issue certificates for the configured domain by using the Issuer or ClusterIssuer resource. Testkube Enterprise Helm chart needs the following config in that case:

global:
  certificateProvider: "cert-manager"
  certManager:
    issuerRef: <issuer|clusterissuer name>

By default, Testkube Enterprise uses a ClusterIssuer cert-manager resource, that can be changed by setting the testkube-cloud-api.api.tls.certManager.issuerKind field to Issuer.

If cert-manager is not installed in your cluster, valid TLS certificates (for API & Dashboard) which cover the following subdomains need to be provided:

  • API (tls secret name is configured with testkube-cloud-api.api.tls.tlsSecret field)
    • api.<your-domain>
    • agent.<your-domain>
    • websockets.<your-domain>
    • status.<your-domain>
  • Dashboard (TLS secret name is configured with testkube-cloud-ui.ingress.tlsSecretName field)
    • dashboard.<your-domain> Also, global.certificateProvider should be set to blank ("").
global:
  certificateProvider: ""

Auth​

Testkube Enterprise utilizes Dex for authentication and authorization. For detailed instruction on configuring Dex, please refer to the Identity Provider document.

Metrics​

Testkube Enterprise exposes Prometheus metrics on the /metrics endpoint and uses a ServiceMonitor resource to expose them to Prometheus. In order for this to work, you need to have Prometheus Operator installed in your cluster so that the ServiceMonitor resource can be created.

Use the following configuration to enable metrics:

testkube-cloud-api:
  prometheus:
    enabled: true

Invitations​

Testkube Enterprise allows you to invite users to Organizations and Environments within Testkube, granting them specific roles and permissions.

There are two supported invitation modes: email and auto-accept. Use email to send an invitation for the user to accept, and auto-accept to automatically add users without requiring acceptance.

Invitations Via Email​

If testkube-cloud-api.api.inviteMode is set to email, Testkube Enterprise will send emails when a user gets invited to an Organization or an Environment and when SMTP settings need to be configured in the API Helm chart.

testkube-cloud-api:
  api:
    email:
      fromEmail: "[email protected]"
      fromName: "Example Invitation"
    inviteMode: email
    smtp:
      host: <smtp host>
      port: <smtp port>
      username: <smtp username>
      password: <smtp password>
      # password can also be referenced by using the `passwordSecretRef` field which needs to contain the key SMTP_PASSWORD
      # passwordSecretRef: <secret name>

Auto-accept Invitations​

If testkube-cloud-api.api.inviteMode is set to auto-accept, Testkube Enterprise will automatically add users to Organizations and Environments when they get invited.

testkube-cloud-api:
  inviteMode: auto-accept

Bring Your Own Infra​

Testkube Enterprise supports integrating with existing infrastructure components such as MongoDB, NATS, Dex, etc.

MongoDB​

Testkube Enterprise uses MongoDB as a database for storing all the data. By default, it will install a MongoDB instance using the Bitnami MongoDB Helm chart.

If you wish to use an existing MongoDB instance, you can configure the following values:

mongodb:
  enabled: false
 
testkube-cloud-api:
  api:
    mongo:
      dsn: <mongodb dsn (mongodb://...)>

NATS​

Testkube Enterprise uses NATS as a message broker for communication between API and Agents.

If you wish to use an existing NATS instance, you can configure the following values:

nats:
  enabled: false
  
testkube-cloud-api:
  api:
    nats:
      uri: <nats uri (nats://...)>

MinIO​

Testkube Enterprise uses MinIO as a storage backend for storing artifacts.

If you wish to use an existing MinIO instance, you can configure the following values:

testkube-cloud-api:
  minio:
    enabled: false
  api:
    minio: {} # check out the `testkube-cloud-api.api.minio` block in the values.yaml for all available settings

Dex​

Testkube Enterprise uses Dex as an identity provider.

If you wish to use an existing Dex instance, you can configure the following values:

global:
  dex:
    issuer: <dex issuer url>
dex:
  enabled: false
testkube-cloud-api:
  api:
    oauth: {} # check out the `testkube-cloud-api.api.oauth` block in the values.yaml for all available settings

Installation​

  1. Add our Testkube Enterprise Helm registry:
    helm repo add testkubeenterprise https://kubeshop.github.io/testkube-cloud-charts
  2. Create a values.yaml with preferred configuration.
  3. Run helm install testkube-enterprise testkubeenterprise/testkube-enterprise -f values.yaml --namespace testkube-enterprise.

IMPORTANT The Bitnami MongoDB Helm chart does not work reliably on ARM architectures. If you are installing MongoDB using this chart, you need to use an ARM compatible image:

mongodb:
  image:
    repository: zcube/bitnami-compat-mongodb
    tag: "6.0.5"

Production Setup​

For best performance and reliability, users should follow this official setup guide and make sure each section is properly configured.

  1. Configure DNS records as described in the Domain section
  2. Configure TLS certificates as described in the TLS section
  3. Configure Dex as described in the Auth section
  4. Configure Ingress as described in the Ingress section
  5. Configure Metrics as described in the Metrics section
  6. Configure Invitations as described in the Invitations section
  7. Configure BYOI components as described in the Bring Your Own Infra section
  8. Install Testkube Enterprise as described in the Installation section

FAQ​

Q: Testkube Enterprise API is crashing (pod is in Error/CrashLoopBackOff state) with the following error:

panic: license file is invalid

A: Make sure the license file ends with a newline character. There should be a new line after the -----END LICENSE FILE----- line in the license file.