Helm Chart Installation and Usage Guide
- Testkube Pro On-Prem Helm Chart Installation and Usage Guide
Welcome to the Testkube Enterprise Helm chart installation and usage guide. This comprehensive guide provides step-by-step instructions for installing and utilizing the Testkube Enterprise Helm chart. Testkube Enterprise is a cutting-edge Kubernetes-native testing platform designed to optimize your testing and quality assurance processes with enterprise-grade features.
Installation of Testkube Enterprise and an Agent in the same cluster
It is possible to deploy an instance of Testkube Enteprise and connect an Agent to it in the same k8s cluster without exposing endpoints to the outside world. This way gives you a more customizable set-up and allows you to have a working environment in just a few minutes. It can be deployed locally or in any other k8s cluster.
For this you will need:
- kubectl
- connection to a running k8s cluster
Simply run bash <(curl -sSLf https://download.testkube.io)
and enter a license key, the script will do the rest.
The script will do a port-forward to the following ports: 8080
, 8090
, 5556
in the background mode. Please make sure they are available.
The installation will take about 4-5 min, once it is completed you will have the Testkube Enterprise deployed in testkube-enterprise
namespace and the Testkube Agent in testkube
namespace. The UI is available at http://localhost:8080. We use Dex for authentication, so once you open the URL you will see a login page - use [email protected]
and password
as a username and a password respectively.
Voila! Now you can create tests, testsuites in both CLI and UI and explore the power of Testkube!
If you close a terminal you may do a port-forward with the following commands:
kubectl port-forward svc/testkube-enterprise-ui 8080:8080 --namespace testkube-enterprise &
kubectl port-forward svc/testkube-enterprise-api 8090:8088 --namespace testkube-enterprise &
kubectl port-forward svc/testkube-enterprise-dex 5556:5556 --namespace testkube-enterprise &
Installation of Testkube Enterprise and an Agent in multiple clusters
Prerequisites
Before you proceed with the installation, please ensure that you have the following prerequisites in place:
- Kubernetes cluster (version 1.21+)
- Helm (version 3+)
- cert-manager (version 1.11+) - Used for TLS certificate management.
- NGINX Controller (version v1.8+) - Used for Ingress configuration.
- (OPTIONAL) Prometheus Operator (version 0.49+) - used for metrics collection
- Own a public/private domain for creating Ingress rules.
- License Key and/or License File, if offline access is required.
NOTE
While it is possible to use custom TLS certificates for the Testkube Enterprise API and Dashboard,
we strongly recommend using cert-manager
for easier certificate management.
Configuration
Docker Images
DEPRECATION NOTICE: As of November 2023, Testkube Enterprise Docker images are publicly accessible. You only need to follow the steps in this section if you wish to re-publish the images to your private Docker registry; otherwise, you may skip this section.
To begin, ensure that you have access to the Testkube Enterprise API & Dashboard Docker images. You can either request access from your Testkube representative or upload the Docker image tarball artifacts to a private Docker registry.
Next, create a secret to store your Docker registry credentials:
kubectl create secret docker-registry testkube-enterprise-registry \
--docker-server=<your-registry-server> \
--docker-username=<your-name> \
--docker-password=<your-pword> \
--docker-email=<your-email> \
--namespace=testkube-enterprise
Make sure to configure the image pull secrets in your values.yaml
file:
global:
imagePullSecrets:
- name: testkube-enterprise-registry
License
Select the appropriate license type for your environment.
For air-gapped & firewalled environments, we offer an option to use an Offline License for enhanced security. An Offline License consists of a License Key and License File.
If your environment has internet access, you can use an Online License, which only requires the License Key.
Online License
If your environment has internet access, you can use an Online License, which only requires the License Key, and can be provided as a Helm parameter or Kubernetes secret.
To provide the License Key as a Helm parameter, use the following configuration:
global:
enterpriseLicenseKey: <your license key>
To provide the License Key as a Kubernetes secret, first we need to create a secret with the required field. Run the following command to create the secret:
kubectl create secret generic testkube-enterprise-license \
--from-literal=LICENSE_KEY=<your license key> \
--namespace=testkube-enterprise
And then use the following Helm chart configuration:
global:
enterpriseLicenseSecretRef: <secret name>
Offline License
For an Offline License, supply both the License Key and License File as either Kubernetes secrets or Helm parameters. Using secrets is safer, as it prevents exposing sensitive license information in Helm chart values.
The Kubernetes secret needs to contain 2 entries: license.lic
and LICENSE_KEY
.
To create the secret with the License Key and License File, run the following command:
kubectl create secret generic testkube-enterprise-license \
--from-literal=LICENSE_KEY=<your license key> \
--from-file=license.lic=<path-to-license-file> \
--namespace=testkube-enterprise
After creating the secret, use the following Helm chart configuration:
global:
enterpriseOfflineAccess: true
licenseFileSecret: testkube-enterprise-license
Alternatively, you can provide the License File as a Helm parameter:
global:
licenseKey: <your license key>
licenseFile: <your license file>
Ingress
Testkube Enterprise requires the NGINX Controller to configure and optimize its protocols. NGINX is the sole supported Ingress Controller, and is essential for Testkube Enterprise's operation.
We highly recommend installing Testkube Enterprise with Ingress enabled. This requires a valid domain (public or private) with a valid TLS certificate. Ingresses are enabled and created by default.
To disable Ingress creation, adjust the following values accordingly. Note that you must then manually configure the API & Dashboard services to maintain accessibility:
global:
ingress:
enabled: false
testkube-cloud-api:
api:
tls:
serveHTTPS: false
Configuration
To ensure the reliable functioning of gRPC and Websockets protocols, Testkube Enterprise is locked in with NGINX Ingress Controller.
Below are current configurations per Ingress resource which ensure Testkube Enterprise protocols work performant and reliably. It is not recommended to change any of these settings!
gRPC Ingress annotations:
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 8m
nginx.ingress.kubernetes.io/client-header-timeout: "10800"
nginx.ingress.kubernetes.io/client-body-timeout: "10800"
Websockets Ingress annotations:
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
If you want to use a different Ingress Controller, please reach out to our support team.
Domain
Testkube Enterprise requires a domain (public or internal) under which it will expose the following services:
Subdomain | Service |
---|---|
dashboard.<your-(sub)domain> | Dashboard UI |
api.<your-(sub)domain> | REST API |
agent.(sub)<your-domain> | gRPC API |
websockets.(sub)<your-domain> | WebSockets API |
storage.(sub)<your-domain> | Storage API |
status.(sub)<your-domain> | Status Pages API |
TLS
For best the performance, TLS should be terminated at the application level (Testkube Enterprise API) instead of NGINX/Ingress level because gRPC and Websockets protocols perform significantly better when HTTP2 protocol is used end-to-end. Note that NGINX, by default, downgrades the HTTP2 protocol to HTTP1.1 when the backend service is using an insecure port.
If cert-manager
(check the Prerequisites for installation guide) is installed in your cluster, it should be configured to issue certificates for the configured domain by using the Issuer
or ClusterIssuer
resource.
Testkube Enterprise Helm chart needs the following config in that case:
global:
certificateProvider: "cert-manager"
certManager:
issuerRef: <issuer|clusterissuer name>
By default, Testkube Enterprise uses a ClusterIssuer
cert-manager
resource, that can be changed by setting the testkube-cloud-api.api.tls.certManager.issuerKind
field to Issuer
.
If cert-manager
is not installed in your cluster, valid TLS certificates (for API & Dashboard) which cover the following subdomains need to be provided:
- API (tls secret name is configured with
testkube-cloud-api.api.tls.tlsSecret
field)api.<your-domain>
agent.<your-domain>
websockets.<your-domain>
status.<your-domain>
- Dashboard (TLS secret name is configured with
testkube-cloud-ui.ingress.tlsSecretName
field)dashboard.<your-domain>
Also,global.certificateProvider
should be set to blank ("").
global:
certificateProvider: ""
Custom certificates
In order to use custom certificates, first a secret needs to be created with the following entries:
tls.crt
- the certificatetls.key
- the private keyca.crt
- the CA certificate (if the certificate is not self-signed)
If certificate-based authentication is required, the custom certificates need to be configured in the following places:
- Enterprise API
- If
MINIO_ENDPOINT
is set to an exposed URL, then the following Helm values need to be configured:- The following Helm parameter needs to be enabled to inject the custom certificate into MinIO
testkube-cloud-api.minio.certSecret.enabled: true
- If the certificate is not self-signed, the CA cert needs to be injected also by enabling the Helm parameter
testkube-cloud-api.minio.mountCACertificate: true
- Custom certificate verification can also be skipped by setting
testkube-cloud-api.minio.skipVerify: true
- The following Helm parameter needs to be enabled to inject the custom certificate into MinIO
- If
MINIO_ENDPOINT
uses the Kubernetes DNS record (testkube-enterprise-minio.<namespace>.svc.cluster.local:9000
),AGENT_STORAGE_HOSTNAME
should be set to point to the exposed storage URL
- If
- Agent
- Agent API
- If the Enterprise API is configured to use certificate-based authentication or is using a certificate signed by a custom CA, the Agent API needs to be configured to use the same certificates by pointing
testkube-api.cloud.tls.certificate.secretRef
to the Kubernetes secret which contains the certificates - Custom certificate verification can also be skipped by setting
testkube-api.cloud.tls.skipVerify: true
- If the Enterprise API is configured to use certificate-based authentication or is using a certificate signed by a custom CA, the Agent API needs to be configured to use the same certificates by pointing
- Storage
- The following Helm parameter needs to be enabled to inject the custom certificate into MinIO
testkube-api.storage.certSecret.enabled: true
- If the certificate is not self-signed, the CA cert needs to be injected also by enabling the Helm parameter
testkube-cloud-api.minio.mountCACertificate: true
- Custom certificate verification can also be skipped by setting
testkube-api.storage.skipVerify: true
- The following Helm parameter needs to be enabled to inject the custom certificate into MinIO
- Agent API
Auth
Testkube Enterprise utilizes Dex for authentication and authorization. For detailed instruction on configuring Dex, please refer to the Identity Provider document.
Metrics
Testkube Enterprise exposes Prometheus metrics on the /metrics
endpoint and uses a ServiceMonitor
resource to expose them to Prometheus.
In order for this to work, you need to have Prometheus Operator
installed in your cluster so that the ServiceMonitor
resource can be created.
Use the following configuration to enable metrics:
testkube-cloud-api:
prometheus:
enabled: true
Invitations
Testkube Enterprise allows you to invite users to Organizations and Environments within Testkube, granting them specific roles and permissions.
There are two supported invitation modes: email
and auto-accept
.
Use email
to send an invitation for the user to accept, and auto-accept
to automatically add users without requiring acceptance.
Invitations Via Email
If testkube-cloud-api.api.inviteMode
is set to email
, Testkube Enterprise will send emails when a user gets invited to
an Organization or an Environment and when SMTP settings need to be configured in the API Helm chart.
testkube-cloud-api:
api:
email:
fromEmail: "[email protected]"
fromName: "Example Invitation"
inviteMode: email
smtp:
host: <smtp host>
port: <smtp port>
username: <smtp username>
password: <smtp password>
# password can also be referenced by using the `passwordSecretRef` field which needs to contain the key SMTP_PASSWORD
# passwordSecretRef: <secret name>
Auto-accept Invitations
If testkube-cloud-api.api.inviteMode
is set to auto-accept
, Testkube Enterprise will automatically add users to
Organizations and Environments when they get invited.
testkube-cloud-api:
inviteMode: auto-accept
Organization and Environment Management
Testkube Pro On-Prem allows you to manage organizations and environments using configuration.
testkube-cloud-api:
api:
features:
bootstrapConfig:
enabled: true
config:
organizations:
- name: prod_organization
environments:
- name: production_1
- name: production_2
On startup, the prod_organization
organization with two environments, production_1
and production_2
will be created.
Next, you can enhance the configuration to automatically add new users to organizations and environments with predefined roles. For example, the following config makes new users join prod_organization
as a member role and use production_1
environment as a run role:
bootstrapConfig:
enabled: true
config:
default_organizations:
- prod_organization
organizations:
- name: prod_organization
default_role: member
default_environments:
- production_1
environments:
- name: production_1
default_role: run
- name: production_2
Note: The default organization and environment mapping only apply on first sign in. After, you can remove users from environments or change roles thru Testkube UI.
Additionally, by default, Testkube Pro creates a personal organization for every new user. When using default organization and environment configuration, you can turn off personal organizations using the following config:
testkube-cloud-api:
api:
features:
disablePersonalOrgs: true
Bring Your Own Infra
Testkube Enterprise supports integrating with existing infrastructure components such as MongoDB, NATS, Dex, etc.
MongoDB
Testkube Enterprise uses MongoDB as a database for storing all the data. By default, it will install a MongoDB instance using the Bitnami MongoDB Helm chart.
If you wish to use an existing MongoDB instance, you can configure the following values:
mongodb:
enabled: false
testkube-cloud-api:
api:
mongo:
dsn: <mongodb dsn (mongodb://...)>
NATS
Testkube Enterprise uses NATS as a message broker for communication between API and Agents.
If you wish to use an existing NATS instance, you can configure the following values:
nats:
enabled: false
testkube-cloud-api:
api:
nats:
uri: <nats uri (nats://...)>
MinIO
Testkube Enterprise uses MinIO as a storage backend for storing artifacts.
If you wish to use an existing MinIO instance, you can configure the following values:
testkube-cloud-api:
minio:
enabled: false
api:
minio: {} # check out the `testkube-cloud-api.api.minio` block in the values.yaml for all available settings
Dex
Testkube Enterprise uses Dex as an identity provider.
If you wish to use an existing Dex instance, you can configure the following values:
global:
dex:
issuer: <dex issuer url>
dex:
enabled: false
testkube-cloud-api:
api:
oauth: {} # check out the `testkube-cloud-api.api.oauth` block in the values.yaml for all available settings
Installation
- Add our Testkube Enterprise Helm registry:
helm repo add testkubeenterprise https://kubeshop.github.io/testkube-cloud-charts
- Create a
values.yaml
with preferred configuration. - Run
helm install testkube-enterprise testkubeenterprise/testkube-enterprise -f values.yaml --namespace testkube-enterprise
.
IMPORTANT The Bitnami MongoDB Helm chart does not work reliably on ARM architectures. If you are installing MongoDB using this chart, you need to use an ARM compatible image:
mongodb:
image:
repository: zcube/bitnami-compat-mongodb
tag: "6.0.5"
Production Setup
For best performance and reliability, users should follow this official setup guide and make sure each section is properly configured.
- Configure DNS records as described in the Domain section
- Configure TLS certificates as described in the TLS section
- Configure Dex as described in the Auth section
- Configure Ingress as described in the Ingress section
- Configure Metrics as described in the Metrics section
- Configure Invitations as described in the Invitations section
- Configure BYOI components as described in the Bring Your Own Infra section
- Install Testkube Enterprise as described in the Installation section
FAQ
Q: Testkube Enterprise API is crashing (pod is in Error
/CrashLoopBackOff
state) with the following error:
panic: license file is invalid
A: Make sure the license file ends with a newline character.
There should be a new line after the -----END LICENSE FILE-----
line in the license file.