Promise Testing Framework

The Promise Testing Framework paves the way for easy testing of Kratix Promises, providing fast feedback and confidence in your Pipelines and Stages without needing to deploy your Promise to a Kubernetes Cluster.

Available as plugin to the Kratix CLI, the framework provides validation of the expected behaviour when Pipelines and Pipeline Stages run and validation of the outputs generated when Pipelines and individual stages run to completion.

Prerequisites

You must have the kratix CLI installed at version 0.9.0 or later.

Installing the Testing Framework

Install the latest version of the gh cli.

Set your Enterprise Releases token as the value of the GH_TOKEN environment variable.

Download the latest kratix-test plugin binary:

# Fetch the latest release tag
LATEST_RELEASE=$(gh release list --repo syntasso/enterprise-releases --limit 100 --json tagName,createdAt \
  --jq '. | sort_by(.createdAt) | reverse | map(select(.tagName | startswith("kratix-test"))) | .[0].tagName')

# Must be one of darwin_amd64, darwin_arm64, linux_amd64
ARCH=

# Download the release
gh release download $LATEST_RELEASE \
  --repo syntasso/enterprise-releases \
  --pattern "kratix-test_$ARCH" \
  --output kratix-test

Make the binary executable and place it anywhere in your PATH.

As the binary is in your path, it should be visible when you run:

kratix plugin list

To confirm that the plugin has been installed without issue, run:

kratix test --help

You should see output similar to:

A command for testing Kratix Promise Pipelines and Stages

Usage:
  test [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  pipeline    A command for testing Kratix Promise Pipelines
  stage       A command for testing Kratix Promise stages

Flags:
  -h, --help   help for test

Use "test [command] --help" for more information about a command.

Testing your Promise with the Testing Framework

Given your Promise was generated with the Kratix CLI, the directory structure should be similar to the following:

├── README.md
├── example-resource.yaml
├── promise.yaml
└── workflows
    └── promise
        └── configure
            └── setup
                ├── deploy-infrastructure
                │   ├── Dockerfile
                │   ├── resources
                │   └── scripts
                │       └── pipeline.sh
                ├── apply-rbac-rules
                │   ├── Dockerfile
                │   ├── resources
                │   └── scripts
                │       └── pipeline.sh
                └── security-scan
                    ├── Dockerfile
                    ├── resources
                    └── scripts
                        └── pipeline.sh

When structured in this format, both the kratix test stage and kratix test pipeline commands can identify the appropriate image to run or build when performing tests against the specified Pipeline or Pipeline Stage. If your directory is not structured in this way, the tests may fail to find the correct container image to run.

Testing individual Pipeline Stages

To test a given Pipeline Stage, you can use the kratix test stage command, specifying the workflow type, workflow action, Pipeline name and stage name that you wish to test. Alongside these arguments, you will also need to provide:

• Test input via the --input-dir flag. This is a directory that should contain the object.yaml that you want test your Stage against
• Expected test outputs via the --expected-output-dir. This should contain files that match the files that you expect to be written to the /kratix/outputs directory by the stage.

For the above example, we could run:

kratix test stage promise/configure/setup deploy-infrastructure \
  --input-dir inputs \
  --expected-output-dir setup-outputs

This will run the container for the Pipeline Stage (as specified in the promise.yaml) with the default workflow environment variables set for you. Upon running the stage container, the test validates whether the outputs generated by the stage match those in the setup-outputs directory. If they match, the command will display a success message and exit 0.

Running
Validating
- Comparing /expected-stage-outputs/configmap.yaml to /test-outputs2423274464/configmap.yaml

Cleaning up
Success (1 assertion(s))

However, if they do not match, the test will display a diff between the files and exit 1.

--- expected
+++ actual
@@ -1,4 +1,5 @@
 apiVersion: v1
 kind: ConfigMap
 metadata:
-  name: infra-config
+  creationTimestamp: null
+  name: map

Cleaning up
Failure (1 assertion(s) failed)

info

You can also provide a tag for a build image via the --image-tag flag in place of the Pipeline and Pipeline Stage arguments. When taking this approach, your promise does not need to match the above directory structure.

Testing a Pipeline

To test a Pipeline by running each of its stages in sequence, you can use the kratix test pipeline command. For the above example, we could run:

kratix test pipeline promise/configure/setup \
  --input-dir inputs \
  --expected-output-dir setup-outputs

This will run each of the stages of the setup pipeline and and validate whether the outputs generated by all of the Stages match those in the setup-outputs directory. If they match, the command will exit 0. However, if they do not match, the test will display a diff between the files and exit 1.

tip

You can use the --build flag to build with both the stage and pipeline commands to build the Pipeline Stages before running the tests