Terraform Enterprise

Kratix supports scheduling workloads via Promises to run on HCP Terraform (Terraform Cloud) or Terraform Enterprise.

This page provides information on how to:

1. Configure TFE as a Kratix Destination.
2. Write a Promise that can schedule workloads to TFE.
3. Use the SKE Aspect to pull information out of TFE.

This documentation assumes the existence of a Terraform Workspace that's configured to apply Terraform configurations from a Git repository. Please refer to the TFE documentation for more information on how to create and configure the Workspace.

Configuring the Destination

Once you have a a Workspace configured to listen to a Git repository, you can configure Kratix to be able to send workloads to it. To do this, you will first need to create a GitStateStore pointing to the same Git repository you're using for your TFE Workspace.

Create the GitStateStore, replacing the placeholders with your own values:

apiVersion: platform.kratix.io/v1alpha1
kind: GitStateStore
metadata:
  name: workspace-repo
spec:
  authMethod: <AUTH METHOD>
  branch: <BRANCH>
  secretRef:
    name: <SECRET NAME>
    namespace: <NAMESPACE>
  url: <MY WORKSPACE REPOSITORY>

Refer to the GitStateStore documentation for more information on how to configure the GitStateStore.

Next, create a Secret with the credentials needed to access your TFE Workspace:

apiVersion: v1
kind: Secret
metadata:
  name: tf-credentials
  namespace: default
stringData:
  organization: <TFE ORG NAME>
  token: <TFE TOKEN>
  workspace: <TFE WORKSPACE NAME>

To generate a token, refer to the TFE documentation.

You can now create a Destination, tying everything together:

apiVersion: platform.kratix.io/v1alpha1
kind: Destination
metadata:
  name: tfe
  annotations:
    kratix.io/tf-credentials: '{"namespace": "default", "name": "tf-credentials"}'
  labels:
    environment: terraform
spec:
  filepath:
    mode: none
  stateStoreRef:
    kind: GitStateStore
    name: workspace-repo

Destination in detail

There are a few important things to note about the Destination above.

• The spec.stateStoreRef is pointing to the GitStateStore you created earlier.
• The spec.filepath.mode is set to none. That's because TFE does not support nested directories, which is how Kratix writes to the State Store by default. It also means your Promise needs to output unique files as part of the Pipeline, otherwise multiple requests will overwrite the same file.
• The metadata.annotations.kratix.io/tf-credentials is a JSON object that contains the reference to the Secret you created earlier. This is used by the SKE Aspect to pull information out of TFE.
• The metadata.labels.environment is set to terraform. This is not required. You will use that label later when writing your Promise.

With all the resources created, you can now write a Promise that schedules workloads to TFE.

Writing a Promise

TFE does not support nested directories in the State Store, so your promise needs to ensure the uniqueness of the files it writes.

Here's an example Promise that schedules a workload to TFE:

Example: Promise which schedules to TFE

apiVersion: platform.kratix.io/v1alpha1
kind: Promise
metadata:
  name: example-tf-promise
spec:
  destinationSelectors:
    - matchLabels:
        environment: terraform
  api:
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      name: terraforms.tfe.ske.io
    spec:
      group: tfe.ske.io
      names:
        kind: Terraform
        plural: terraforms
        singular: terraform
      scope: Namespaced
      versions:
        - name: v1alpha1
          schema:
            openAPIV3Schema:
              type: object
              properties:
                spec:
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
          served: true
          storage: true
  workflows:
    promise:
      configure:
        - apiVersion: platform.kratix.io/v1alpha1
          kind: Pipeline
          metadata:
            name: promise
            namespace: default
          spec:
            containers:
              - image: ghcr.io/syntasso/kratix-pipeline-utility:v0.0.1
                name: generate-output
                command: [ "sh" ]
                args:
                  - -c
                  - |
                    cat <<EOF > /kratix/output/main.tf
                    output "promise-configure" {
                      value = "some-value"
                    }
                    EOF
    resource:
      configure:
        - apiVersion: platform.kratix.io/v1alpha1
          kind: Pipeline
          metadata:
            name: resource
            namespace: default
          spec:
            containers:
              - image: ghcr.io/syntasso/kratix-pipeline-utility:v0.0.1
                name: generate-output
                command: [ "sh" ]
                args:
                  - -c
                  - |
                    name=$(yq '.metadata.name' /kratix/input/object.yaml)
                    name=$(yq '.metadata.namespace' /kratix/input/object.yaml)
                    cat <<EOF > /kratix/output/${name}-${namespace}.tf
                    output "${name}-${namespace}-resource-configure" {
                      value = "some-value"
                    }
                    EOF

Let's break down the Promise above. It starts with the destinationSelectors:

spec:
  destinationSelectors:
    - matchLabels:
        environment: terraform

This Promise will only run on Destinations that have the label environment: terraform. This is the label we set on the Destination earlier. The value of the label is not important, as long as it's the same on the Destination and the Promise. Please refer to Managing Multiple Destinations for more information.

Next, check the spec.promise.configure workflow. You will find the following in the pipeline:

cat <<EOF > /kratix/output/main.tf
output "promise-configure" {
  value = "some-value"
}
EOF

Since the Promise workflow will run only once per Destination, the outputs it generates need only be unique within a single Destination. Your Terraform code will likely to be more complex, but this is a simple example to get you started.

Finally, check the spec.resource.configure workflow. You will find the following:

name=$(yq '.metadata.name' /kratix/input/object.yaml)
name=$(yq '.metadata.namespace' /kratix/input/object.yaml)
cat <<EOF > /kratix/output/${name}-${namespace}.tf
output "${name}-${namespace}-resource-configure" {
  value = "some-value"
}
EOF

Since the Promise workflow will run once per resource, the outputs it generates need to be unique. The example above uses the resource's name and namespace to generate a unique filename and output name. You may need to adapt your code to follow the same pattern.

Adding the SKE TF State Finder Aspect

If you followed the steps above, you should now have a Promise that schedules workloads to TFE. However, given that the Terraform code will be planned and executed by TFE, the users of your Platform won't have access to the Terraform outputs directly.

To solve this, you can use the SKE TF State Finder Aspect. This Aspect will pull the Terraform outputs from TFE and make them available in the Resource's status.

To use the Aspect, you need to add it to your Promise as a second Pipeline in the spec.resource.configure workflow:

apiVersion: platform.kratix.io/v1alpha1
kind: Promise
# ...
spec:
  workflows:
    resource:
      configure:
        - apiVersion: platform.kratix.io/v1alpha1
          kind: Pipeline
          # ... workflow that generates the Terraform configuration files

        - apiVersion: platform.kratix.io/v1alpha1
          kind: Pipeline
          metadata:
            name: output-writer
            namespace: default
          spec:
            rbac:
              permissions:
              - apiGroups:
                  - ""
                resources:
                  - secrets
                verbs:
                  - get
                resourceNamespace: "*" # if the namespace of the secret is static this can be modified
              - apiGroups:
                  - platform.kratix.io
                resources:
                  - workplacements
                verbs:
                  - list
              - apiGroups:
                  - platform.kratix.io
                resources:
                  - destinations
                verbs:
                  - list
                resourceNamespace: "*"
            containers:
              - image: ghcr.io/syntasso/ske-tfstate-finder:v0.1.0
                name: fetch-output
                env:
                - name: TIMEOUT
                  value: "30m"

The TIMEOUT environment variable can be configured to match the amount of time you expect the Terraform plan and apply to take. The Aspect will keep trying to fetch the outputs until the timeout is reached.

If TIMEOUT is not set, the default timeout is 5 minutes.

The Pipeline has a set of additional rbac.permisisons set. It requires these to successfully fetch the outputs from TFE. They are used for:

• Listing WorkPlacements to find the Destination associated with the Resource
• Listing Destinations to find the credentials needed to access TFE
• Getting the Secret with the TFE credentials

Example: Promise with the SKE Aspect included

apiVersion: platform.kratix.io/v1alpha1
kind: Promise
metadata:
  name: example-tf-promise
spec:
  destinationSelectors:
    - matchLabels:
        environment: terraform
  api:
    apiVersion: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    metadata:
      name: terraforms.tfe.ske.io
    spec:
      group: tfe.ske.io
      names:
        kind: Terraform
        plural: terraforms
        singular: terraform
      scope: Namespaced
      versions:
        - name: v1alpha1
          schema:
            openAPIV3Schema:
              type: object
              properties:
                spec:
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
          served: true
          storage: true
  workflows:
    promise:
      configure:
        - apiVersion: platform.kratix.io/v1alpha1
          kind: Pipeline
          metadata:
            name: promise
            namespace: default
          spec:
            containers:
              - image: ghcr.io/syntasso/kratix-pipeline-utility:v0.0.1
                name: generate-output
                command: [ "sh" ]
                args:
                  - -c
                  - |
                    cat <<EOF > /kratix/output/main.tf
                    output "promise-configure" {
                      value = "some-value"
                    }
                    EOF
    resource:
      configure:
        - apiVersion: platform.kratix.io/v1alpha1
          kind: Pipeline
          metadata:
            name: resource
            namespace: default
          spec:
            containers:
              - image: ghcr.io/syntasso/kratix-pipeline-utility:v0.0.1
                name: generate-output
                command: [ "sh" ]
                args:
                  - -c
                  - |
                    name=$(yq '.metadata.name' /kratix/input/object.yaml)
                    name=$(yq '.metadata.namespace' /kratix/input/object.yaml)
                    cat <<EOF > /kratix/output/${name}-${namespace}.tf
                    output "${name}-${namespace}-resource-configure" {
                      value = "some-value"
                    }
                    EOF
        - apiVersion: platform.kratix.io/v1alpha1
          kind: Pipeline
          metadata:
            name: output-writer
            namespace: default
          spec:
            rbac:
              permissions:
              - apiGroups:
                  - ""
                resources:
                  - secrets
                verbs:
                  - get
                resourceNamespace: "*" # if the namespace of the secret is static this can be modified
              - apiGroups:
                  - platform.kratix.io
                resources:
                  - workplacements
                verbs:
                  - list
              - apiGroups:
                  - platform.kratix.io
                resources:
                  - destinations
                verbs:
                  - list
                resourceNamespace: "*"
            containers:
              - image: ghcr.io/syntasso/ske-tfstate-finder:v0.1.0
                name: fetch-output
                env:
                - name: TIMEOUT
                  value: "300s"

info

The ske-tfstate-finder image is hosted in a private registry. To access the image, you will need to use the access token provided by Syntasso and your Kubernetes cluster must have the permissions to pull images from the registry.

You can also set imagePullSecrets in the Pipeline spec.

Writing Resource-specific outputs to the Resource Status

By default, the Aspect will add the Terraform outputs from any .tf files in the Resource's scheduled workloads to the Resource's status.outputs. For example, in the spec.resource.configure workflow above, and assuming the name my-resource in the default namespace, the following would be added to the Resource status:

outputs:
  tfe: # Destination name
    my-resource-default-resource-configure: "some-value" # TFE output name/value

If you want to customise what gets persisted to the status, you can skip this step by setting the SKIP_STATUS environment variable in the Aspect's container to "true". In this case, the *-tfoutput.yaml files in /kratix/metadata will still be written, so that you can access the outputs in the Pipeline.

Accessing outputs in the Pipeline

The Aspect will also write a set of *-tfoutput.yaml files to /kratix/metadata containing all of the Terraform workspace outputs present at the time the Resource or Promise workloads were written to the State Store.

To ensure uniqueness, the naming convention for the output files is <destinationName>-<commitSha>-tfoutput.yaml, where destinationName is the name of the Terraform Destination from which the outputs were read, and commitSha is the SHA of the commit in which the workloads were written to the State Store.

In the most common case, the .tf files are scheduled to a single Terraform Destination in a single previous Pipeline (as in the example above). When this is the case, you can find the output file within the Pipeline as follows:

file=$(find /kratix/metadata -type f -name '*-tfoutput.yaml' | head -n 1)

If there were multiple files, you could narrow your search by Destination name:

file=$(find /kratix/metadata -type f -name '<destinationName>*-tfoutput.yaml' | head -n 1)

The output files will look similar to the following:

- id: wsout-usr3cvFTLCepr4CL
  name: promise-configure
  sensitive: false
  type: string
  value: some-value
  detailedtype: string
- id: wsout-YbSiFpvauuVTeQbV
  name: resource-configure
  sensitive: false
  type: string
  value: some-value
  detailedtype: string
- id: wsout-AIGhavgdsiHVnaHP
  name: other-output
  sensitive: false
  type: string
  value: some-value
  detailedtype: string

Example: Accessing an output and writing it to a custom Resource status field

apiVersion: platform.kratix.io/v1alpha1
kind: Promise
# ...
spec:
  workflows:
    resource:
      configure:
        - apiVersion: platform.kratix.io/v1alpha1
          kind: Pipeline
          # ... workflow that generates the Terraform configuration files

        - apiVersion: platform.kratix.io/v1alpha1
          kind: Pipeline
          metadata:
            name: output-writer
            namespace: default
          spec:
            containers:
              - image: ghcr.io/syntasso/ske-tfstate-finder:v0.1.0
                name: fetch-output
              - image: ghcr.io/syntasso/kratix-pipeline-utility:v0.0.1
                name: custom-status
                command: [ "sh" ]
                args:
                  - -c
                  - |
                    file=$(find /kratix/metadata -type f -name '*-tfoutput.yaml' | head -n 1)
                    value=$(yq e '.[] | select(.name == "promise-configure") | .value' "${file}")
                    echo "customStatusField: ${value}" >> /kratix/metadata/status.yaml

This example uses the kratix-pipeline-utility image, which comes with yq pre-installed. It also assumes a single output file for simplicity.