Skip to main content

SKE Health Agent

The SKE Health Agent is a health check agent to be installed in Kubernetes Destinations where health checks can be executed.

The Agent is released separately from SKE. Its releases can be found here.

Features​

The Agent will:

  • Schedule the execution of Health Check Workflows from the Destination
  • Persist the data from a health check into a state store

Requirements​

The Agent will write the health information to a state store. Something on the Platform cluster then has to pick those records up, and there are two ways to do that:

  • A GitOps agent on the Platform listening to the state store. This works for both Git and bucket state stores.
  • The SKE Platform Manager reading the records itself using a HealthSource. This needs the state store to be a bucket, and needs no GitOps agent on the Platform. See Move health records onto a bucket.

You need one of the two, not both, though running both at once is supported while you migrate from one to the other.

note

Running both only behaves as you would expect as long as they are not reading the same records from the same place. If a GitOps agent and a HealthSource both deliver the same records from the same store, each resource ends up with two HealthRecords on the Platform and the worse of the two states wins. That is exactly what the migration guide relies on and explains, but it is not a state to sit in by accident.

Install​

The health agent is released separately from SKE. Its releases can be found in the SKE Health Agent release page and it can be installed using the helm chart.

To install the agent using the Helm chart, start by adding the Syntasso Helm repository:

helm repo add syntasso https://syntasso.github.io/helm-charts
helm repo update

Configure values​

Helm can accept values via the values file. Any configuration should be placed in a values.yaml file that is then passed in to the helm install command.

tip

You can see all available configuration in the helm chart repository.

The SKE license​

SKE Health Agent images are hosted in a private registry. To pull them, your cluster needs image pull credentials: either your Syntasso registry token (when pulling from the Syntasso registry directly) or your own credentials if you are using a self-hosted registry.

Option 1: Provide the license token directly

If pulling from the Syntasso registry, set skeLicense in your values file and the chart will create the image pull secret automatically:

skeLicense: LICENSE # provided to you by Syntasso

Option 2: Pre-create the image pull secret

To avoid inlining credentials in your values file, omit skeLicense and instead pre-create a kubernetes.io/dockerconfigjson Secret manually. The credentials will depend on the registry you are pulling from. This could still be your Syntasso registry token, or your own credentials if you are using a self-hosted registry.

For example, when pulling from the Syntasso registry:

kubectl create secret docker-registry syntasso-registry \
  --namespace=k8s-health-agent-system \
  --docker-server=ghcr.io \
  --docker-username=syntasso-pkg \
  --docker-password=<YOUR_LICENSE_TOKEN>

Then reference the secret in your values file. The chart will skip creating the pull secret and use the pre-existing one instead:

imageRegistry:
  imagePullSecret: "syntasso-registry"
warning

For the SKE Health Agent to work, you will need to make sure that your Destination cluster can access the Image Registry.

Image registry for Air-gapped Environments​

You may want to manage the agent images within your own environment either due to air-gapped requirements or other security measures.

To do this, you will need to set up image mirroring and configure the SKE Health Agent chart to pull image from your own registry.

You must ensure that images listed below are mirrored to your own image registry and tagged according to the version:

  • ghcr.io/syntasso/k8s-health-agent
  • ghcr.io/syntasso/health-check-adapter

You can then customise imageRegistry.host in your values file:

# Configuration for the image registry
# Update these values if you are using a private image registry
imageRegistry:
  host: "my-org.registry.io"
  imagePullSecret: "secret-name"
# ...

Deploy the agent​

Next, install the agent by running:

helm install health-agent syntasso/k8s-health-agent --values values.yaml

Set up access to the State Store​

You will need to create a ConfigMap and Secret with the credentials to access the state store. The format depends on the kind of state store you use:

apiVersion: v1
kind: ConfigMap
metadata:
  name: health-state-store-config
  namespace: k8s-health-agent-system
data:
  stateStoreKind: BucketStateStore
  provider: # generic or azure (default: generic)
  endpoint: # address
  bucketName: # bucket name
  authMethod: # accessKey or IAM (default: accessKey); sharedKey for azure
  secretName: <secret name> # required for accessKey
  path: # path within the bucket; optional
  insecure: # "true" or "false" (default: "false"); optional string
  destinationName: # optional; see "Running the agent on multiple Destinations"
---
apiVersion: v1
kind: Secret
metadata:
  name: <secret name>
  namespace: k8s-health-agent-system
stringData:
  accessKeyID: # Access Key ID
  secretAccessKey: # Secret Access Key
warning

The configuration should be created after installing the agent, otherwise the namespace will not exist.

Examples​

apiVersion: v1
kind: ConfigMap
metadata:
  name: health-state-store-config
  namespace: k8s-health-agent-system
data:
  stateStoreKind: BucketStateStore
  endpoint: s3.<AWS_REGION>.amazonaws.com
  bucketName: <BUCKET_NAME>
  authMethod: accessKey
  secretName: health-store-credentials
  path: health
  destinationName: <DESTINATION_NAME>
---
apiVersion: v1
kind: Secret
metadata:
  name: health-store-credentials
  namespace: k8s-health-agent-system
stringData:
  accessKeyID: <YOUR_ACCESS_KEY_ID>
  secretAccessKey: <YOUR_SECRET_ACCESS_KEY>

With authMethod: IAM the agent uses the ambient instance identity and needs no Secret.

Set up the HealthSource on the Platform​

If you are using Azure Blob Storage, or you do not want to run a GitOps agent on the Platform for health records, configure the SKE Platform Manager to read bucket records with a HealthSource. If your existing GitOps agent already syncs health records from Git or an S3-compatible bucket and that works for you, skip this section.

A HealthSource names one place to read from: one bucket, one prefix, one credential. It is cluster-scoped, and it belongs on the Platform cluster, not the Destination. Create it once the agent above is writing.

The credentials are deliberately separate

The Destination needs write access to the store; the Platform needs only read. Give the HealthSource its own credential rather than reusing the agent's, so one Destination's credential cannot reach another Destination's records. On an S3-compatible store, that credential needs nothing beyond list and get under spec.path.

apiVersion: v1
kind: Secret
metadata:
  name: health-store-reader
  namespace: default
stringData:
  accessKeyID: <YOUR_READ_ONLY_ACCESS_KEY_ID>
  secretAccessKey: <YOUR_READ_ONLY_SECRET_ACCESS_KEY>
---
apiVersion: platform.syntasso.io/v1alpha1
kind: HealthSource
metadata:
  name: <DESTINATION_NAME>
spec:
  bucket:
    endpoint: # the same endpoint the agent writes to, without a scheme
    bucketName: # the same bucket
    authMethod: accessKey # or IAM
    secretRef:
      name: health-store-reader
      namespace: default
  path: # the same prefix the agent writes under
  pollInterval: 10m

Check it is working:

kubectl get healthsources
NAME     BUCKET   READY   REASON          OBSERVED   LAST GOOD POLL   AGE
worker-1 kratix   True    PollSucceeded   14         12s              2m

READY should be True, OBSERVED should match the number of resources the Destination reports on, and LAST GOOD POLL should be recent. If READY is not True, the REASON column names the thing at fault.

Examples​

apiVersion: v1
kind: Secret
metadata:
  name: health-store-reader
  namespace: default
stringData:
  accessKeyID: <YOUR_READ_ONLY_ACCESS_KEY_ID>
  secretAccessKey: <YOUR_READ_ONLY_SECRET_ACCESS_KEY>
---
apiVersion: platform.syntasso.io/v1alpha1
kind: HealthSource
metadata:
  name: worker-1
spec:
  bucket:
    endpoint: s3.<AWS_REGION>.amazonaws.com
    bucketName: <BUCKET_NAME>
    authMethod: accessKey
    secretRef:
      name: health-store-reader
      namespace: default
  path: health
  pollInterval: 10m

The endpoint, bucketName and path must match what the agent on that Destination is configured with, or the source polls successfully and observes nothing.

For every field, see the Health Source reference. To move an existing Destination off a GitOps sync without a gap in health, see Move health records onto a bucket.

Running the agent on multiple Destinations​

The agent names each health record after the promise and the resource it checks. If agents on two Destinations check the same resource, they generate the same name and overwrite each other's record.

To keep the records separate, set destinationName in each agent's ConfigMap to a value unique to that Destination; the name of the Kratix Destination is a good choice. The value becomes part of the health record's name, so the Platform keeps one record per Destination and shows them all in the resource's health status.

If you set or change destinationName on an agent that has already written records, the records written under the old name are not cleaned up automatically; delete them from the state store manually.

Example: one Platform, three Destinations​

A Platform cluster with three Destinations registered in Kratix as dev, prod and test. All three run the SKE Health Agent, and all three write to the same bucket under the same health prefix.

destinationName is the part of each file name that keeps the three apart. Without it all three agents write redis-default-example-<hash>.yaml and overwrite one another.

apiVersion: v1
kind: ConfigMap
metadata:
  name: health-state-store-config
  namespace: k8s-health-agent-system
data:
  stateStoreKind: BucketStateStore
  endpoint: s3.<AWS_REGION>.amazonaws.com
  bucketName: kratix
  authMethod: accessKey
  secretName: health-store-credentials
  path: health
  destinationName: dev

Only destinationName differs. Each agent needs its own Secret on its own cluster, but the credentials can be the same, since all three are writing to the same place.

On the Platform, one HealthSource covers all three, because they share a prefix:

apiVersion: platform.syntasso.io/v1alpha1
kind: HealthSource
metadata:
  name: all-destinations
spec:
  bucket:
    endpoint: s3.<AWS_REGION>.amazonaws.com
    bucketName: kratix
    authMethod: accessKey
    secretRef:
      name: health-store-reader
      namespace: default
  path: health
  pollInterval: 10m

A resource checked on more than one Destination now shows one record per Destination in its status.healthStatus, and the worst state across them wins.