Sub-pipeline

In the Radix pipeline, optionally a sub-pipeline can be run. It is run after "Build components step" (if components need to be built) or after "Prepare pipeline" step. This sub-pipeline is based on the Tekton CI/CD framework.

tip

Note on nomenclature! The content in this guide concerns a Tekton pipeline which is defined as a pipeline within a parent Radix pipeline. In the context of the Radix platform, a Tekton pipeline is referred to as a sub-pipeline or Tekton pipeline, while the parent Radix pipeline is referred to as a pipeline or Radix pipeline.

Configure sub-pipeline

Sub-pipeline is configured with file pipeline.yaml. This file will in turn have references to one or several other YAML-files which define tasks for the sub-pipeline.

Pipeline and task files

• Default folder for sub-pipeline is tekton, next to the Radix configuration file of the application.
• Default name for the sub-pipeline is pipeline.yaml.
• Files with sub-pipeline task configurations should be located next to the file pipeline.yaml.

Example: a sub-pipeline pipeline.yaml refers to tasks in files clone.yaml, build.yaml, migration.yaml

/
├── component1
├── component2
├── tekton/
│   ├── pipeline.yaml
│   ├── clone.yaml
│   ├── build.yaml
│   └── migration.yaml
└── radixconfig.yaml

Suppose an app has a sub-pipeline defined, like the example above. Within the Radix pipeline step "Prepare pipeline", the following logic will be executed:

1. the sub-pipeline defined in pipeline.yaml is loaded
2. task files referred to inside the pipeline.yaml file are loaded, which are clone.yaml, build.yaml and migration.yaml
3. if an error occurred during loading of the sub-pipeline or its tasks, the step "Prepare pipeline" and entire Radix pipeline job is considered failed
4. the sub-pipeline is run
5. if any step of any task is failed - the pipeline gets status "failed", the step "Run sub-pipeline" and entire Radix pipeline job gets the status "failed"

Errors in stage (3) can be caused by:

• an invalid format of a sub-pipeline or tasks files
• an empty list of tasks in a sub-pipeline
• a missing task, referenced in a sub-pipeline
• empty step list in a task

Follow the Tekton documentation to configure a sub-pipeline and its tasks, particularly Tekton pipeline and task documentation.

Custom values supplied by Radix

Radix provides a set of runtime values that can be referenced directly in sub-pipeline definitions.

• $(radix.git-deploy-key) is the placeholder of a read-only volume created by Radix. Mount this volume in a task step when the step needs SSH credentials for Git operations, such as cloning the application's repository. Refer to example Sub-pipeline with GitHub deploy keys for usage.

• $(radix.build-secrets) is the placeholder for the application's configured build secrets. Use it when you want to expose build secrets to a task step through env, envFrom, or mounted files. Refer to example Sub-pipeline with build secrets for usage.

• $(params.radix) is a reserved parameter injected by Radix into the sub-pipeline and automatically forwarded to every task. Do not define a parameter named radix yourself in the pipeline or task files. Refer to example Sub-pipeline with GitHub deploy keys for usage.

  You can reference individual properties with the syntax $(params.radix.<property-name>), for example $(params.radix.git-commit).

  The following properties are available on $(params.radix):

  Property	Description
  pipeline-type	Type of the parent Radix pipeline that is running the sub-pipeline, for example deploy, build-deploy, or promote.
  environment	Name of the target environment for the current sub-pipeline run.
  git-ssh-url	SSH clone URL for the application repository. Use this together with $(radix.git-deploy-key) when a task needs to clone the repository over SSH.
  git-ref	Git branch or tag for the pipeline run.
  git-ref-type	Type of git-ref. The value is branch or tag.
  git-commit	Resolved Git commit SHA for the pipeline run.
  git-tags	Space-separated list of Git tags that point to git-commit. This value is empty when the commit has no matching tags.

• $(params.radix-image.<component-name>) references the fully qualified image built by Radix for a specific component or job. Use this in a task step's image field to run the step using the same image as the named component or job, without hardcoding registry paths or image tags.

  Replace <component-name> with the name of the component or job as defined in radixconfig.yaml. The parameter is injected automatically by Radix and forwarded to every task in the sub-pipeline.

  This is useful for tasks that need to execute code that is part of your application such as database migrations, schema validation or smoke tests. The step runs the exact image being deployed, with the app's runtime and tooling already present.

  apiVersion: tekton.dev/v1
  kind: Task
  metadata:
    name: run-migrations
  spec:
    steps:
      - image: $(params.radix-image.server)
        name: run-db-migrations
        script: /bin/some-binary migrate-database

  In this example, server is the name of a component defined in radixconfig.yaml. The step runs using that component's image, which was built earlier in the same Radix pipeline run.

  tip

  The $(params.radix-image.<component-name>) parameter is available in build-deploy, and promote pipeline runs.

Limitations

In Radix platform, the following limitations are applied to sub-pipelines:

• sub-pipeline does not support workspaces. However, it is possible to use volumes in sub-pipeline tasks.
• sub-pipeline Task step cannot mount secrets as volumes, with some exceptions:
  • the secret to access private image repository, which is mounted automatically
  • build secrets
• sub-pipeline Task step cannot run as a privileged container (e.g. cannot run as root) or with a host network
  • if a container image used in a step is configured to run as a root, this user can (and should) be changed to a non-root user with a field securityContext.runAsUser in the step definition, securityContext.runAsGroup is also supported. runAsUser and runAsGroup cannot have value 0 (= root user).

  apiVersion: tekton.dev/v1
  kind: Task
  metadata:
  name: my-task
  spec:
    steps:
      - image: alpine
        name: show-user-id
        script: |
          #!/usr/bin/env sh
          id
          :
        securityContext:
          runAsUser: 1000

  Suggestion

  The following command can be used to find out with which user the image runs its container:

  docker run -it alpine id

Hints

• Tekton pipeline and tasks can be developed and tested on PC within local Kubernetes cluster.

• Name of a task, file name of a task and a name of a task in the Tekton pipeline task list - all can be different. It is important only to use the same name in the task field metadata.name and in the Tekton pipeline field taskRef.name. In the example below it is name build-image:

• File pipeline.yaml:

  apiVersion: tekton.dev/v1
  kind: Pipeline
  metadata:
    name: pipeline
  spec:
    tasks:
      - name: some-build-task
        taskRef:
          name: build-image

  File build-image-task.yaml:

  apiVersion: tekton.dev/v1
  kind: Task
  metadata:
    name: build-image
  spec:
    steps:
      ...

• It is not important in which order to put tasks in the sub-pipeline - tasks can run in parallel or in sequences, defined by fields runAfter, conditions, from.

• If a task has a field runAfter - it will be started only when all tasks, referenced in the field runAfter are complete.

• Task details:

  • Each sub-pipeline task runs in its own Kubernetes pod (replica).

  • Task step runs in its own container of this task's pod.

  • Task step can be configured individually: which container image and how many resources to use, how to proceed on an error, specify a timeout, if the task runs script - is it bash or PowerShell script, etc.

  • When task step uses script - it would be recommended to finish this script with the no-op command: put : (colon) on the last new line of the script. It will help to avoid some irrelevant errors (e.g. in the example below: run of this task raises an error, when the command printenv|grep "DB" is on the last line of the script and there are no environment variables with a fragment "DB" in names). Or just put a command like echo ""

    spec:
      steps:
      - image: alpine
        name: show-db-env-vars
        script: |
          #!/usr/bin/env sh
          printenv|grep "DB"
          :

Examples

• Simple sub-pipeline
• Sub-pipeline with multiple tasks
• Sub-pipeline with build environment variables
• Sub-pipeline with build environment variables for environments
• Sub-pipeline with build secrets
• Sub-pipeline with GitHub deploy keys
• Sub-pipeline with Azure Workload Identity