CI Pipelines#

This section describes the pre-defined CI workflows available in the toolkit. For the initial setup of the repository, see Quick start.

If you want to include all or most CI job definitions in your project, include the main file ci-functions.yml in your .gitlab-ci.yml and define all required stages:

include:
  - project: 'cta-computing/common/aiv-toolkit'
    # should be the ref of the latest tag
    ref: 7ebf44c2b02bfd72f596e7ea27ea0e9a1f925903
    file: 'ci-functions.yml'

stages:
  - prepare
  - lint
  - build
  - sign
  - tests
  - publish
  - sonarqube
  - report
  - changelog

If you want to disable certain steps, you can do so by adding a when: never rule:

k8s-integration-tests:
  rules:
    - when: never

Alternatively, you can only include the steps you need by including individual files:

include:
  - project: 'cta-computing/common/aiv-toolkit'
    # should be the ref of the latest tag
    ref: 7ebf44c2b02bfd72f596e7ea27ea0e9a1f925903
    file:
     - 'ci-common.yml' # common should always be included first
     - 'ci-lint.yml'
     - 'ci-test.yml'

Secrets#

Running all the defined pipelines requires several CI/CD secrets to be setup for the gitlab project.

These variables should be created as “Masked and hidden” and not “Expand variable reference”. With the exception of the PYPI_TOKEN, these are needed also in merge request builds, so “Protect variable” should not be set.

  • GITLAB_TOKEN this needs to be a personal or group access token with read_api and read_repository permissions. This token is required for the following jobs: pages for deploying the documentation to GitLab Pages, build-test-report compiling the release report and changelog checking for towncrier snippets in MRs.

  • HARBOR_TOKEN and HARBOR_LOGIN. These secrets are needed to publish docker images and helm charts to the harbor. See Build Docker Images and Publish Helm Charts.

    These are the credentials of a Harbor Robot Account. HARBOR_LOGIN needs to be set to the part of the name after the robot$, i.e. dpps+datapipe for robot$dpps+datapipe.

    Create it in the group you want to push or pull (in case of non-public artifacts) artifacts to or from. As this needs the Project Admin permission level, you probably need to ask the AIV team to create a robot for you. The robot account should have the Repository permissions list, read, push, pull, update.

  • COSIGN_PRIVATE_KEY. A private key for signing docker images.

  • SONAR_TOKEN. A private token for uploading SonarQube analysis results.

  • SONAR_API_TOKEN. A private token for accessing the SonarQube API, needed for the build-test-report job to fetch SonarQube analysis results and include them in the release report. Different from SONAR_TOKEN.

  • JAMA_GATEWAY_USER and JAMA_GATEWAY_PASSWORD for accessing use cases and requirements from JAMA via the JAMA gateway. Only used by the build-test-report job. Obtain this information from the maintainers of the toolkit.

How to use private images in CI

In order to use private images from harbor in the CI pipelines as the image for a job when using image: <image URL>, you need to do the following steps:

  • Create a robot account in harbor for that project with read permissions on the repository.

  • To get the necessary secret value, run jq -n -c --arg auth "$(echo 'robot$<robot name>:<token>' | base64)" '{auths: {"harbor.cta-observatory.org": {auth: $auth}}}'

  • Create the CI/CD env variable with type “File”, masked and hidden, key DOCKER_AUTH_CONFIG, value the minified json output of previous step.

  • No extra configuration is needed in the CI pipeline, as the GitLab runner will automatically use the DOCKER_AUTH_CONFIG variable to authenticate with the registry when pulling images.

The pipeline should be able to pull the private image, both on the shared DESY runner and the test cluster toolkit runner.

Custom secret values in tests#

Some tests may require custom secret values, e.g. to test integration with external services (this practice is generally discouraged in favor of using mock services or test environments). In this case, the secrets can be included in the CI/CD variables and passed to the helm charts using the CHART_EXTRA_VALUES variable, see Tests for more details.

Modules#

Below, you can find the links to the individual CI modules: