CI Pipelines#

This section describes the pre-defined CI workflows available in the toolkit. For the initial setup of the repository, see Quick start.

If you want to include all or most CI job definitions in your project, include the main file ci-functions.yml in your .gitlab-ci.yml and define all required stages:

include:
  - project: 'cta-computing/dpps/aiv/dpps-aiv-toolkit'
    # should be the ref of the latest tag
    ref: 7ebf44c2b02bfd72f596e7ea27ea0e9a1f925903
    file: 'ci-functions.yml'

stages:
  - prepare
  - lint
  - build
  - sign
  - tests
  - publish
  - sonarqube
  - report
  - changelog

If you want to disable certain steps, you can do so by adding a when: never rule:

k8s-integration-tests:
  rules:
    - when: never

Alternatively, you can only include the steps you need by including individual files:

include:
  - project: 'cta-computing/dpps/aiv/dpps-aiv-toolkit'
    # should be the ref of the latest tag
    ref: 7ebf44c2b02bfd72f596e7ea27ea0e9a1f925903
    file:
     - 'ci-common.yml' # common should always be included first
     - 'ci-lint.yml'
     - 'ci-test.yml'

Secrets#

Running all the defined pipelines requires several CI/CD secrets to be setup for the gitlab project.

These variables should be created as “Masked and hidden” and not “Expand variable reference”. With the exception of the PYPI_TOKEN, these are needed also in merge request builds, so “Protect variable” should not be set.

  • GITLAB_TOKEN this needs to be a personal or group access token with read_api permission. This token is required for the pages job deploying the documentation to GitLab Pages and for the build-test-report job.

  • HARBOR_TOKEN and HARBOR_LOGIN. These secrets are needed to publish docker images and helm charts to the harbor. See Build Docker Images and Publish Helm Charts.

    These are the credentials of a Harbor Robot Account. HARBOR_LOGIN needs to be set to the part of the name after the robot$, i.e. dpps+datapipe for robot$dpps+datapipe.

    Create it in the group you want to push or pull (in case of non-public artifacts) artifacts to or from. As this needs the Project Admin permission level, you probably need to ask the AIV team to create a robot for you. The robot account should have the Repository permissions list, read, push, pull, update.

  • PYPI_TOKEN. A PyPI access token for uploading python packages to PyPI, needed by the job pypi. Should be a token only valid for the package(s) in the current repository. “Protect variable” should be enabled here.

  • COSIGN_PRIVATE_KEY. A private key for signing docker images.

  • SONAR_TOKEN. A private token for uploading SonarQube analysis results.

Modules#

Below, you can find the links to the individual CI modules:

On this page

This Page