.. _ci-pipelines:

CI Pipelines
============

This section describes the pre-defined CI workflows available in the toolkit.
For the initial setup of the repository, see :ref:`quick-start`.

If you want to include all or most CI job definitions in your project, include the main
file ``ci-functions.yml`` in your ``.gitlab-ci.yml`` and define all required stages:

.. code:: yaml

   include:
     - project: 'cta-computing/common/aiv-toolkit'
       # should be the ref of the latest tag
       ref: 7ebf44c2b02bfd72f596e7ea27ea0e9a1f925903
       file: 'ci-functions.yml'

   stages:
     - prepare
     - lint
     - build
     - sign
     - tests
     - publish
     - sonarqube
     - report
     - changelog


If you want to disable certain steps, you can do so by adding a ``when: never`` rule:

.. code:: yaml

   k8s-integration-tests:
     rules:
       - when: never

Alternatively, you can only include the steps you need by including individual files:

.. code:: yaml

   include:
     - project: 'cta-computing/common/aiv-toolkit'
       # should be the ref of the latest tag
       ref: 7ebf44c2b02bfd72f596e7ea27ea0e9a1f925903
       file:
        - 'ci-common.yml' # common should always be included first
        - 'ci-lint.yml'
        - 'ci-test.yml'


.. _ci-secrets:

Secrets
-------

Running all the defined pipelines requires several `CI/CD secrets <https://docs.gitlab.com/ci/variables/>`_ to be setup
for the gitlab project.

These variables should be created as "Masked and hidden" and not "Expand variable reference".
With the exception of the ``PYPI_TOKEN``, these are needed also in merge request builds, so "Protect variable" should not be set.

- ``GITLAB_TOKEN`` this needs to be a `personal <https://docs.gitlab.com/user/profile/personal_access_tokens/>`_
  or `group access token <https://docs.gitlab.com/user/group/settings/group_access_tokens/>`_
  with ``read_api`` and ``read_repository`` permissions.
  This token is required for the following jobs: ``pages`` for deploying the documentation to GitLab Pages,
  ``build-test-report`` compiling the release report and ``changelog`` checking for towncrier snippets in MRs.

- ``HARBOR_TOKEN`` and ``HARBOR_LOGIN``.
  These secrets are needed to publish docker images and helm charts to the harbor.
  See :ref:`ci-build` and :ref:`ci-publish`.

  These are the credentials of a `Harbor Robot Account <https://goharbor.io/docs/2.13.0/working-with-projects/project-configuration/create-robot-accounts/>`_.
  ``HARBOR_LOGIN`` needs to be set to the part of the name after the ``robot$``, i.e. ``dpps+datapipe`` for ``robot$dpps+datapipe``.

  Create it in the group you want to push or pull (in case of non-public artifacts) artifacts to or from.
  As this needs the ``Project Admin`` permission level, you probably need to ask the AIV team to create a robot for you.
  The robot account should have the ``Repository`` permissions ``list``, ``read``, ``push``, ``pull``, ``update``.

- ``COSIGN_PRIVATE_KEY``. A private key for signing docker images.

- ``SONAR_TOKEN``. A private token for uploading SonarQube analysis results.

- ``SONAR_API_TOKEN``. A private token for accessing the SonarQube API, needed for the ``build-test-report`` job to fetch SonarQube analysis results and include them in the release report. Different from ``SONAR_TOKEN``.

- ``JAMA_GATEWAY_USER`` and ``JAMA_GATEWAY_PASSWORD`` for accessing use cases and requirements from JAMA via the JAMA gateway.
  Only used by the ``build-test-report`` job. Obtain this information from the maintainers of the toolkit.

.. admonition:: How to use private images in CI
   :collapsible: closed

   In order to use private images from harbor in the CI pipelines as the image for a job when using ``image: <image URL>``,
   you need to do the following steps:

   - Create a robot account in harbor for that project with read permissions on the repository.
   - To get the necessary secret value, run ``jq -n -c --arg auth "$(echo 'robot$<robot name>:<token>' | base64)" '{auths: {"harbor.cta-observatory.org": {auth: $auth}}}'``
   - Create the CI/CD env variable with type "File", masked and hidden, key ``DOCKER_AUTH_CONFIG``,
     value the minified json output of previous step.
   - No extra configuration is needed in the CI pipeline, as the GitLab runner will automatically
     use the ``DOCKER_AUTH_CONFIG`` variable to authenticate with the registry when pulling images.

   The pipeline should be able to pull the private image, both on the shared DESY runner and the test cluster toolkit runner.


Custom secret values in tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Some tests may require custom secret values, e.g. to test integration with external services (this practice is generally discouraged in favor of using mock services or test environments).
In this case, the secrets can be included in the CI/CD variables and passed to the helm charts using the ``CHART_EXTRA_VALUES`` variable, see :ref:`ci-tests` for more details.


Modules
-------

Below, you can find the links to the individual CI modules:

.. toctree::
    :maxdepth: 1

    precommit
    lint
    build
    cwl
    changelog
    docs
    tests
    sonar
    test-report
    pypi
    publish