.. _ci-tests:

Tests
=====

The file ``ci-test.yml`` provides job definition for testing projects and collecting
test artifacts for reporting.

Helm Tests using ``kind``
-------------------------

Defines the ``k8s-integration-tests`` job that is using the
``aiv-toolkit/Makefile`` to run helm tests using ``kind``.

Variables which can be overridden in the extended job:

- ``CHART_VALUES``: path to a helm values file. Can be empty, in which case chart default is used.

- ``CHART_EXTRA_VALUES``: extra arguments to helm install, upgrade or template commands. Can be empty, in which case no extra arguments are used. Is useful for setting extra values from the command line, e.g. ``--set image.tag=latest`` including secret values from CI variables, e.g. ``--set secret.key=$SECRET``. Note that this means that the chart should be designed to accept secret values, e.g. by using ``{{ .Values.secret.key }}`` in the chart templates. See :ref:`best-practices` for more details on handling secrets in helm charts.

GUI Tests using ``playwright``
------------------------------

The toolkit `test chart <https://gitlab.cta-observatory.org/cta-computing/common/aiv-toolkit/-/tree/main/test-chart>`_ includes an example test using ``playwright``.
The tests are still python-based, and follow the same structure as other python tests. See the
`playwright documentation <https://playwright.dev/python/docs/intro>`_ for more details on GUI writing tests.


Upgrade tests
-------------

By default, the upgrade tests are run using the ``k8s-upgrade-integration-tests`` job, which is included in the default pipeline.

By default, the upgrade tests are run using the previous release as the base version.
To set a different base version, set the ``UPGRADE_BASE_REF`` variable to the desired git reference, e.g. a branch name, tag name or commit hash.


Rollback tests
~~~~~~~~~~~~~~

If ``TEST_ROLLBACK`` is set to ``true``, the upgrade tests will also include a rollback test, which will be run after the upgrade test. The rollback test will check that the cluster can be rolled back to the previous version after the upgrade.

Patching pre-upgrade version
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In rare cases, it may be necessary to apply patches to the pre-upgrade version before testing the upgrade. This can be done by setting the ``BASE_REF_PATCH_FILES`` variable to a space-separated list of patch files. The patch files should be in the format of a standard git patch file, and should be relative to the project root directory.

This possibility should be only used as a last result to compensate for changes in the testing infrastructure and not for changes in the project itself. Otherwise the upgrade test does not actually test the upgrade from the previous release.

To make a patch file, you can do the following:

- Start local cluster with the branch of the MR
- Checkout the previous release locally, e.g. by doing ``git checkout <previous-release-tag>``
- Do ``aiv-deploy helm-upgrade`` and ``aiv-deploy helm-test``, check that all passes
- If it does not pass, make changes without committing them
- Once the changes work, ``git diff`` and save the patch into a file
- Come back to the branch of the MR, add the patch file to the project, and set the ``BASE_REF_PATCH_FILES`` variable to point to it.

Manual tests
------------

If automated verification is not possible, the project may provide explanation of manual verification.
Reports of different forms manual verification can be placed in ``report`` directory in the root of the project.
The following files can be contributed:

- ``report/inspection.tex``: Inspection report
- ``report/demonstration.tex``: Demonstration report.
- ``report/manual.tex``: Manual verification which does not fit into other categories.

Structured claims of manual Test Case execution
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Manual content can specify structured instructions which will be parsed and included in the report:

.. code:: latex

   \ManualTestCase{<Use Case ID>}{<Test Case Result>}{<Comment>}


The following values are allowed for the ``Test Case Result`` field:

- ``passed``
- ``failed``
- ``skipped``

The ``Comment`` field should contain a short description of the result. It will be included in the report.


.. _collect-test-artifacts:

Collecting Test Artifacts
-------------------------

This job should have artifacts with predefined names: ``report.xml`` and ``coverage.xml``.
Since it is likely that multiple different test jobs will be used, some run in parallel,
``collect-test-artifacts`` simply aggregates the results of all other jobs.

Each project can customize the job to include the results of other jobs.
By default, the ``collect-test-artifacts`` job combines results of the jobs ``collect-manual-tests``
and ``k8s-integration-tests``:

.. code:: yaml

    collect-test-artifacts:
        needs:
        - job: collect-manual-tests
          artifacts: true
        - job: k8s-integration-tests
          artifacts: true


In the example below, we show how to combine results of three jobs:
``unittests``, ``collect-manual-tests`` and ``k8s-integration-tests``.
The ``unittests`` job should be provided by the project,
while ``collect-manual-tests`` and ``k8s-integration-tests`` are included by the default pipeline.

.. code:: yaml

    collect-test-artifacts:
        needs:
        - job: unittests
          artifacts: true
        - job: collect-manual-tests
          artifacts: true
        - job: k8s-integration-tests
          artifacts: true


The following example makes use of a single ``unittests`` job.  Note that in this case it makes sense to disable the other two jobs, since they are not used for the tests:

.. code:: yaml

    collect-test-artifacts:
        needs:
        - job: unittests
          artifacts: true

    collect-manual-tests:
        rules:
        - when: never

    k8s-integration-tests:
        rules:
        - when: never