Tests#

The file ci-test.yml provides job definition for testing projects and collecting test artifacts for reporting.

Helm Tests using kind#

Defines the k8s-integration-tests job that is using the aiv-toolkit/Makefile to run helm tests using kind.

Variables which can be overridden in the extended job:

  • CHART_VALUES: path to a helm values file. Can be empty, in which case chart default is used.

  • CHART_EXTRA_VALUES: extra arguments to helm install, upgrade or template commands. Can be empty, in which case no extra arguments are used. Is useful for setting extra values from the command line, e.g. --set image.tag=latest including secret values from CI variables, e.g. --set secret.key=$SECRET. Note that this means that the chart should be designed to accept secret values, e.g. by using {{ .Values.secret.key }} in the chart templates. See AIV best practices for more details on handling secrets in helm charts.

GUI Tests using playwright#

The toolkit test chart includes an example test using playwright. The tests are still python-based, and follow the same structure as other python tests. See the playwright documentation for more details on GUI writing tests.

Upgrade tests#

By default, the upgrade tests are run using the k8s-upgrade-integration-tests job, which is included in the default pipeline.

By default, the upgrade tests are run using the previous release as the base version. To set a different base version, set the UPGRADE_BASE_REF variable to the desired git reference, e.g. a branch name, tag name or commit hash.

Rollback tests#

If TEST_ROLLBACK is set to true, the upgrade tests will also include a rollback test, which will be run after the upgrade test. The rollback test will check that the cluster can be rolled back to the previous version after the upgrade.

Patching pre-upgrade version#

In rare cases, it may be necessary to apply patches to the pre-upgrade version before testing the upgrade. This can be done by setting the BASE_REF_PATCH_FILES variable to a space-separated list of patch files. The patch files should be in the format of a standard git patch file, and should be relative to the project root directory.

This possibility should be only used as a last result to compensate for changes in the testing infrastructure and not for changes in the project itself. Otherwise the upgrade test does not actually test the upgrade from the previous release.

To make a patch file, you can do the following:

  • Start local cluster with the branch of the MR

  • Checkout the previous release locally, e.g. by doing git checkout <previous-release-tag>

  • Do aiv-deploy helm-upgrade and aiv-deploy helm-test, check that all passes

  • If it does not pass, make changes without committing them

  • Once the changes work, git diff and save the patch into a file

  • Come back to the branch of the MR, add the patch file to the project, and set the BASE_REF_PATCH_FILES variable to point to it.

Manual tests#

If automated verification is not possible, the project may provide explanation of manual verification. Reports of different forms manual verification can be placed in report directory in the root of the project. The following files can be contributed:

  • report/inspection.tex: Inspection report

  • report/demonstration.tex: Demonstration report.

  • report/manual.tex: Manual verification which does not fit into other categories.

Structured claims of manual Test Case execution#

Manual content can specify structured instructions which will be parsed and included in the report:

\ManualTestCase{<Use Case ID>}{<Test Case Result>}{<Comment>}

The following values are allowed for the Test Case Result field:

  • passed

  • failed

  • skipped

The Comment field should contain a short description of the result. It will be included in the report.

Collecting Test Artifacts#

This job should have artifacts with predefined names: report.xml and coverage.xml. Since it is likely that multiple different test jobs will be used, some run in parallel, collect-test-artifacts simply aggregates the results of all other jobs.

Each project can customize the job to include the results of other jobs. By default, the collect-test-artifacts job combines results of the jobs collect-manual-tests and k8s-integration-tests:

collect-test-artifacts:
    needs:
    - job: collect-manual-tests
      artifacts: true
    - job: k8s-integration-tests
      artifacts: true

In the example below, we show how to combine results of three jobs: unittests, collect-manual-tests and k8s-integration-tests. The unittests job should be provided by the project, while collect-manual-tests and k8s-integration-tests are included by the default pipeline.

collect-test-artifacts:
    needs:
    - job: unittests
      artifacts: true
    - job: collect-manual-tests
      artifacts: true
    - job: k8s-integration-tests
      artifacts: true

The following example makes use of a single unittests job. Note that in this case it makes sense to disable the other two jobs, since they are not used for the tests:

collect-test-artifacts:
    needs:
    - job: unittests
      artifacts: true

collect-manual-tests:
    rules:
    - when: never

k8s-integration-tests:
    rules:
    - when: never