CWL#

AIV toolkit provides a set of tools to lint, test, package, publish and integrate CWL tools and workflows.

CWL files are assumed to be in a fixed directory of the repository, workflows.

If there are no CWL files in the repository (workflows/**/*.cwl), all CI jobs related to CWL will be skipped.

All files in workflows which match the pattern *.cwl are considered as CWL files. The files may be in nested directories.

These files will be:

  • Linted using cwltool --validate command.

  • If possible, converted to visualizations and stored in the artifacts.

  • Packaged and uploaded to DPPS harbor.

    • In all packaged CWL files, any reference to dockerPull will be replaced with the image built in the current CI pipeline.

Testing CWL#

There is no special functionality for testing CWL files in the AIV toolkit. They should be tested as any other code in the repository.

It is responsibility of the DPPS developer to ensure that the CWL files are tested. The developer can select the location of the tests as needed, and implement output verification as required.

The constraint imposed by the AIV toolkit is that pytest will be able to discover the tests according to the provided configuration (typically in pyproject.toml file).

Customizing CWL CI Jobs#

It is possible to extend, override, and customize the CWL CI jobs. The following jobs are available in the AIV toolkit:

discover-lint-cwl lints and renders the CWL files.

build-push-cwl pushes the CWL files to the DPPS harbor.

Fetching CWL files from harbor#

We are storing CWL in harbor as generic OCI artifacts, similar to docker images and helm charts, in repositories with cwl suffix, e.g. calibpipe-cwl.

Harbor UI provides only limited visualizations of the CWL files or any other artifacts. In order to view and explore the CWL files stored in harbor, we need to fetch them using OCI protocol.

We recommend to use ORAS CLI tool to manipulate OCI artifacts. It can be installed following the instructions in the ORAS documentation. To pull the CWL files from harbor for example, you can use the following command:

oras pull harbor.cta-observatory.org/dpps/calibpipe-cwl:v0.1.0-rc9

And the files will be stored in the current directory (compare to docker pull and helm pull commands).

For viewing convenience, the most recent CWL artifacts from all pipeline subsystems are synchronized to a gitlab repository every hour at 07 minutes.