CWL === AIV toolkit provides a set of tools to lint, test, package, publish and integrate CWL tools and workflows. CWL files are assumed to be in a fixed directory of the repository, ``workflows``. If there are no CWL files in the repository (``workflows/**/*.cwl``), all CI jobs related to CWL will be skipped. All files in ``workflows`` which match the pattern ``*.cwl`` are considered as CWL files. The files may be in nested directories. These files will be: - Linted using ``cwltool --validate`` command. - If possible, converted to visualizations and stored in the artifacts. - Packaged and uploaded to DPPS harbor. - In all packaged CWL files, any reference to ``dockerPull`` will be replaced with the image built in the current CI pipeline. Testing CWL ----------- There is no special functionality for testing CWL files in the AIV toolkit. They should be tested as any other code in the repository. It is responsibility of the DPPS developer to ensure that the CWL files are tested. The developer can select the location of the tests as needed, and implement output verification as required. The constraint imposed by the AIV toolkit is that ``pytest`` will be able to discover the tests according to the provided configuration (typically in ``pyproject.toml`` file). Customizing CWL CI Jobs ----------------------- It is possible to extend, override, and customize the CWL CI jobs. The following jobs are available in the AIV toolkit: ``discover-lint-cwl`` lints and renders the CWL files. ``build-push-cwl`` pushes the CWL files to the DPPS harbor. Fetching CWL files from harbor ------------------------------ We are storing CWL in harbor as generic OCI artifacts, similar to docker images and helm charts, in repositories with ``cwl`` suffix, e.g. ``calibpipe-cwl``. Harbor UI provides only limited visualizations of the CWL files or any other artifacts. In order to view and explore the CWL files stored in harbor, we need to fetch them using OCI protocol. We recommend to use ORAS CLI tool to manipulate OCI artifacts. It can be installed following the instructions in the `ORAS documentation `_. To pull the CWL files from harbor for example, you can use the following command: .. code:: bash oras pull harbor.cta-observatory.org/dpps/calibpipe-cwl:v0.1.0-rc9 And the files will be stored in the current directory (compare to ``docker pull`` and ``helm pull`` commands). For viewing convenience, the most recent CWL artifacts from all pipeline subsystems are synchronized to a `gitlab repository `_ every hour at 07 minutes.