Python#

Python is a strongly and dynamically typed, interpreted general programming language with strong interoperability with several compiled languages. It offers are rich ecosystem, particularly for science and astronomy and is one of the most popular languages.

Large parts of the CTAO software are written in Python, and it is considered the default programming language if there are no special requirements for a given project that require a different choice.

Reference Version#

The current CTAO reference version for Python is CPython 3.12.

Official packages are available for Alma Linux 9, the current reference OS,:

# dnf install python3.12

Alternatively, pyenv or the conda-forge package using micromamba might be used.

It is recommended that libraries and tools that are of wider interest to the consortium and the community are following NEP 29. This NumPy enhancement proposal provides a detailed schedule on which Python versions to support over time and is also followed by Astropy, Gammapy and ctapipe.

All projects shall indicate the python versions they are compatible with, see Build Systems and Packaging.

Libraries#

Todo

Add libraries

Tools and Configurations#

Template Project#

A template project that is kept up-to-date with the recommendations given here is available in the CTAO Gitlab.

Unit Tests#

The recommended framework for unit tests is pytest, using the in-tree layout to enable testing an installed package using pytest --pyargs <package-name>.

Coverage should be reported using pytest-cov and examples included in the documentation should also be run as part of the tests using the doctest integration of pytest.

Tests can be run in parallel on multiple cores using the pytest-xdist plugin.

A full run of the test-suite including coverage report in HTML format for local viewing could look like this:

$ pytest src/ docs/ --doctest-modules --doctest-glob="*.rst" --cov --cov-report=html

Documentation#

The recommended tool to provide API and user documentation for Python projects is sphinx. API documentation should be done following the Astropy documentation guidelines in docstrings in the code that is included automatically into the sphinx documentation. The recommended theme is the pydata-sphinx-theme.

The documentation should be published to GitLab pages. The template project uses a custom python tool to enable multiple versions of the documentation on GitLab pages with a version selector.

Changelog#

The documentation should also contain a change log detailing changes between releases.

A tool that helps maintainers in creating the changelog is towncrier, which works by including categorized, small text snippets for each change. These snippets should be added as part of the corresponding merge request. If merge request !5 introduces a new feature, it should also add a description of this new feature in docs/changes/5.feature.rst.

Just before a new release is made, the maintainer renders the snippets into a full changelog using the towncrier tool, automatically creating the change log of all merged changes:

$ towncrier build --version <version>

This will create a new section in the changelog for the corresponding release. The maintainer should then proof-read / edit the rendered changelog, after which it can be committed.

For the latest version of the docs, i.e. the current main branch, the snippets can be included into the documentation using the sphinx-changelog extension.

The relevant parts in the template project are

the tool.towncrier and the project.optional-dependencies sections in pyproject.toml
the docs/conf.py sphinx configuration (extensions and exclude_patterns)
the docs/changes directory, which contains the template.rst for the change log
the docs/changelog.rst page for including the changelog in the documentation

Static Code Checks#

The ruff tool is a fast and flexible linter supporting many rules to statically check python code bases, e.g. for common errors like unused imports or undefined variables.

The pre-commit tool should be used to run the same checks locally for developers before they commit changes and also in the CI. The pre-commit tool also comes with general, language agnostic checks that should be enabled.

See the configuration in the pyproject.toml, .pre-commit.yml and .gitlab-ci.yml of the template project.

Auto-Formatting#

Python code should be auto-formatted, including the sorting of import statements according to common conventions. ruff also provides formatting, including import statements. See the corresponding [tool.ruff] sections in the pyproject.toml of the template project.

Build Systems and Packaging#

Unfortunately, build systems and packaging is a complex story in Python, there is not the one official tool that supports all use cases and for each part of the build and release process, several solutions exist.

Python packages can be differentiated into two large categories: those that only consist of pure Python code and those that contain compiled, native extensions.

Building, packaging and distribution is much easier for pure Python packages, as they do not require building and distributing the platform specific binaries that are required for native extensions.

Naming Conventions#

There are a couple of different names important for python packages:

the name of the Project in the GitLab
the name of the PyPI distribution package, what you pass to pip install <distribution package name>
the name of the python packages and modules shipped in the distribution, this is what determine the name in import <module name>.
(Optional) the name of the conda package

It is recommended that all of these are as similar as possible, preferably equal. There however are situations, where deviations are needed. E.g. the repository and the conda package of python bindings to a C++ project called foo might be named foo-python, while the PyPI package and the import name are just foo.

Official ctao software should have PyPI package name ctao-<wp>-<package name>, e.g. ctao-dpps-datapipe.

Pure-Python Packages#

The recommended packaging solution for pure python packages is setuptools, using the pyproject.toml file specified in PEP 517, PEP 518 and PEP 621.

Python-bindings to C++ Projects#

The recommended solution for generating python bindings for C++ projects is nanobind. pybind11 might be used as a fallback in case of the C++ code not being able to fulfill the more restrictive requirements of nanobind.

scikit-build-core should be used as the build system for such projects.

The protozfits-python repository is an example of this setup.

Versioning#

Versioning should be implemented by using git tags for releases and including this information in the package using setuptools_scm. The template project contains a solution originally developed for Astropy, that avoids some common pitfalls. The version should be exposed at the top-level module as <module>.__version__ string attribute and all command line tools should expose a --version switch.

PyPI#

All python packages should be published to the Python Packaging Index, both as source distribution and as wheels.

For pure python packages, providing source distribution and the platform independent wheel is straight forward.

For packages containing compiled extensions, platform specific binary wheels should be provided, at least for the manylinux standard compatible with the production computing environment.

This can be achieved through the docker images provided by the manylinux project to build standard wheels. See .gitlab-ci.yaml in protozfits-python to see how to build manylinux wheels using the gitab-ci.

conda-forge#

Additionally to PyPi, packages – especially packages that comprise binary extensions – shall be published using the conda-forge infrastructure. See the conda-forge documentation for how to contribute a package and the protozfits feedstock repository for a CTA binary project published through conda-forge.