Extra info ========== .. _project-structure: 📂 Project Structure -------------------- .. code:: bash /qualpipe-webapp │ ├── kind-dev-config.yml # Kind configuration file for local development ├── chart/ # Helm chart instructions for deployment ├── Makefile # Makefile to build Backend and Frontend │ ├── docs/ # Document folder │ ├── src/ │ └── qualpipe_webapp/ │ ├── backend/ # FastAPI backend │ │ ├── main.py # Main FastAPI app for backend │ │ ├── backends/ # Backend implementations │ │ ├── codegen/ # Code generation templates and scripts │ │ ├── quality_reports/ # Quality report generation templates and scripts │ │ ├── requirements.txt # Backend dependencies │ │ ├── requirements-dev.txt # Backend dependencies for developer │ │ └── Dockerfile # Backend container │ │ │ ├── frontend/ # FastAPI frontend │ │ ├── /templates/ # Template pages │ │ ├── /static/ # Static files (css, js, lib) │ │ ├── main.py # Main FastAPI app for frontend │ │ ├── requirements.txt # Frontend dependencies │ │ ├── requirements-dev.txt # Frontend dependencies for developer │ │ └── Dockerfile # Frontend container │ └── .gitignore .. _pixi-commands: Pixi Commands ------------- All the available Pixi tasks are listed and described in the table below. +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Command: ``pixi`` | Description | +===========================================+=====================================================================================================================================================================+ | ``shell`` | Activate the pixi environment | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run install`` | Install editable pixi environment | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run install-all`` | Install editable pixi environment with all extra dependencies | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run compile-requirements-backend`` | Generate/update backend requirements.txt | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run compile-requirements-frontend`` | Generate/update frontend requirements.txt | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run compile-requirements-backend-dev`` | Generate/update backend requirements-dev.txt | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run compile-requirements-frontend-dev`` | Generate/update frontend requirements-dev.txt | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run compile-requirements`` | Generate/update backend and frontend requirements.txt | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run compile-requirements-dev`` | Generate/update backend and frontend requirements-dev.txt | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run npm-install`` | Install NodeJS dependencies | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run generate-codegen`` | Generate Pydantic models from qualpipe criteria | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run generate-frontend-schema`` | Generate yaml file for javascript frontend validation (combines those produced by ``generate-codegen``) | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run build-backend`` | Build backend image | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run build-backend-dev`` | Build backend *dev* image | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run build-frontend`` | Build frontend image | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run build-frontend-dev`` | Build frontend *dev* image | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run build-images`` | Build backend- and frontend- images | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run build-images-dev`` | Build backend- and frontend- *dev* images | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run kind-create`` | Create local kind cluster | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run kind-delete`` | Delete local kind cluster | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run kind-load-images`` | Load docker images on the cluster | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run kind-load-dev-images`` | Load docker *dev* images on the cluster | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run kind-status`` | Show kubernetes cluster status | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run kind-clean-failed`` | Remove kubernetes resources left from previously failed installation | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run helm-install`` | Install Helm chart | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run helm-dev-install`` | Install Helm chart for local development | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run helm-uninstall`` | | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run helm-upgrade`` | Upgrade Helm chart | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run helm-dev-upgrade`` | Upgrade Helm chart for local development | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run setup`` | Install dependencies, execute ``generate-codegen`` and ``generate-frontend-schema``, and install nodejs dependencies | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run dev-setup`` | Install *dev* dependencies in editable mode, execute ``generate-codegen`` and ``generate-frontend-schema``, and install nodejs dependencies | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run up`` | Production deployment | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run dev-up`` | Deploy the complete development environment (create kubernetes cluster, build docker dev images, install dev helm chart) | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run dev-up-no-build`` | Deploy the complete development environment without rebuilding images | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run restart`` | Re-build images and reinstall Helm chart | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run dev-restart`` | Re-build *dev* images and reinstall Helm chart | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run dev-restart-no-build`` | Reinstall Helm chart | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run stop`` | Stop pods preserving cluster | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run stop-and-delete`` | Stop pods and delete the cluster | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run prune`` | Cleanup dangling Docker images | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run clean-all`` | Stop pods, delete cluster, and remove docker images | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run browser-install`` | Install Playwright browsers for e2e tests | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run test-backend`` | Run backend python unit tests with `Pytest `__ | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run test-frontend-py`` | Run frontend python unit tests with `Pytest `__ | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run test-frontend-js`` | Run frontend javascript unit tests with `Mocha `__ | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run test-frontend-e2e`` | Run frontend javascript end-to-end tests with `Playwright `__ | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run all-tests`` | Run all backend and frontend tests | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run check-backend`` | Check backend health | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run check-frontend`` | Check frontend health | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run check`` | Check backend and frontend health | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run kind-logs-backend`` | Show backend container logs | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run kind-logs-frontend`` | Show frontend container logs | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run kind-logs`` | Show backend and frontend container logs | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run dev-health`` | Check *dev* environment, backend and frontend health, API endpoint, running pods | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run format`` | Run ruff formatting | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run lint`` | Run ruff linting | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ``run lint-fix`` | Run ruff linting and fix | +-------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+ .. _pixi-graph-tasks: Pixi dependency graph tasks ~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code:: bash install ─┐ │ generate-codegen ─┐ │ compile-requirements-backend ──┐ ├─→ compile-requirements compile-requirements-frontend ─┘ compile-requirements-backend-dev ──┐ ├─→ compile-requirements-dev compile-requirements-frontend-dev ─┘ install-all ─┐ │ generate-codegen ─┐ │ compile-requirements-backend-dev ──┐ ├─→ compile-requirements-dev compile-requirements-frontend-dev ─┘ │ │ npm-install ──────────────────────────────────────────┤ ├─→ dev-setup generate-frontend-schema ─────────────────────────────┘ kind-create ─────────────────────────────────┐ build-backend-dev ─────────────┐ │ ├─→ build-images-dev ─┐ build-frontend-dev ─────────────┘ │ ├─→ dev-up helm-dev-install ─────────────────────────────────────┘ kind-create ───────┐ ├─→ dev-up-no-build helm-dev-install ──┘ build-backend ──┐ ├─→ build-images build-frontend ─┘ kind-clean-failed ─┐ ├─→ helm-install kind-load-images ──┘ kind-clean-failed ────┐ ├─→ helm-dev-install kind-load-dev-images ─┘ kind-load-images ──→ helm-upgrade kind-load-dev-images ──→ helm-dev-upgrade helm-uninstall ───┐ build-images ─────┤─→ restart helm-install ─────┘ helm-uninstall ───┐ build-images-dev ─┤─→ dev-restart helm-install ─────┘ helm-uninstall ───┐ ├─→ dev-restart-no-build helm-dev-install ─┘ helm-uninstall ──→ stop kind-delete ─┐ ├─→ stop-and-delete └─→ clean-all test-backend ──────────────┐ test-frontend-py ──────────┤ test-frontend-js ──────────┤ browser-install ──→ test-frontend-e2e └─→ all-tests check-backend ──┐ ├─→ check check-frontend ─┘ kind-logs-backend ──┐ ├─→ kind-logs kind-logs-frontend ─┘ .. _makefile-commands: 🛠️ Makefile Commands -------------------- ``Makefile`` commands, described in the table below, are used only for advanced debug pourposes. +-----------------------+--------------------------------------+ | Command: ``make`` | Description | +=======================+======================================+ | ``dev-forward`` | Manual port-forward (debug/fallback) | +-----------------------+--------------------------------------+ | ``dev-debug-network`` | Debug network cluster/pod/ingress | +-----------------------+--------------------------------------+ | ``dev-trace-request`` | Trace end-to-end request | +-----------------------+--------------------------------------+ | ``dev-debug-setup`` | Advance diagnostic cluster setup | +-----------------------+--------------------------------------+ | ``kind-status-all`` | Complete health check cluster/app | +-----------------------+--------------------------------------+ .. _code-generation-workflow: Code Generation Workflow ------------------------ The project automatically generates Pydantic models from qualpipe criteria classes: .. code:: bash # Generate models manually pixi run generate-codegen # Or use the console script directly (after installation) qualpipe-generate-models --module qualpipe.core.criterion \ --out-generated src/qualpipe_webapp/backend/generated \ --out-schemas src/qualpipe_webapp/frontend/static The code generation creates: - **Python Models**: :file:`src/qualpipe_webapp/backend/generated/__init__.py` - Pydantic models for each criterion type - Validation logic for telescope parameters - Type-safe configuration classes - **JSON/YAML Schemas**: :file:`src/qualpipe_webapp/frontend/static/`` - :file:`criteria_schema.json` - JSON schema for frontend criteria report validation - :file:`criteria_schema.yaml` - YAML schema for configuration Such files are then implemented into :file:`metadata_schema.yaml` executing: .. code:: bash pixi run generate-frontend-schema which is used for complete frontend validations, with all the correct references read from the configuration file :file:`config.js`. The :file:`metadata_schema.yaml` file is auto-generated from a template, so .. important:: Do not edit :file:`metadata_schema.yaml` directly — edit :file:`template_metadata_schema.yaml` instead. Integration with CI/CD ---------------------- The generated models are automatically created during: - **Local development**: with ``pixi run dev-setup`` - **CI/CD pipelines**: code generation runs before CI tests - **Package installation**: post-install hooks generate models .. note:: 🙈 **Generated files are git-ignored** - They're created automatically and should not be committed. 🔄 **Automatic regeneration** - Models update automatically when qualpipe criteria change. .. warning:: ⚠️ **Some tests depend on generated models** - Always run code generation before testing, if your changes had an impact on the models.