Metadata-Version: 2.4
Name: ichec_django_core
Version: 6.1.0
Summary: Library of base Django app building blocks and utilities.
Author-email: Irish Centre for High End Computing <platformengineering@ichec.ie>
License: MIT
Project-URL: Repository, https://git.ichec.ie/platform-engineering/modules/web/ichec-django-core
Project-URL: Homepage, https://git.ichec.ie/platform-engineering/modules/web/ichec-django-core
Keywords: Web Application,Django
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.14
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: django>=6.0
Requires-Dist: markdown
Requires-Dist: pillow
Requires-Dist: pyyaml
Requires-Dist: pydantic
Requires-Dist: djangorestframework>=3.18
Requires-Dist: django-filter
Requires-Dist: django-countries
Requires-Dist: django-cors-headers
Requires-Dist: django-downloadview
Requires-Dist: django-prometheus>=2.5
Requires-Dist: mozilla-django-oidc
Requires-Dist: drf-spectacular>=0.30.0
Requires-Dist: django-fsm-2>=4.0
Requires-Dist: django-axes>=8
Requires-Dist: django-health-check>=4
Provides-Extra: async
Requires-Dist: celery>=5.3; extra == "async"
Requires-Dist: redis>=5.0; extra == "async"
Provides-Extra: container
Requires-Dist: psycopg2-binary; extra == "container"
Requires-Dist: gunicorn; extra == "container"
Provides-Extra: s3
Requires-Dist: django-storages[s3]; extra == "s3"
Provides-Extra: keycloak
Requires-Dist: python-keycloak; extra == "keycloak"
Provides-Extra: codegen
Requires-Dist: datamodel-code-generator[ruff]==0.83.0; extra == "codegen"
Provides-Extra: workflows
Requires-Dist: django-simple-history>=3.13.0; extra == "workflows"
Provides-Extra: types
Requires-Dist: types-requests; extra == "types"
Provides-Extra: sentry
Requires-Dist: sentry-sdk[django]>=2; extra == "sentry"
Dynamic: license-file

# ICHEC Django Core

`ichec-django-core` is the base for ICHEC's Django web apps. It gives your app
secure default settings, member and organisation models, sign-in through
Keycloak, and a REST API for all of these. You write the parts that are specific
to your app.

The [reference app](https://git.ichec.ie/platform-engineering/infrastructure/reference-app)
is a complete app built on it, deployed with the `ichec.platform` Ansible
collection.

## What you get

- **Members and organisations.** Every Django user is a `Member`. Members
  belong to organisations and groups, get notifications, and can export or
  erase their personal data.
- **Sign-in.** Keycloak through OpenID Connect, Django's own login for local
  work, and personal API tokens for scripts.
- **A REST API.** Each model has an endpoint under `/api/`, with an OpenAPI 3.1
  schema, ETags, and errors as RFC 9457 problem details.
- **Access rules.** Each model declares once who may do what with it. Every
  list, check and `available_actions` field comes from that declaration.
- **Status changes.** A model's status changes only through named transitions.
  Each one is its own endpoint, such as `POST /api/applications/{id}/submit/`.
- **Files.** Uploads, image thumbnails, and delivery through Django, nginx or
  S3, with resumable uploads through tusd.
- **Forms.** An opt-in app for forms defined by JSON Schema, with drafts and
  file answers.
- **Workflows.** An opt-in app for review flows held as data, so each call
  sets its own steps: reviews, decisions, waits and checklists, with ranking.

## Try the example app

The [app](./app) directory is a minimal portal built on the library. Run it
locally to see what you get before you write any code. You need
[uv](https://docs.astral.sh/uv/) (`brew install uv`).

```shell
git clone https://git.ichec.ie/platform-engineering/modules/web/ichec-django-core.git
cd ichec-django-core
uv run tox exec -e dev -- python manage.py migrate
uv run tox exec -e dev -- python manage.py createsuperuser
uv run tox -e dev
```

Sign in at <http://localhost:8000/accounts/login/> as the admin you created,
then open <http://localhost:8000/api/> to browse the API. The `dev` env in
[pyproject.toml](./pyproject.toml) holds the development settings.

## Use it in your app

This guide assumes you can build a basic Django app. If not, work through the
[Django tutorial](https://www.djangoproject.com/start/) first.

Add the library to your dependencies:

```shell
uv add ichec-django-core
```

### Settings

Import the library's settings, then set the few values that depend on your
project layout. Your settings file then looks like [app/settings.py](./app/settings.py):

```python
from pathlib import Path

from ichec_django_core import settings
from ichec_django_core.settings import *

BASE_DIR = Path(__file__).resolve().parent.parent

ROOT_URLCONF = "app.urls"
WSGI_APPLICATION = "app.wsgi.application"
ASGI_APPLICATION = "app.asgi.application"

TEMPLATES = settings.get_templates(BASE_DIR)
DATABASES = settings.get_databases(BASE_DIR)
STATIC_ROOT = settings.get_static_root(BASE_DIR)
MEDIA_ROOT = settings.get_media_root(BASE_DIR)
```

Override any other setting below the import.

The library reads its settings from environment variables. Django won't start
without `DJANGO_SECRET_KEY` and `DJANGO_ALLOWED_HOSTS`, or without
`DJANGO_SQL_PASSWORD` when it uses a database server rather than SQLite.
For local work, set them in a tox env, as the `dev` env in
[pyproject.toml](./pyproject.toml) does. In production, the deployment passes
them to the container.
[settings.py](./src/ichec_django_core/settings.py) lists every variable and its
default.

The environment holds deployment settings only. What an admin changes, the
portal's name, contact address and help link, is in the database, at
`/api/portal/`, for holders of `change_portal`. The manifest sends it to the
frontend. Members with the `receive_signup_alerts` permission get an email
when someone new signs in.

Logs go to the console, which a container sends to journald. To also write a
rotating log file, set `DJANGO_LOG_ROOT` to a directory the app can write. If it
can't, the app warns and logs to the console only.

Django's own pages send a strict Content Security Policy. Set
`DJANGO_CSP_REPORT_ONLY=1` to try it first. `settings.py` shows how to extend it.

### URLs

Register the library's API views on your own router, and include its other views,
as in [app/urls.py](./app/urls.py):

```python
from django.urls import include, path
from rest_framework import routers

from ichec_django_core.urls import register_drf_views

router = routers.DefaultRouter()
register_drf_views(router)

urlpatterns = [
    path("api/", include(router.urls)),
    path("", include("ichec_django_core.urls")),
]
```

Register your app's own viewsets on the same router, so the whole API sits under
one `/api/` root.

Errors come back as RFC 9457 problem details, `application/problem+json`, with
a JSON Pointer to each invalid field. Set the error handlers too, so errors on
`/api/` paths outside DRF's views do the same:

```python
handler404 = "ichec_django_core.views.errors.not_found"
handler500 = "ichec_django_core.views.errors.server_error"
```

`/health/live/` and `/health/` serve liveness and readiness checks. Extend
`HEALTH_CHECKS` to add your own.

If Django serves your Angular app's `index.html`, add the library's route last,
and set `FRONTEND_INDEX` to the file:

```python
from ichec_django_core.views.spa import spa_route

urlpatterns += [spa_route("reports/")]
```

It serves the SPA for every path except the library's and the ones you pass.
It sends none of Django's Content Security Policy, which would block the inline
styles Angular adds, so nginx must set the SPA's policy. Don't write your own
view for this: one that sends Django's policy serves the app unstyled.

## Say who may do what

Each model declares an `Access` policy: the roles someone can have towards a
row, then the roles that may take each action. An action the policy leaves out
is refused to everyone.

```python
from django.db import models

from ichec_django_core.access import HasPermission, Owner, Policy


class Application(models.Model):
    class Access(Policy):
        applicant = Owner("applicant")
        manager = HasPermission("change_application")

        view = applicant | manager
        change = applicant.when(status="draft")
        submit = applicant
        extra_actions = ("submit",)
```

A viewset built on `BaseModelViewSet` lists only the rows the caller may view,
and refuses the actions the policy doesn't allow them. `policy_for(Application).table()`
prints the rules in words, for review and for tests. `register` sets the policy
of a model you don't own, such as the forms app's `PopulatedForm`. To hide a
field from callers who may see the row, such as reviewer notes from the
applicant, list it in the policy's `restricted_fields` with the action that
shows it, and add `RestrictedFieldsMixin` to the serializer. With
`Allowed("application")`, a row follows a related row's policy, such as a form
its application's. [access.py](./src/ichec_django_core/access.py) lists the role types.

A status changes through a django-fsm-2 `@transition` method. The policy says
who may run it, and the transition says from which states. Add
`TransitionsMixin` to the viewset and list the transitions in `transitions`,
and each becomes an endpoint. `run_transition` is the one path for every
trigger, whether it comes from the API, a scheduled task or an event. See
[transitions.py](./src/ichec_django_core/transitions.py).

A `HasPermission` rule reads Django permissions, which members get through
their groups. Declare each group's permissions once, in the `ROLES` setting:

```python
ROLES = {
    "members": ["myapp.view_application"],
    "admins": ["myapp.view_application", "myapp.change_application"],
}
```

Run `python manage.py sync_roles` on every deploy. It gives each group exactly
these permissions. A system check fails on a permission that doesn't exist,
and the test fixtures grant the same permissions.

## Generate your app's client

Your app's API schema is `schema.yaml`, and its typed Python client is
generated from it. Install the `codegen` extra in your development tools, give
your app a `client` package, and run:

```shell
python manage.py generate_client myapp.client
```

It writes `schema.yaml`, and `generated.py` and `registry.py` in the package.
Register the registry after the library's:

```python
from ichec_django_core.client.models import register_resources

from .generated import *  # noqa: F403
from .registry import RESOURCES

register_resources(RESOURCES)
```

Commit what it writes. In CI, `generate_client myapp.client --check` fails if
any of it is out of date.

If a field of yours shares a name with one of the library's, such as
`id_type`, but has other choices, the schema fails with an enum name clash.
Name your enum in your settings:

```python
SPECTACULAR_SETTINGS = {
    **settings.SPECTACULAR_SETTINGS,
    "ENUM_NAME_OVERRIDES": {"FacilityIdTypeEnum": "myapp.models.ID_CHOICES"},
}
```

## Sign in with Keycloak

The example app uses Django's own login, which lets anyone register. Real
deployments sign users in through Keycloak with OpenID Connect (OIDC).

Register your app as a client in the Keycloak realm, then set:

```shell
WITH_OIDC=1
OIDC_RP_CLIENT_ID=my-app
OIDC_RP_CLIENT_SECRET=...
OIDC_OP_AUTHORIZATION_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/auth
OIDC_OP_TOKEN_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/token
OIDC_OP_USER_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/userinfo
OIDC_OP_JWKS_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/certs
OIDC_OP_LOGOUT_ENDPOINT=https://keycloak.example.org/realms/my-realm/protocol/openid-connect/logout
```

Leave `DJANGO_WITH_USER_LOGIN` unset in production. The built-in login locks out
repeated failed sign-ins, and Django must only be reachable through nginx.
To sign in through a Keycloak on your machine, with a `Test` realm on port 8080,
run `uv run tox -e dev-oidc`.
[mozilla-django-oidc](https://mozilla-django-oidc.readthedocs.io/en/stable/installation.html)
does the OIDC work, and its docs explain each setting.

## Optional features

Each of these is off until you turn it on.

- **Background tasks.** Emails and Keycloak syncs run inline by default. To run them
  on a Celery worker, install the `async` extra and set
  `DJANGO_TASK_BACKEND=celery`, `CELERY_BROKER_URL` and `CELERY_RESULT_BACKEND`.
  Then call `create_celery_app()` in your app's `celery.py`, as in
  [app/celery.py](./app/celery.py), and run `celery -A your_app worker`. To
  run your own functions the same way, decorate them with
  `@background_task("your_app.task_name")` from `ichec_django_core.tasks`.
- **Error tracking.** Install the `sentry` extra and set `SENTRY_DSN`. Reports
  carry no personal data.
- **File storage.** Uploaded files are served by Django by default. Set
  `DJANGO_DOWNLOAD_DELIVERY` to `xaccel` to have nginx serve them, or to `s3` to
  keep them in object storage. The comments in
  [settings.py](./src/ichec_django_core/settings.py) explain each mode.
- **Resumable uploads.** Large files can go to a [tusd](https://github.com/tus/tusd)
  sidecar, which resumes them after a dropped connection. Set
  `DJANGO_TUS_UPLOADS=1`, route `/api/files/tus/` to tusd, and run it with
  `-base-path /api/files/tus/ -behind-proxy -hooks-http <app>/api/tus_hook/
  -hooks-enabled-events pre-create,pre-finish`. tusd must write where Django
  reads: `-upload-dir` is `$MEDIA_ROOT/uploads/tus`, or on S3 set
  `-s3-object-prefix uploads/tus/`. The final PATCH returns the upload, as
  `POST /api/files/` does. Browsers must send their `X-CSRFToken` header to tusd.
  The manifest's `tus_uploads` tells clients whether tusd is there.
- **Forms.** Forms defined by JSON Schema, with drafts and file answers. Add
  `"ichec_django_core.forms"` to `INSTALLED_APPS`, and `register_drf_views` and
  the library's URLs include the forms API. The model that owns a form, such as
  an application, submits it. Register a policy for `PopulatedForm` to say who
  sees and edits answers, such as
  `change = Owner("application.applicant").when(application__status="draft")`,
  and one for `Form` to say who downloads its templates. See
  [forms/services.py](./src/ichec_django_core/forms/services.py).

  Forms used to be part of the core app. If one of your models links to
  a form, point the link at `ichec_forms` in a migration that depends on
  `("ichec_forms", "0001_initial")` and sets
  `run_before = [("ichec_django_core", "0036_move_forms_to_ichec_forms")]`.
- **Workflows.** Each call runs its applications through a flow held as
  data, so a portal changes a call's review without a release. Install the
  `workflows` extra, and add `"simple_history"`, `"ichec_django_core.forms"`
  and `"ichec_django_core.workflows"` to `INSTALLED_APPS`.

  Add `WorkflowOwnerMixin` to the model that owns a flow, such as a call, and
  `WorkflowSubjectMixin` to the model that goes through it, such as an
  application. Set `owner_field` on the subject to its foreign key to the
  owner. Serve their transitions, such as `submit`, `close`, `reopen` and
  `move`, with `TransitionsMixin`. `workflows/serializers.py` has the request
  bodies for the ones that take a date or a reason.

  A flow is a list of steps of four types: `collect` a form from each person
  in a role, `decide` an outcome, `wait` for an event or a date, and work
  through a `checklist`. Each step's `role` is a path from the subject to
  members, such as `call.board_members`. Coordinators edit flows at
  `/api/flows/` and `/api/steps/`, and assignees work their tasks at
  `/api/tasks/`. Run `python manage.py run_workflow_schedule` every few
  minutes from cron, to close calls at their closing time.

  A call's copy of a flow has its own step forms, which `/api/forms/` changes.
  Whoever may change a call may change its flow and steps until it opens, by
  the call's own policy.

  Keep each flow in a YAML file, and list it in `WORKFLOW_TEMPLATES`, which
  loads it after each `migrate`, or load it with `python manage.py load_flow
  FILE`. A loaded flow is a template. Each call gets its own copy when it is
  created, so a coordinator can change one call's steps and leave the others
  alone. Loading the file again updates the template, not the copies.
  [workflows/loader.py](./src/ichec_django_core/workflows/loader.py) describes
  the format, and
  [tests/consumer/review_flow.yaml](./tests/consumer/review_flow.yaml) is
  an example.

  Some things need a person's judgement, so the flow leaves them to someone
  the policy allows:

  - A collect step with `declines: wait` holds when someone declines, until
    a coordinator assigns a replacement with `replaces`, or someone runs
    `proceed` on the subject to move on without one.
  - `proceed` finishes any step but a decision early. Late responses still
    count.
  - `reopen` gives one subject more time after its call closes, whether it
    is still a draft or already in review.
  - `move` corrects a subject's place, including after a withdrawal.

  A member given a task gets a notification linking to
  `WORKFLOW_TASK_LINK`, `/tasks/{id}` by default. Set a step's `notify` to
  false to send your own for that step. To run your own code,
  connect to `engine.assigned`, `engine.entered` or `engine.decided`, such as
  to record the facility the chair chose.

  The subject's policy says who reads the work done on it, with a
  `view_tasks` action, such as the chair of the call. Register a policy for
  `PopulatedForm` with `view = Allowed("task")` and
  `change = Allowed("task", "change")`, so each task's form follows its task.

  Reviews follow the application, not the reviewer. Make the subject an
  `AnonymisableMixin` with `"tasks"` in `pii_forward_fields`, and erasing it
  erases the reviews of it. A system check fails until you do. Erasing a
  reviewer keeps their reviews and rankings, as the record of a decision, and
  only their member row stops naming them. Both find the reviews in their
  export. See
  [workflows/models.py](./src/ichec_django_core/workflows/models.py) and
  [step_types.py](./src/ichec_django_core/workflows/step_types.py).
- **Keycloak attributes.** The app can copy values such as a member's role or
  avatar onto their Keycloak user, so other apps in the realm can read them.
  Set `KEYCLOAK_ATTRIBUTE_SYNC_ENABLED=1`, install the `keycloak` extra, and
  list the attributes in `KEYCLOAK_ATTRIBUTE_SYNC`. See
  [sso/\_\_init\_\_.py](./src/ichec_django_core/sso/__init__.py).

## Test your app

`ichec_django_core.test` has helpers for your app's tests:

- `AuthAPITestCase` calls your API as each test user.
- `PolicyEnforcementTestCase` proves your API enforces each model's policy. It
  calls each of your viewsets as each test user, on every row, and checks
  what they see and may do against the policy.
- `AuthorizationSweepTestCase` calls every endpoint in your schema as each
  role. It checks each view with an access policy against the policy, and the
  rest against a file listing who may call them. List only your own views
  without a policy. The library adds its own. For a sweep of a deployed stack,
  `python manage.py sweep_declaration` writes the whole declaration.
- The checks in `test/privacy.py` prove that erasing a member removes all their
  personal data, including from your app's models.

### Test user journeys, in-process and deployed

`ichec_django_core.e2e` runs a user journey, written as a `Scenario`, in
process on every merge request and against a deployed stack. List your test
users in an identity map, such as `e2e/identities.yml`, with their groups.
`InProcessFixtures` creates them in the test database, with the permissions
`ROLES` gives their groups. `HttpFixtures` finds them in the deployment. Give
both your own bundles, which create what the journey needs through the API, so
the journey runs the same way in both. See
[e2e/fixtures.py](./src/ichec_django_core/e2e/fixtures.py).

For the deployed tests, import the fixtures in your e2e `conftest.py`, and
override `e2e_config`:

```python
from pathlib import Path

import pytest

from ichec_django_core.e2e.pytest_plugin import *  # noqa: F403
from ichec_django_core.e2e.pytest_plugin import E2EConfig


@pytest.fixture(scope="session")
def e2e_config() -> E2EConfig:
    return E2EConfig(identity_map=Path(__file__).parent / "identities.yml")
```

`ichec_django_core.e2e.suites` has the tests every deployment runs: sign-in and
the session cookie, how the SPA is served and that no Content Security Policy
blocks it, and the authorization sweep.
Subclass each in a test module. With `users="run"`, each run creates its test
users in Keycloak and deletes them after. To create a permanent user, such as a
scanner's, run `python -m ichec_django_core.e2e.realm apply --map FILE`.
[e2e/pytest_plugin.py](./src/ichec_django_core/e2e/pytest_plugin.py) lists
the variables a deployment sets.

## Work on this library

Install the development tools and run the checks that CI runs:

```shell
uv sync --group dev --extra async
uv run pytest
uv run ruff format --check src tests
uv run ruff check src tests
uv run mypy src
```

If you use [direnv](https://direnv.net/), run `direnv allow` once and the
environment activates when you enter the directory.

If you change a model or serializer, regenerate the API schema and client, and
commit them:

```shell
uv run python manage.py generate_client ichec_django_core.client
```

## Licence

Copyright of the Irish Centre for High End Computing (ICHEC), released under the
MIT License. See [LICENSE](./LICENSE).
