Metadata-Version: 2.4
Name: leaf-portal
Version: 1.1.93
Summary: LEAF Portal
License-File: LICENSE
Requires-Python: >=3.12,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: asyncpg (>=0.31.0)
Requires-Dist: bcrypt (>=5.0.0)
Requires-Dist: httpx (>=0.28.1,<0.29.0)
Requires-Dist: ipykernel (>=7.2.0)
Requires-Dist: nicegui[testing] (>=3.12.0)
Requires-Dist: pandas (>=3.0.3)
Requires-Dist: plotly (>=6.7.0)
Requires-Dist: pyotp (>=2.9.0)
Requires-Dist: python-dotenv (>=1.2.2)
Requires-Dist: qrcode (>=7.4.2)
Description-Content-Type: text/markdown

# leaf-portal

Web portal for the LEAF framework. Built with [NiceGUI](https://nicegui.io) and [asyncpg](https://github.com/MagicStack/asyncpg), backed by a TimescaleDB/PostgreSQL database.

## Features

- **Organisation & department management** — hierarchical grouping of entities
- **User management** — superadmin, org admin, and regular users with bcrypt-hashed passwords; admin impersonation
- **Access management** — time-windowed grants per department/entity
- **Entity management** — hide entities from regular users and the sensor catalog
- **Raw data retention** — per-entity override of how long raw `sensor_data` is kept before a scheduled purge (aggregated history is always kept)
- **MQTT accounts** — department-scoped MQTT credentials for VerneMQ's Postgres-auth plugin, when configured (optional; hidden if the deployment doesn't have `vmq_auth_acl`)
- **Sensor data explorer** — browse and filter readings by entity, metric, and time range (multi-select), with the same aggregation levels as Plots
- **Interactive plots** — Plotly-based time-series visualization
- **Alarm rules** — threshold-based alerts (configurable per-rule check interval) with email notifications on trigger and auto-resolve
- **API token management** — generate and revoke tokens for REST API access
- **REST API** — token-authenticated endpoints for sensor data retrieval (Swagger UI at `/api/docs`)
- **First-run setup wizard** — browser-based DB connection and superadmin creation at `/setup`
- **Password reset** — email-based reset flow (`/forgot-password`, `/reset-password`)

## Requirements

- Python 3.12+
- PostgreSQL 16+ or TimescaleDB
- SMTP server (optional — required for alarm emails and password reset)

## Installation

From PyPI:

```bash
pip install leaf-portal
```

From source:

```bash
poetry install
```

## Configuration

Create a `.env` file in the working directory. All variables are optional at startup — the setup wizard at `/setup` will prompt for DB credentials on first run and persist them to `.env`.

```env
# Database (defaults shown)
PGHOST=timescaledb
PGPORT=5432
PGDATABASE=leaf
PGUSER=postgres
PGPASSWORD=

# Mail (required for alarm emails and password reset)
SMTP_HOST=smtp.example.com
SMTP_PORT=587
SMTP_USER=user@example.com
SMTP_PASSWORD=secret
SMTP_FROM=noreply@example.com

# Portal public URL (used in password-reset emails)
PORTAL_URL=http://localhost:8081

# NiceGUI session secret — change in production
STORAGE_SECRET=change-me-in-production

# Required to complete first-run setup at /setup — without it, the setup
# wizard refuses to create the initial superadmin account. Generate a
# random value and pass it to `deploy/deploy.py run` too, so it reaches
# the container.
SETUP_TOKEN=change-me-in-production

# Demo mode -- disables self-service password/2FA changes (profile page,
# forgot/reset password, and an admin resetting their own password) so a
# public demo deployment can't have its own credentials changed out from
# under it. Off by default.
DEMO_MODE=false

# Grafana account mirroring (optional) — whenever the portal has a user's
# plaintext password in hand (login, password reset), it pushes a matching
# account to Grafana via its admin API, so users can log into Grafana
# directly with the same credentials. Deleting a portal user deletes the
# mirrored Grafana account too.
#
# The same sync also mirrors LEAF's organisation/department structure: one
# Grafana Org per organisation, one Team per department (named
# "ORG :: CODE", using the department's short code -- e.g. "WUR :: SSB" --
# not its full display name), with membership kept in sync on every
# login/password change. Teams are looked up by a stable
# {department_id}@leaf-dept.local marker email, not by name, so renaming a
# department or its code in LEAF updates the existing team in place instead
# of orphaning its membership under a new, empty duplicate. Org membership
# is additive-only; team membership is a full sync (joined AND left, so
# leaving a department in the portal removes the matching Grafana team too).
# Datasources are not provisioned automatically -- add one per Grafana Org
# by hand.
#
# A superadmin can also generate auto-built overview dashboards -- one per
# department, plus one per entity category -- from /admin/settings
# ("Grafana Overview Dashboards" card, only shown when GRAFANA_URL is set).
# It's an on-demand admin action, not run automatically or on login: it
# scans every department system-wide, which has nothing to do with any one
# user's login event. See leaf_portal/dashboard_gen.py -- team lookup there
# reuses this same sync's convention, so it resolves to the same team
# rather than creating a lookalike duplicate.
#
# Leave GRAFANA_URL unset to disable all of the above.
#
# Grafana's admin user-management API only accepts HTTP Basic auth from a
# Grafana server-admin user (service account tokens are rejected), so this
# must be the actual Grafana admin credentials -- the same ones set via
# GF_SECURITY_ADMIN_USER / GF_SECURITY_ADMIN_PASSWORD on the Grafana side.
GRAFANA_URL=http://grafana:3000
GRAFANA_ADMIN_USER=admin
GRAFANA_ADMIN_PASSWORD=
```

## Running

```bash
leaf-portal
# or
python -m leaf_portal
```

The portal listens on `0.0.0.0:8081` by default.

On first run, navigate to `http://localhost:8081` — you will be redirected to the setup wizard to configure the database connection and create the initial superadmin account.

## REST API

All endpoints require a token passed via the `Authorization: Bearer <token>` header.

Tokens are generated from the **API Tokens** page (`/tokens`).

The API is versioned under `/api/v1`. Unversioned `/api/...` paths still work as aliases for backward compatibility but are deprecated — use `/api/v1/...` for new integrations.

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/managements` | List managements accessible to the token |
| `GET` | `/api/v1/data/recent` | Most recent readings across all accessible departments (`?limit=20`) |
| `GET` | `/api/v1/data` | Filtered sensor data — requires `organisation` + `department`; optional: `entity`, `metric` (comma-separated for multiple), `from`, `to` (ISO 8601), `limit` (max 10 000) |

Interactive docs: `/api/docs`

## Development

```bash
poetry install --with dev

# Run tests
poetry run pytest tests/ -v

# Lint and format
poetry run ruff check leaf_portal/ tests/
poetry run ruff format leaf_portal/ tests/
```

## Database

The schema lives at `deploy/deploy.sql` (targets TimescaleDB) and is applied automatically by the setup wizard or on startup when the DB is reachable.

Service-account credentials live separately at `deploy/pre.sql` — it creates the 6 login roles (`leaf_portal_user`, `leaf_grafana_user`, ...) with passwords supplied via `psql -v`, and must be run once, before `deploy/deploy.sql`, against a bare database. `deploy/deploy.sql` itself contains no passwords: it only grants those already-created logins access to the schema it defines, so it stays safe to reapply (e.g. via the admin "Reapply Schema" button) without ever touching credentials.

Both files are bundled into the Docker image (`/app/deploy/deploy.sql`, `/app/deploy/pre.sql`), so downstream deployments can extract them directly from whatever image tag they run instead of keeping a separately-maintained copy that can drift out of sync — see the `oak` stack's `schema-extract` service for an example.

For CI and Docker-based local development, `docker/init_db.py` waits for Postgres to be ready and then applies `deploy/pre.sql` (with placeholder passwords — CI never actually connects as these accounts) followed by `deploy/deploy.sql`, since `deploy.sql`'s grants require the 6 logins to already exist.

## Backups

A full `pg_dump` of the database doesn't make sense once `sensor_data` grows
large — it would re-export the entire sensor history every day. Instead,
`docker/backup` builds a small standalone image that splits the backup in two:

1. **Operational tables** (`organisation`, `department`, `user_account`,
   `management`, `alarm_*`, `mapper_*`, `api_token`, ...) — these are tiny, so
   they're fully `pg_dump`'d (custom format) every day to `leaf_YYYY-MM-DD.dump`.
   `sensor_data` and TimescaleDB's internal chunk/catalog tables are excluded.
2. **`sensor_data`** — exported per UTC day via `\copy` to
   `sensor_data_YYYY-MM-DD.csv.gz`. The last `ROLLING_DAYS` days are
   re-exported (overwritten) on every run to catch late-arriving readings;
   older files are write-once.

The image is read-only against the database (`leaf_backup_user`, member of
the `backup_readers` role created by `deploy/deploy.sql`) and runs once per
invocation — schedule it with a Kubernetes CronJob (see
`docker/backup/cronjob.example.yaml`) or any host cron running `docker run`.

```bash
docker build -t leaf-backup docker/backup

docker run --rm \
  -e PGHOST=... -e PGUSER=leaf_backup_user -e PGPASSWORD=... -e PGDATABASE=leaf \
  -e ROLLING_DAYS=3 -e KEEP_DUMPS=14 \
  -v /path/to/backups:/backups \
  leaf-backup
```

Or, using `deploy/deploy.py` (builds/pushes via `build-backup`, runs once via
`backup`):

```bash
python3 deploy/deploy.py build-backup   # build & push to the registry

export PGHOST=... PGUSER=leaf_backup_user PGPASSWORD=... PGDATABASE=leaf
export BACKUP_DIR=/path/to/backups      # default: ./backups
python3 deploy/deploy.py backup
```

Restore:

```bash
# 1. Recreate the schema (also recreates the sensor_data hypertable)
python3 deploy/deploy.py schema

# 2. Restore operational tables
pg_restore --data-only --disable-triggers -d <db> leaf_YYYY-MM-DD.dump

# 3. Re-import sensor_data for each day
zcat sensor_data_YYYY-MM-DD.csv.gz | psql -d <db> -c "\copy sensor_data FROM STDIN WITH (FORMAT csv, HEADER true)"
```

The backup image pins its `pg_dump`/`pg_restore` version to the TimescaleDB
version this project targets (`timescale/timescaledb:2.17.2-pg16`) — custom-format
dumps aren't readable by an older `pg_restore`. To back up a different
Postgres major version, change the `FROM` tag in `docker/backup/Dockerfile`
and rebuild. Before dumping anything, `backup.sh` checks that the server's
major version matches its bundled `pg_dump` and exits with an error
(without writing any files) if they've drifted apart.

## Deployment

`deploy/deploy.py` is a helper script for building and running the portal in production. It requires no extra dependencies beyond Docker (and `psql` for remote schema application).

```
python3 deploy/deploy.py <command>
```

| Command | What it does |
|---------|--------------|
| `build` | Builds a multi-arch (`amd64`/`arm64`) Docker image, tags it with the current git tag or short commit hash, and pushes it to `docker-registry.wur.nl/m-unlock/docker/leaf-portal`. |
| `schema` | Applies `deploy/pre.sql` then `deploy/deploy.sql` to the target database. Locally it runs `psql` inside the `timescaledb` container; against a remote host it calls `psql` directly. |
| `run` | Pulls the portal image from the registry and starts it as a container named `leaf-portal` on port 8081. |
| `stop` | Stops and removes the `leaf-portal` container. |

`schema` requires passwords for the PostgreSQL service accounts `pre.sql` creates — never use defaults:

```bash
export LEAF_PORTAL_PASSWORD=...   # leaf_portal_user  (portal, read+write)
export LEAF_GRAFANA_PASSWORD=...  # leaf_grafana_user (Grafana, read-only)
export LEAF_API_PASSWORD=...      # leaf_api_user     (external API access)
export LEAF_BACKUP_PASSWORD=...   # leaf_backup_user  (backup job, read-only)
export LEAF_VERNEMQ_PASSWORD=...  # leaf_vernemq_user (VerneMQ Postgres-auth, SELECT-only on vmq_auth_acl)
export LEAF_NODERED_AUTH_PASSWORD=... # leaf_nodered_auth_user (Node-RED admin login, column-limited SELECT on user_account)
```

Typical production flow:

```bash
# 1. Apply the schema (once, or after schema changes)
python3 deploy/deploy.py schema

# 2. Start the portal
python3 deploy/deploy.py run
```

Run `build` only when cutting a new release.

## Page routes

| Route | Description |
|-------|-------------|
| `/` | Redirects to `/dashboard` or `/login` |
| `/login` | Login page |
| `/setup` | First-run setup wizard |
| `/forgot-password` | Password reset request |
| `/reset-password` | Password reset with token |
| `/dashboard` | Overview dashboard |
| `/admin/organisations` | Organisation management |
| `/admin/departments` | Department management |
| `/admin/users` | User management |
| `/admin/access-management` | Access grant management |
| `/admin/mapper` | Entity/metric mapping |
| `/admin/retention` | Raw data retention management |
| `/admin/mqtt-accounts` | MQTT account management (optional — requires VerneMQ Postgres auth) |
| `/admin/settings` | Application settings |
| `/dept/members` | Department member management |
| `/entities` | Entity management (hide/show from regular users) |
| `/categories` | Category management |
| `/data/explorer` | Sensor data explorer |
| `/data/plots` | Time-series plots |
| `/alarms` | Alarm rules and event history |
| `/tokens` | API token management |
| `/profile` | User profile |
| `/api/docs` | Swagger UI for the REST API |

