Metadata-Version: 2.4
Name: frequenz-cs-reporting
Version: 0.5.0
Summary: Streamlit Library for reporting and solar monitoring
Author-email: Frequenz Energy-as-a-Service GmbH <floss@frequenz.com>
License-Expression: MIT
Project-URL: Documentation, https://frequenz-floss.github.io/frequenz-cs-reporting/
Project-URL: Changelog, https://github.com/frequenz-floss/frequenz-cs-reporting/releases
Project-URL: Issues, https://github.com/frequenz-floss/frequenz-cs-reporting/issues
Project-URL: Repository, https://github.com/frequenz-floss/frequenz-cs-reporting
Project-URL: Support, https://github.com/frequenz-floss/frequenz-cs-reporting/discussions/categories/support
Keywords: frequenz,python,lib,library,frequenz-cs-reporting,tooling,notebooks,streamlit
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Software Development :: Libraries
Classifier: Typing :: Typed
Requires-Python: <4,>=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: css-inline>=0.17.0
Requires-Dist: frequenz-gridpool<0.9.0,>=0.8.0
Requires-Dist: frequenz-lib-notebooks<0.18.0,>=0.17
Requires-Dist: frequenz-client-marketmetering<0.2.0,>=0.1.2
Requires-Dist: matplotlib>=3.10.6
Requires-Dist: numpy>=2.3.2
Requires-Dist: pandas>=2.3.2
Requires-Dist: plotly>=6.3.0
Requires-Dist: pyarrow>=21.0.0
Requires-Dist: python-box>=7.3.2
Requires-Dist: python-dateutil>=2.9.0.post0
Requires-Dist: python-dotenv>=1.1.1
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: streamlit>=1.49.1
Requires-Dist: streamlit-aggrid>=1.1.8.post1
Requires-Dist: watchdog>=6.0.0
Provides-Extra: dev-flake8
Requires-Dist: flake8==7.5.0; extra == "dev-flake8"
Requires-Dist: flake8-datetimez==26.8.1; extra == "dev-flake8"
Requires-Dist: flake8-docstrings==1.7.0; extra == "dev-flake8"
Requires-Dist: flake8-pyproject==1.2.4; extra == "dev-flake8"
Requires-Dist: pydoclint==0.12.0; extra == "dev-flake8"
Requires-Dist: pydocstyle==6.3.0; extra == "dev-flake8"
Provides-Extra: dev-formatting
Requires-Dist: black==26.10.1; extra == "dev-formatting"
Requires-Dist: isort==8.0.1; extra == "dev-formatting"
Provides-Extra: dev-mkdocs
Requires-Dist: Markdown==3.11.2; extra == "dev-mkdocs"
Requires-Dist: black==26.10.1; extra == "dev-mkdocs"
Requires-Dist: mike==2.2.0; extra == "dev-mkdocs"
Requires-Dist: mkdocs-gen-files==0.6.1; extra == "dev-mkdocs"
Requires-Dist: mkdocs-literate-nav==0.6.3; extra == "dev-mkdocs"
Requires-Dist: mkdocs-macros-plugin==1.5.0; extra == "dev-mkdocs"
Requires-Dist: mkdocs-material==9.8.0; extra == "dev-mkdocs"
Requires-Dist: mkdocstrings[python]==1.0.8; extra == "dev-mkdocs"
Requires-Dist: mkdocstrings-python==2.0.10; extra == "dev-mkdocs"
Requires-Dist: pymdown-extensions==12.3; extra == "dev-mkdocs"
Requires-Dist: frequenz-repo-config[lib]==0.19.0; extra == "dev-mkdocs"
Provides-Extra: dev-mypy
Requires-Dist: mypy==2.4.1; extra == "dev-mypy"
Requires-Dist: types-Markdown==3.11.2.20261029; extra == "dev-mypy"
Requires-Dist: types-pytz; extra == "dev-mypy"
Requires-Dist: pandas-stubs; extra == "dev-mypy"
Requires-Dist: frequenz-cs-reporting[dev-mkdocs,dev-noxfile,dev-pytest]; extra == "dev-mypy"
Provides-Extra: dev-noxfile
Requires-Dist: nox==2026.10.20; extra == "dev-noxfile"
Requires-Dist: frequenz-repo-config[lib]==0.19.0; extra == "dev-noxfile"
Provides-Extra: dev-pylint
Requires-Dist: frequenz-cs-reporting[dev-mkdocs,dev-noxfile,dev-pytest]; extra == "dev-pylint"
Provides-Extra: dev-pytest
Requires-Dist: pytest==9.2.0; extra == "dev-pytest"
Requires-Dist: pylint==4.1.3; extra == "dev-pytest"
Requires-Dist: frequenz-repo-config[extra-lint-examples]==0.19.0; extra == "dev-pytest"
Requires-Dist: pytest-mock==3.17.0; extra == "dev-pytest"
Requires-Dist: pytest-asyncio==1.4.1; extra == "dev-pytest"
Requires-Dist: async-solipsism==0.10; extra == "dev-pytest"
Provides-Extra: dev
Requires-Dist: frequenz-cs-reporting[dev-flake8,dev-formatting,dev-mkdocs,dev-mypy,dev-noxfile,dev-pylint,dev-pytest]; extra == "dev"
Dynamic: license-file

# Frequenz CS Reporting Library

[![Build Status](https://github.com/frequenz-floss/frequenz-cs-reporting/actions/workflows/ci.yaml/badge.svg)](https://github.com/frequenz-floss/frequenz-cs-reporting/actions/workflows/ci.yaml)
[![PyPI Package](https://img.shields.io/pypi/v/frequenz-cs-reporting)](https://pypi.org/project/frequenz-cs-reporting/)
[![Docs](https://img.shields.io/badge/docs-latest-informational)](https://frequenz-floss.github.io/frequenz-cs-reporting/)

## Overview

Streamlit library that ships a ready-to-use client reporting UI. It fetches data from the Frequenz reporting API, applies the energy
reporting utilities from `frequenz-lib-notebooks`, and renders dashboards, tables, and plots with reusable Streamlit components.

## Features

- Pre-built Streamlit app with navigation, landing page, and reporting view.
- Connects to the Frequenz reporting API to fetch microgrid measurements.
- Ready-made dashboards (metrics, plots, and tables) powered by
  `frequenz-lib-notebooks`.
- Reusable components (sidebar filters, charts, tables) for your own pages.

## Quick start

1. Install the library (Python 3.12):
   ```bash
   pip install "frequenz-cs-reporting"
   ```
2. Provide environment variables (see below). A `.env` file works with Streamlit:
   ```bash
   REPORTING_API_URL=https://your-reporting-endpoint
   FREQUENZ_API_KEY=your-api-key
   FREQUENZ_API_SECRET=your-signing-secret
   MICROGRID_CONFIG_DIR=toml_directory/
   ```
3. Add .toml files to the toml_directory.
4. Run the bundled UI from the repo root:
   ```bash
   streamlit run app.py
   ```
   Use the sidebar to pick a microgrid, date range, timezone, and resolution.

## Configuration

### Environment

- `REPORTING_API_URL` **(required)**: Base URL for the Frequenz reporting API.
- `FREQUENZ_API_KEY` and `FREQUENZ_API_SECRET` **(required)**: Credentials used
  to authenticate and sign Reporting and Assets API requests.
- `MICROGRID_CONFIG_DIR` *(optional)*: Directory containing TOML microgrid
  configs. Defaults to `toml_directory/`.
- `PARQUET_STORAGE_PATH` *(optional)*: Persistent directory for reporting data.
  Defaults to `data/parquet`.
- `SYNC_INTERVAL_MINUTES` *(optional)*: Minimum interval between overlap
  refreshes. Defaults to `15`.
- `SYNC_OVERLAP_MINUTES` *(optional)*: Tail window re-fetched to pick up recent
  corrections. Defaults to `5`.
- `API_REQUEST_TIMEOUT_SECONDS` *(optional)*: Reporting request timeout.
  Defaults to `300`.

### Microgrid configs

Microgrid definitions are loaded from TOML files in `MICROGRID_CONFIG_DIR`. The UI only lists configured microgrids returned by the Assets API for the current `FREQUENZ_API_KEY` and `FREQUENZ_API_SECRET`; inaccessible TOML entries are not displayed.

## Running the Streamlit app

The app entry point is `app.py`. When you run `streamlit run app.py`, it:

- Discovers pages from `frequenz.cs_reporting.app_pages` (the default
  build ships `Home` and `Reporting` pages).
- Loads microgrid configs from `MICROGRID_CONFIG_DIR` and lists available IDs.
- Fetches data via the reporting API.

### Persistent data cache

Active-power, battery-SOC, and ENTSO-E day-ahead price time series are stored as
Parquet. Microgrid measurements are separated by microgrid and resolution. The
first request downloads the requested range;
later requests read covered ranges locally and fetch only missing intervals.
When a dataset is due for refresh, its trailing overlap is fetched again and
merged by timestamp. Writes use a same-directory temporary file followed by an
atomic replacement, while a file lock prevents duplicate writers across
Streamlit sessions and local worker processes. Sync metadata is stored beside
the Parquet directory in `data/metadata` by default. Runtime duration entries
are appended to `data/metadata/duration_log.txt`; when
`PARQUET_STORAGE_PATH` is customized, the log uses the sibling `metadata`
directory of that configured path.

In Deepnote, the app automatically restores `parquet` and `metadata` from the
persistent `/work/data` directory into the faster
`/tmp/frequenz-cs-reporting` directory once per machine lifetime. Scheduled
synchronization copies the updated local snapshot back to `/work/data` after a
successful run. The Streamlit app also queues this copy in a background worker
after rendering and after background dataset synchronization, so Parquet data,
sync metadata, and duration logs survive machine restarts without delaying the
page. Set `DEEPNOTE_HYBRID_CACHE_ENABLED=false` to disable this, or customize
the roots with `DEEPNOTE_PERSISTENT_CACHE_ROOT` and
`DEEPNOTE_LOCAL_CACHE_ROOT`.

Interactive reporting reads active-power data from the local cache and starts
one shared background refresh when cached data is available. An empty cache is
populated synchronously once so the initial load does not depend on browser
timers in Deepnote; cached dashboards remain usable during later refreshes.
Duration records are queued and written in batches by a background writer so
diagnostic file I/O does not block Streamlit.

The reporting client used by this app exposes time-range queries, but not
record modification timestamps, stable record IDs, change tokens, ETags, or a
deletion feed. Consequently this is range-incremental synchronization, not a
complete change-data-capture implementation. Corrections older than the overlap
window require a wider explicit reconciliation request or rebuilding that
dataset. Pagination remains owned by the upstream Frequenz clients. The cache
retries transient connection, timeout, HTTP 429, and server errors with
exponential backoff and honors `Retry-After` when the client exposes it.

For production, run synchronization independently from Streamlit, for example
every 15 minutes with cron or the deployment platform's scheduler:

```bash
python -m frequenz.cs_reporting.services.sync_data \
  --microgrid-id 241 --lookback-days 30 --resolution-minutes 15
```

Synchronize the 15-minute day-ahead price dataset once per day after the new
prices have been published (for example, after 14:00 local time):

```bash
python -m frequenz.cs_reporting.services.sync_data \
  --dataset day-ahead-prices --country-code DE_LU --lookback-days 30
```

When `--end` is omitted, the price job requests through the following day so
the newly published prices remain available until the next scheduled run.

Streamlit reads day-ahead prices from this local cache without blocking the
dashboard. If the requested range is missing, one shared in-process background
job populates it while a small polling fragment waits. The dashboard initially
renders without price-dependent content and reruns once when prices arrive.
The daily scheduled job remains recommended so users normally receive an
immediate cache hit.

Microgrid measurements remain a safe pull-through fallback: cache hits do not
call the API, and failed overlap refreshes continue serving the last successful
file. Point `PARQUET_STORAGE_PATH` at persistent storage. All replicas must see
the same filesystem with working POSIX locks; ephemeral container disks and
independent per-replica volumes do not provide a coherent shared cache. For
those deployments, use a shared POSIX volume or replace this backend with
object-storage/database coordination.

### Running in Deepnote
- Running in Deepnote is supported; required environment variables can be injected
via the Deepnote integration.
- Add this library as a requirement in requirements.txt
- Add the docker image from dockerhub (currently named: CS-Reporting in deepnote).
- Copy the app.py to the folder structure in Deepnote.
- Click on create_streamlit_application in Deepnote UI to create the app.

## Library usage

Fetch microgrid data programmatically (sync wrapper shown):

```python
from datetime import datetime, timedelta
from frequenz.cs_reporting.services.data_service import get_microgrid_data

df = get_microgrid_data(
    microgrid_id=241,
    start_date=datetime(2024, 1, 1),
    end_date=datetime(2024, 1, 2),
    resolution=timedelta(minutes=15),
)
```

Build your own Streamlit page and add it to the navigation by defining a
`PageSpec` in `frequenz.cs_reporting.app_pages`:

```python
# app_pages/custom.py
from frequenz.cs_reporting.rep_cs_core.page_spec import PageSpec
import streamlit as st

def render() -> None:
    st.title("Custom view")
    st.write("Add your own charts or tables here.")

PAGE = PageSpec(key="custom", title="Custom", icon="🛠️", order=10, render=render)
```

## Development

- Install dev tools: `pip install -e ".[dev]"`.
- Run tests: `nox -l` to see sessions, e.g. `nox -s tests`.
- Build docs with MkDocs (`README.md` is the landing page). After installing the
  mkdocs extra you can use the `docs` nox session (if available) or run
  `mkdocs serve`.

## Supported Platforms

The following platforms are officially supported (tested):

- **Python:** 3.12
- **Operating System:** Ubuntu Linux 20.04
- **Architectures:** amd64, arm64

## Contributing

If you want to know how to build this project and contribute to it, please
check out the [Contributing Guide](CONTRIBUTING.md).
