Metadata-Version: 2.4
Name: detquantlib
Version: 5.13.1
Summary: An internal library containing functions and classes that can be used across Quant models.
License-File: LICENSE.txt
Author: Dynamic Energy Trading
Requires-Python: >=3.10,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Requires-Dist: colorlog (>=6.10.1,<7.0.0)
Requires-Dist: cryptography (>=50.0.2,<51.0.0)
Requires-Dist: numpy (>=2.2.6,<3.0.0)
Requires-Dist: openpyxl (>=3.1.5)
Requires-Dist: pandas (>=2.3.3,<3.0.0)
Requires-Dist: paramiko (>=4.0.0,<5.0.0)
Requires-Dist: plotly (>=6.3.1,<7.0.0)
Requires-Dist: pyarrow (>=21.0.0)
Requires-Dist: pyodbc (>=5.2.0,<6.0.0)
Requires-Dist: python-dateutil (>=2.9.0,<3.0.0)
Requires-Dist: pytz (>=2020.1)
Requires-Dist: redis (>=7.4.0,<8.0.0)
Requires-Dist: requests (>=2.34.2,<3.0.0)
Requires-Dist: scipy (>=1.15.2,<2.0.0)
Requires-Dist: snowflake-connector-python[pandas] (>=4.8.0,<5.0.0)
Requires-Dist: sqlalchemy (>=2.0.43,<3.0.0)
Requires-Dist: tomli (>=2.0,<3.0) ; python_version < "3.11"
Description-Content-Type: text/markdown

# DET Quant Library

## Table of Contents

<!--TOC-->

- [DET Quant Library](#det-quant-library)
  - [Table of Contents](#table-of-contents)
  - [Overview](#overview)
  - [Package Contents](#package-contents)
    - [Assets](#assets)
    - [Converters](#converters)
    - [Data](#data)
    - [Dates](#dates)
    - [Figures](#figures)
    - [Forecasting](#forecasting)
    - [Outputs](#outputs)
    - [Snapshot Testing](#snapshot-testing)
    - [Stats](#stats)
    - [Tradable Products](#tradable-products)
    - [Utils](#utils)
  - [Exposed Symbols](#exposed-symbols)
    - [List of Exposed Symbols](#list-of-exposed-symbols)
  - [Configuration](#configuration)
    - [Project Structure](#project-structure)
    - [Dependencies](#dependencies)
      - [Dependency Manager](#dependency-manager)
      - [Dependabot](#dependabot)
    - [Invoke Tasks](#invoke-tasks)
    - [GitHub Actions](#github-actions)
      - [Workflow: Python CI](#workflow-python-ci)
        - [CI Check: Code Formatting](#ci-check-code-formatting)
        - [CI Check: Unit Tests](#ci-check-unit-tests)
      - [Workflow: Package Publisher](#workflow-package-publisher)
        - [Checking Version Updates](#checking-version-updates)
        - [Creating Version Tags](#creating-version-tags)
        - [Publishing Package Updates to PyPI](#publishing-package-updates-to-pypi)
    - [Release Notes](#release-notes)

<!--TOC-->

## Overview

The DET Quant Library is an internal library containing functions and classes that can be used
across Quant models.

## Package Contents

The package is split into subpackages, one per functional area. The sections below briefly
describe each of them. See [List of Exposed Symbols](#list-of-exposed-symbols) for the symbols
that can be imported directly from each subpackage.

### Assets

The `detquantlib.assets` package models physical energy assets:

- `BatteryState` stores the characteristics and current state of a battery (capacity, power, state
  of charge, depth of discharge, roundtrip efficiency). Live battery states can be loaded from
  Redis with `BatteryState.build_from_redis()`, for the Eniris and Smartbox energy management
  systems.
- `WindTurbine` stores wind turbine information (rated power, cut-in and cut-out wind speeds,
  number of turbines, power curve) and converts wind speeds to power or energy.
  `WindTurbineModels` contains the specifications of known turbine models, which can be loaded
  with `WindTurbine.build_from_model()`.

### Converters

The `detquantlib.converters` package converts values between units:

- `convert_power()` and `convert_energy()` convert between power units and between energy units.
- `power_to_energy()` and `energy_to_power()` convert between power and energy, for a given
  tenor.
- `convert_price()` converts a price from one currency and volume unit to another, and
  `flag_cents()` detects whether a currency string is expressed in cents.

### Data

The `detquantlib.data` package provides clients for the data sources used by Quant models:

- `SnowflakeClient` and `AzureClient` are small, safe clients for Snowflake and Azure SQL
  databases, configured with `SnowflakeConfig` and `AzureConfig` (which can be read from
  environment variables). They can query, load, list and write tables, and exchange data as
  pandas DataFrames.
- `DetDatabase` is a client for the DET database, with dedicated methods to load and process
  data such as ENTSOE day-ahead and imbalance prices, futures settlement prices, commodities,
  instruments, account positions, customer volume forecasts and clients. It is deprecated and
  will be removed in a future version: use `AzureClient` instead.
- `list_to_sql_set()` and `list_to_sql_columns()` format Python lists for use in SQL queries.
- `Sftp` imports and exports files from and to an SFTP server.
- `Entsoe` imports and processes ENTSOE data, such as day-ahead spot prices from the ENTSOE SFTP
  server.
- `TennetClient` is a wrapper around the [TenneT publications API](https://developer.tennet.eu),
  to fetch settlement prices and balance delta data.

### Dates

The `detquantlib.dates` package contains date and time helpers: rounding timestamps up or down
to a given frequency (`ts_ceil()`, `ts_floor()`), counting the number of hours between two dates
(`count_nr_hours()`), computing the number of months between two dates (`calc_months_diff()`),
and converting dates to month codes (`datetime_to_month_code()`).

### Figures

The `detquantlib.figures.plotly_figures` module contains helpers for Plotly figures: applying a
standard layout, saving figures to JSON or HTML files, and displaying figures saved as JSON.

### Forecasting

The `detquantlib.forecasting` package contains forecasting strategies. `forecast_knife_strategy()`
generates a random walk forecast, where each value is set to the most recent observed value with
the same time of day and the same day type (weekday, Saturday or Sunday).

### Outputs

The `detquantlib.outputs` package manages and exports model outputs. `OutputItem` holds a single
output and exports it to a CSV, JSON, HTML or NPZ file. `OutputSet` groups several `OutputItem`
objects, and `PathDefinitions` defines the folders the outputs are exported to (by default an `Outputs`
folder in the working directory, or the `OUTPUT_DIR` environment variable if set).

### Snapshot Testing

The `detquantlib.snapshot_testing` package provides snapshot (regression) tests for Quant models:
each test runs a model end to end on a fixed set of inputs, and compares its outputs with a
reviewed reference copy. Database calls are recorded once, locally, and replayed from disk in CI,
which has no database access. The `SnowflakeClient` and `AzureClient` classes are supported out of
the box.

See [the snapshot testing guide](detquantlib/snapshot_testing/README.md) for how the framework
works and how to set it up in a model repository.

### Stats

The `detquantlib.stats.data_analysis` module contains statistical analysis tools.
`filter_outliers()` iteratively removes outliers from a time series, based on a rolling mean and
standard deviation, and `create_outliers_iteration_plot()` plots the result of each iteration.

### Tradable Products

The `detquantlib.tradable_products` package contains helpers for tradable products and their
tenors: converting tenors to pandas offset aliases or numbers of hours, counting delivery periods,
and converting between delivery start dates and maturities.

### Utils

The `detquantlib.utils` package contains general utilities: setting up a logger with colored
console output and a log file (`setup_logger()`), adding timestamped log messages (`add_log()`),
converting lists to strings (`list_to_str()`), and dividing arrays element-wise while replacing NaN
and infinite results with user-defined values (`divide_with_nan()`).

## Exposed Symbols

Some of the package's symbols (i.e. functions, classes, modules, etc.) are exposed via
`__init__.py` files. They can therefore be imported with a more concise notation that does not
require specifying their full path.

This section references all exposed symbols, and for each one of them, provides an exhaustive
list of the ways they can be imported.

Note: The list below is auto-generated and can be updated with the following command in the
terminal:

```cmd
poetry run python docs.py
```

To only check whether the list is up to date, without updating it, add the `--check` option. The
CI check runs it with that option (see [CI Check: Code Formatting](#ci-check-code-formatting)),
and `inv lint` runs it as well.

### List of Exposed Symbols

<!-- START EXPOSED SYMBOLS AUTO-GENERATED -->

Classes:

- `ApiCacheMissError`:
  - `from detquantlib.snapshot_testing import ApiCacheMissError`
  - `from detquantlib.snapshot_testing.errors import ApiCacheMissError`
- `AzureClient`:
  - `from detquantlib.data import AzureClient`
  - `from detquantlib.data.databases.azure_client import AzureClient`
- `AzureConfig`:
  - `from detquantlib.data import AzureConfig`
  - `from detquantlib.data.databases.azure_client import AzureConfig`
- `BatteryState`:
  - `from detquantlib.assets import BatteryState`
  - `from detquantlib.assets.battery import BatteryState`
- `DbCacheMissError`:
  - `from detquantlib.snapshot_testing import DbCacheMissError`
  - `from detquantlib.snapshot_testing.errors import DbCacheMissError`
- `DetDatabase`:
  - `from detquantlib.data import DetDatabase`
  - `from detquantlib.data.databases.detdatabase import DetDatabase`
- `Entsoe`:
  - `from detquantlib.data import Entsoe`
  - `from detquantlib.data.entsoe.entsoe import Entsoe`
- `NetworkAccessBlockedError`:
  - `from detquantlib.snapshot_testing import NetworkAccessBlockedError`
  - `from detquantlib.snapshot_testing.errors import NetworkAccessBlockedError`
- `OutputItem`:
  - `from detquantlib.outputs import OutputItem`
  - `from detquantlib.outputs.outputs_interface import OutputItem`
- `OutputSet`:
  - `from detquantlib.outputs import OutputSet`
  - `from detquantlib.outputs.outputs_interface import OutputSet`
- `PathDefinitions`:
  - `from detquantlib.outputs import PathDefinitions`
  - `from detquantlib.outputs.outputs_interface import PathDefinitions`
- `SecretLeakError`:
  - `from detquantlib.snapshot_testing import SecretLeakError`
  - `from detquantlib.snapshot_testing.errors import SecretLeakError`
- `Sftp`:
  - `from detquantlib.data import Sftp`
  - `from detquantlib.data.sftp.sftp import Sftp`
- `SnapshotCase`:
  - `from detquantlib.snapshot_testing import SnapshotCase`
  - `from detquantlib.snapshot_testing.cases import SnapshotCase`
- `SnapshotConfig`:
  - `from detquantlib.snapshot_testing import SnapshotConfig`
  - `from detquantlib.snapshot_testing.config import SnapshotConfig`
- `SnowflakeClient`:
  - `from detquantlib.data import SnowflakeClient`
  - `from detquantlib.data.databases.snowflake_client import SnowflakeClient`
- `SnowflakeConfig`:
  - `from detquantlib.data import SnowflakeConfig`
  - `from detquantlib.data.databases.snowflake_client import SnowflakeConfig`
- `TennetAPIError`:
  - `from detquantlib.data import TennetAPIError`
  - `from detquantlib.data.tennet.tennet_client import TennetAPIError`
- `TennetClient`:
  - `from detquantlib.data import TennetClient`
  - `from detquantlib.data.tennet.tennet_client import TennetClient`
- `WindTurbine`:
  - `from detquantlib.assets import WindTurbine`
  - `from detquantlib.assets.wind_turbine import WindTurbine`
- `WindTurbineModels`:
  - `from detquantlib.assets import WindTurbineModels`
  - `from detquantlib.assets.wind_turbine import WindTurbineModels`

Functions:

- `add_log`:
  - `from detquantlib.utils import add_log`
  - `from detquantlib.utils.utils import add_log`
- `calc_months_diff`:
  - `from detquantlib.dates import calc_months_diff`
  - `from detquantlib.dates.dates import calc_months_diff`
- `convert_delivery_start_date_to_maturity`:
  - `from detquantlib.tradable_products import convert_delivery_start_date_to_maturity`
  - `from detquantlib.tradable_products.tradable_products import convert_delivery_start_date_to_maturity`
- `convert_energy`:
  - `from detquantlib.converters import convert_energy`
  - `from detquantlib.converters.energy import convert_energy`
- `convert_maturity_to_delivery_start_date`:
  - `from detquantlib.tradable_products import convert_maturity_to_delivery_start_date`
  - `from detquantlib.tradable_products.tradable_products import convert_maturity_to_delivery_start_date`
- `convert_power`:
  - `from detquantlib.converters import convert_power`
  - `from detquantlib.converters.energy import convert_power`
- `convert_price`:
  - `from detquantlib.converters import convert_price`
  - `from detquantlib.converters.price import convert_price`
- `count_delivery_periods`:
  - `from detquantlib.tradable_products import count_delivery_periods`
  - `from detquantlib.tradable_products.tradable_products import count_delivery_periods`
- `count_nr_hours`:
  - `from detquantlib.dates import count_nr_hours`
  - `from detquantlib.dates.dates import count_nr_hours`
- `create_case`:
  - `from detquantlib.snapshot_testing import create_case`
  - `from detquantlib.snapshot_testing.cases import create_case`
- `datetime_to_month_code`:
  - `from detquantlib.dates import datetime_to_month_code`
  - `from detquantlib.dates.dates import datetime_to_month_code`
- `discover_cases`:
  - `from detquantlib.snapshot_testing import discover_cases`
  - `from detquantlib.snapshot_testing.cases import discover_cases`
- `divide_with_nan`:
  - `from detquantlib.utils import divide_with_nan`
  - `from detquantlib.utils.utils import divide_with_nan`
- `energy_to_power`:
  - `from detquantlib.converters import energy_to_power`
  - `from detquantlib.converters.energy import energy_to_power`
- `flag_cents`:
  - `from detquantlib.converters import flag_cents`
  - `from detquantlib.converters.price import flag_cents`
- `forecast_knife_strategy`:
  - `from detquantlib.forecasting import forecast_knife_strategy`
  - `from detquantlib.forecasting.forecasting import forecast_knife_strategy`
- `list_to_sql_columns`:
  - `from detquantlib.data import list_to_sql_columns`
  - `from detquantlib.data.databases.helpers import list_to_sql_columns`
- `list_to_sql_set`:
  - `from detquantlib.data import list_to_sql_set`
  - `from detquantlib.data.databases.helpers import list_to_sql_set`
- `list_to_str`:
  - `from detquantlib.utils import list_to_str`
  - `from detquantlib.utils.utils import list_to_str`
- `power_to_energy`:
  - `from detquantlib.converters import power_to_energy`
  - `from detquantlib.converters.energy import power_to_energy`
- `print_cases`:
  - `from detquantlib.snapshot_testing import print_cases`
  - `from detquantlib.snapshot_testing.cases import print_cases`
- `rows_written`:
  - `from detquantlib.snapshot_testing import rows_written`
  - `from detquantlib.snapshot_testing.db_recording import rows_written`
- `setup_logger`:
  - `from detquantlib.utils import setup_logger`
  - `from detquantlib.utils.logging import setup_logger`
- `tenor_to_nr_hours`:
  - `from detquantlib.tradable_products import tenor_to_nr_hours`
  - `from detquantlib.tradable_products.tradable_products import tenor_to_nr_hours`
- `tenor_to_pd_offset_alias`:
  - `from detquantlib.tradable_products import tenor_to_pd_offset_alias`
  - `from detquantlib.tradable_products.tradable_products import tenor_to_pd_offset_alias`
- `ts_ceil`:
  - `from detquantlib.dates import ts_ceil`
  - `from detquantlib.dates.dates import ts_ceil`
- `ts_floor`:
  - `from detquantlib.dates import ts_floor`
  - `from detquantlib.dates.dates import ts_floor`

Constants:

- `DEFAULT_DB_CONNECTION_TARGETS`:
  - `from detquantlib.snapshot_testing import DEFAULT_DB_CONNECTION_TARGETS`
  - `from detquantlib.snapshot_testing.config import DEFAULT_DB_CONNECTION_TARGETS`
- `DEFAULT_DB_QUERY_TARGETS`:
  - `from detquantlib.snapshot_testing import DEFAULT_DB_QUERY_TARGETS`
  - `from detquantlib.snapshot_testing.config import DEFAULT_DB_QUERY_TARGETS`
- `DEFAULT_DB_WRITE_TARGETS`:
  - `from detquantlib.snapshot_testing import DEFAULT_DB_WRITE_TARGETS`
  - `from detquantlib.snapshot_testing.config import DEFAULT_DB_WRITE_TARGETS`
- `DEFAULT_IGNORE_PATTERNS`:
  - `from detquantlib.snapshot_testing import DEFAULT_IGNORE_PATTERNS`
  - `from detquantlib.snapshot_testing.config import DEFAULT_IGNORE_PATTERNS`
- `DEFAULT_SECRET_ENV_VARS`:
  - `from detquantlib.snapshot_testing import DEFAULT_SECRET_ENV_VARS`
  - `from detquantlib.snapshot_testing.config import DEFAULT_SECRET_ENV_VARS`

<!-- END EXPOSED SYMBOLS AUTO-GENERATED -->

## Configuration

### Project Structure

The project mainly follows the standard Poetry structure:

```
detquantlib
├── pyproject.toml
├── README.md
├── detquantlib
│   └── __init__.py
└── tests
    └── __init__.py
```

### Dependencies

#### Dependency Manager

Project dependencies are managed by [Poetry](https://python-poetry.org/).

#### Dependabot

Automated dependency updates are executed with
[Dependabot](https://docs.github.com/en/code-security/dependabot).

### Invoke Tasks

The project uses the [Invoke](https://www.pyinvoke.org/) package to simplify certain tasks such
as testing and code formatting.

All tasks are defined in a tasks.py file.

Invoke tasks can be executed directly from the terminal, using the `inv` (or `invoke`) command
line tool.

For guidance on the available Invoke tasks, execute the following command in the terminal:

```cmd
inv --list
```

Use the `-h` (or `--help`) argument for help about a particular Invoke task. For example:

```cmd
inv lint -h
```

### GitHub Actions

The project's CI/CD pipeline is enforced with [GitHub actions](https://docs.github.com/en/actions)
workflows.

#### Workflow: Python CI

The Python CI (i.e. continuous integration) workflow runs linters to check the consistency of the
codebase's format, and then unit tests to check the integrity of its content.

The workflow was inspired by the following preconfigured templates:

- [Python package](https://github.com/actions/starter-workflows/blob/main/ci/python-package.yml):
  A general workflow template for Python packages.
- [Poetry action](https://github.com/marketplace/actions/install-poetry-action): A GitHub action
  for installing and configuring Poetry.

##### CI Check: Code Formatting

Linters are used to check that the code is properly formatted:

- [Ruff](https://github.com/astral-sh/ruff) to check for errors in the code and in the docstrings
  descriptions, and to sort the imports section
- [Black](https://github.com/psf/black) for the main code
- docs.py to check that the [list of exposed symbols](#list-of-exposed-symbols) in README.md is up
  to date
- [Pymarkdown](https://github.com/jackdewinter/pymarkdown) for the markdown file README.md
- [Deptry](https://github.com/fpgmaas/deptry) to check the dependencies in pyproject.toml for
  unused (main dependencies only), missing, transitive and misplaced dependencies. Deptry only
  reports issues: they must be fixed manually in pyproject.toml.

The CI check is executed with the following Invoke task:

```cmd
inv lint -c
```

If the CI check fails, execute the following command in the terminal:

```cmd
inv lint
```

This command fixes the parts of the code that should be reformatted. Adding the `-c` (or
`--check`) optional argument instructs the command to only _check_ if parts of the code should be
reformatted, without applying any actual changes.

##### CI Check: Unit Tests

Code changes are tested with the [Pytest](https://github.com/pytest-dev/pytest) package.

The CI check is executed with the following Invoke task:

```cmd
inv test -c
```

#### Workflow: Package Publisher

The package publisher workflow checks the validity of package version updates, creates version
tags, and publishes package updates to PyPI.

##### Checking Version Updates

The workflow enforces version control:

- The version number is specified via the `version` field in the pyproject.toml file.
- The version number needs to be updated with every new master commit. If the version is not
  updated, the GitHub workflow will fail.
- Version numbers should follow semantic versioning (i.e. `X.Y.Z`). That is:
  - `X` increments represent major, non-backward compatible updates.
  - `Y` increments represent minor, backward compatible functionality updates.
  - `Z` increments represent patch/bugfix, backward compatible updates.

##### Creating Version Tags

If the new package version is valid, the workflow automatically creates a new tag for every new
master commit.

##### Publishing Package Updates to PyPI

The workflow automatically publishes every new master commit to
[PyPI](https://pypi.org/project/detquantlib/).

### Release Notes

When deemed necessary (especially in case of major updates), developers can document code
changes in dedicated GitHub release notes.

Release notes can be created via <https://github.com/Dynamic-Energy-Trading/detquantlib/releases.>

In any case, all codes changes should always be properly described/documented in GitHub issues
and/or pull requests.

