Metadata-Version: 2.4
Name: kohlrahbi
Version: 1.13.1
Summary: Tool to generate machine readable files from AHB documents
Project-URL: Changelog, https://github.com/Hochfrequenz/kohlrahbi/releases
Project-URL: Homepage, https://github.com/Hochfrequenz/kohlrahbi
Author-email: Kevin Krechan <kevin.krechan@hochfrequenz.de>
License: GPL
License-File: LICENSE
Keywords: ahb,automation,bdew,edi@energy
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Requires-Dist: beautifulsoup4==4.15.0
Requires-Dist: edi-energy-scraper>=2.0.0
Requires-Dist: efoli>=2.2.0
Requires-Dist: httpx==0.28.1
Requires-Dist: more-itertools>=10.5.0
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: pandas>=3.0.2
Requires-Dist: pdfplumber==0.11.10
Requires-Dist: pydantic>=2.9.2
Requires-Dist: python-docx>=1.1.2
Requires-Dist: pytz>=2024.2
Requires-Dist: tomlkit>=0.13.2
Requires-Dist: typer[all]>=0.15.0
Requires-Dist: xlsxwriter>=3.2.0
Provides-Extra: sqlmodels
Requires-Dist: sqlalchemy[mypy]>=2.0.37; extra == 'sqlmodels'
Requires-Dist: sqlmodel>=0.0.22; extra == 'sqlmodels'
Description-Content-Type: text/markdown

# kohlr_AHB_i

<p align="center">
  <img src="kohlrahbi-image.png" alt="kohlrahbi-logo" width="512" height="512">
</p>

[![License: GPL](https://img.shields.io/badge/License-GPL-yellow.svg)](LICENSE)
![Python Versions (officially) supported](https://img.shields.io/pypi/pyversions/kohlrahbi.svg)
![Unittests status badge](https://github.com/Hochfrequenz/kohlrahbi/workflows/Unittests/badge.svg)
![Coverage status badge](https://github.com/Hochfrequenz/kohlrahbi/workflows/Coverage/badge.svg)
![Checks status badge](https://github.com/Hochfrequenz/kohlrahbi/workflows/checks/badge.svg)
![PyPI](https://img.shields.io/pypi/v/kohlrahbi)

Kohlr_AHB_i generates machine-readable files from AHB documents.
Kohlr_AHB_i's sister is [MIG_mose](https://github.com/Hochfrequenz/migmose).

If you'd like to create a nice diff view of AHBs in different versions, try [`ahlbatross`](https://github.com/Hochfrequenz/ahlbatross).

If you're looking for a tool to process the **official** BDEW XMLs for AHBs (available since 2024), checkout [fundamend](https://github.com/Hochfrequenz/xml-fundamend-python).

## Rationale

German utilities exchange data using [EDIFACT](https://en.wikipedia.org/wiki/EDIFACT); This is called market communication (mako).
The _Forum Datenformate_ of the BDEW publishes the technical regulations of the EDIFACT based market communication on [`edi-energy.de`](https://www.edi-energy.de/).
These rules are not stable but change twice a year (in theory) or few times per year (in reality).

Specific rules, which are binding for every German utility are kind of formalised in so called "**A**nwendungs**h**and**b**üchern" (AHB).
Those AHBs are basically long tables that describe:
> As a utility, if I want to exchange data about business process XYZ with a market partner, then I have to provide the following information: [...]

In total the regulations from these Anwendungshandbücher span several thousand pages.
And by pages, we really _mean_ pages.
EDIFACT communication is basically the API between German utilities for most of their B2B processes.
However, the technical specifications of this API are

* prose
* on DIN A4 pages.

The Anwendungshandbücher are the epitome of digitization with some good intentions.

Although the AHBs are publicly available as PDF or Word files on `edi-energy.de`, they are hardly accessible in a technical sense:

* You cannot automatically extract information from the AHBs.
* You cannot run automatic comparisons between different versions.
* You cannot automatically test your own API against the set of rules, described in the AHBs (as prose).
* You cannot view or visualize the information from the AHBs in any more intuitive or practical way, than the raw tables from the AHB files.
* ...any many more...

The root cause for all these inaccessibility is a technical one:
Information that are theoretically structured are published in an unstructured format (PDF or Word), which is not suited for technical specifications in IT.

Kohlr_AHB_i as a tool helps you to break those chains and access the AHBs as you'd expect it from technical specs: easy and automatically instead of with hours of mindless manual work.

**Kohlr_AHB_i takes the `.docx` files published by `edi-energy.de` as an input and returns truly machine-readable data in a variety of formats (JSON, CSV...) as a result.**

Hence, Kohlr_AHB_i is the key for unlocking any automation potential that is reliant on information hidden in the Anwendungshandbücher.

We're all hoping for the day of true digitization on which this repository will become obsolete.
**Update 2025**: Indeed the machine-readable (XML) MIGs and AHBs available in a paid BDEW subscription make lots of the kohlrahbi features obsolete 🎉.
Still, some issues remain:
  1. The XML AHBs do not contain an "Änderungshistorie" although with some work you could compute them from the XMLs (more of a ahlbatross feature)
  2. Kohlr_AHB_i generates the data model for the AHB Tabellen web application.

## Installation

Kohlr_AHB_i is a Python based tool.
Therefore you have to make sure, that Python is running on your machine.

We recommend to use virtual environments to keep your system clean.

Create a new virtual environment with

```bash
python -m venv .venv
```

The activation of the virtual environment depends on your used OS.

### Windows

```powershell
.venv\Scripts\activate
```

### MacOS/Linux

```zsh
source .venv/bin/activate
```

Finally, install the package with

```bash
pip install kohlrahbi
```

## Usage

Kohlrahbi is a command line tool. Run `kohlrahbi --help` to see all available commands.

```bash
kohlrahbi --help
```

### Commands overview

| Command                          | Description                                                  |
| -------------------------------- | ------------------------------------------------------------ |
| `kohlrahbi ahb`                  | Extract AHB tables from `.docx` files                        |
| `kohlrahbi conditions`           | Extract conditions and packages from `.docx` files           |
| `kohlrahbi changehistory docx`   | Extract change histories from `.docx` files                  |
| `kohlrahbi changehistory bnetza` | Download documents from a BNetzA URL and extract change histories |

---

### `kohlrahbi ahb` — Extract AHB tables

> [!NOTE]
> This command requires a local clone of the [edi_energy_mirror](https://github.com/Hochfrequenz/edi_energy_mirror/) repository, which contains the `.docx` files of the AHBs.
> The folder structure should look like this:
> ```plaintext
> .
> ├── edi_energy_mirror
> └── kohlrahbi
> ```

Extract all AHB tables for a specific format version:

```bash
kohlrahbi ahb -eemp ../edi_energy_mirror/ --output-path ./output/ --file-type csv --format-version FV2310
```

Extract tables for specific Prüfidentifikatoren:

```bash
kohlrahbi ahb -eemp ../edi_energy_mirror/ --output-path ./output/ --file-type csv --pruefis 13002 --format-version FV2310
```

You can provide multiple `--pruefis` and multiple `--file-type` values:

```bash
kohlrahbi ahb -eemp ../edi_energy_mirror/ --output-path ./output/ \
  --file-type csv --file-type xlsx --file-type flatahb \
  --pruefis 13002 --pruefis 13003 --pruefis 13005 \
  --format-version FV2310
```

To remove old output files for Prüfidentifikatoren that no longer appear in the input, add `--clear-output-path`:

```bash
kohlrahbi ahb -eemp ../edi_energy_mirror/ --output-path ./output/ --file-type csv --format-version FV2310 --clear-output-path
```

---

### `kohlrahbi conditions` — Extract conditions and packages

> [!NOTE]
> This command also requires a local clone of the [edi_energy_mirror](https://github.com/Hochfrequenz/edi_energy_mirror/).

```bash
kohlrahbi conditions -eemp ../edi_energy_mirror/ --output-path ./output/ --format-version FV2310
```

This extracts all conditions and packages found in all AHBs (including the condition texts from package tables) within the `.docx` files. The output is saved per EDIFACT format as `conditions.json` and `packages.json`.

> [!NOTE]
> The conditions collected here may be more comprehensive than those collected via `kohlrahbi ahb`, because `conditions` uses a different extraction routine.

---

### `kohlrahbi changehistory` — Extract change histories

The `changehistory` command has two subcommands depending on your data source.

#### `kohlrahbi changehistory docx` — From `.docx` files

> [!NOTE]
> This command requires a local clone of the [edi_energy_mirror](https://github.com/Hochfrequenz/edi_energy_mirror/).

```bash
kohlrahbi changehistory docx -eemp ../edi_energy_mirror/ --output-path ./output/ --format-version FV2310
```

#### `kohlrahbi changehistory bnetza` — From BNetzA documents

Downloads all linked documents from a BNetzA "Mitteilung" URL, extracts the change history
(`Änderungshistorie`) tables, and writes one Excel sheet per document.

It handles every document the page links, not just PDFs: newer pages serve most EDIFACT
documents as `.html`-named downloads (whose body is actually a PDF) and some as Office files.
Each file's real type is detected from its content, so nothing is skipped because of its URL
extension.

```bash
kohlrahbi changehistory bnetza \
  --url "https://www.bundesnetzagentur.de/DE/Beschlusskammern/BK06/BK6_83_Zug_Mess/835_mitteilungen_datenformate/Mitteilung_55/Mitteilung_Nr_55.html" \
  --output-path ./output/
```

The downloaded documents are saved to `<output-path>/pdfs/` and the resulting Excel file to
`<output-path>/change_history.xlsx`. On completion a summary reports how many links were found,
how many documents were downloaded (by type), how many sheets were written, and which documents
contained no change history.

## `.docx` Data Sources

Kohlr_AHB_i internally relies on a [specific naming schema](https://github.com/Hochfrequenz/kohlrahbi/blob/22a78dc076c7d5f9248cb9e8707b0cc14a2981d3/src/kohlrahbi/read_functions.py#L57) of the `.docx` files in which the file name holds information about the edifact format and validity period of the AHBs contained within the file.
The easiest way to be compliant with this naming schema is to clone our [edi_energy_mirror](https://github.com/Hochfrequenz/edi_energy_mirror/) repository to your localhost.

## Results

There is a kohlr_AHB_i based CI pipeline from the edi_energy_mirror mentioned above to the repository [machine-readable_anwendungshandbuecher](https://github.com/Hochfrequenz/machine-readable_anwendungshandbuecher) where you can find scraped AHBs as JSON, CSV or Excel files.

## Workflow

```mermaid
flowchart TB
    S[Start] --> RD[Read docx]
    RD --> RPT[Read all paragraphs <br> and tables]
    RPT --> I[Start iterating]
    I --> NI[Read next item]
    %% check for text paragraph %%
    NI --> CTP{Text Paragraph?}
    CTP -- Yes --> NI
    CTP -- No --> CCST{Is item just<br>Chapter or Section Title?}
    CCST -- Yes --> CTAenderunghistorie{Is Chapter Title<br>'Änderungshistorie'?}
    CTAenderunghistorie -- Yes --> EXPORT[Export Extract]
    CCST -- No --> CT{Is item a table<br>with prüfis?}
    CT -- Yes --> Extract[Create Extract]
```

## AHB page number per Format

The following table shows the page number of the AHBs for each format of the format version FV2310.

| Format       | Page number | Hint                                                |     |
| ------------ | ----------- | --------------------------------------------------- | --- |
| UTILMD Strom | 1064        |                                                     |     |
| UTILMD Gas   | 345         |                                                     |     |
| REQOTE       | 264         | together with QUOTES, ORDERS, ORDRSP, ORDCHG        |     |
| QUOTES       | 264         | together with REQOTE, ORDERS, ORDRSP, ORDCHG        |     |
| ORDRSP       | 264         | together with REQOTE, QUOTES, ORDERS, ORDCHG        |     |
| ORDERS       | 264         | together with REQOTE, QUOTES, ORDRSP, ORDCHG        |     |
| ORDCHG       | 264         | together with REQOTE, QUOTES, ORDERS, ORDRSP        |     |
| MSCONS       | 164         |                                                     |     |
| UTILMD MaBis | 133         |                                                     |     |
| REMADV       | 91          | together with INVOIC                                |     |
| INVOIC       | 91          | together with REMADV                                |     |
| IFTSTA       | 82          |                                                     |     |
| CONTRL       | 72          | together with APERAK, contains no Prüfis            |     |
| APERAK       | 72          | together with CONTRL, contains no Prüfis            |     |
| PARTIN       | 69          |                                                     |     |
| UTILTS       | 34          |                                                     |     |
| ORDRSP       | 30          | together with ORDERS                                |     |
| ORDERS       | 30          | together with ORDRSP                                |     |
| PRICAT       | 25          |                                                     |     |
| COMDIS       | 10          | good test for tables which are above change history |     |

## Development

This project uses [uv](https://docs.astral.sh/uv/) to manage dependencies and the development environment.

### Setup

To set up the development environment, install [uv](https://docs.astral.sh/uv/getting-started/installation/) and sync the dev dependency group.

```bash
uv sync --group dev
```

This creates a `.venv` and installs the package together with all development tools (tests, linting, type checking, formatting, spelling).

### Run tests, linters and other checks

```bash
uv run --group test pytest --cov kohlrahbi --cov-report term-missing --cov-fail-under 79
uv run --group lint ruff check src/kohlrahbi
uv run --group typecheck mypy src/kohlrahbi unittests --strict
uv run --group lint ruff format --check src/kohlrahbi unittests
uv run --group spelling codespell src/kohlrahbi README.md
```

See our [Python Template Repository](https://github.com/Hochfrequenz/python_template_repository#how-to-use-this-repository-on-your-machine) for detailed explanations.

## Contribute

You are very welcome to contribute to this template repository by opening a pull request against the main branch.

## Related Tools and Context

This repository is part of the [Hochfrequenz Libraries and Tools for a truly digitized market communication](https://github.com/Hochfrequenz/digital_market_communication/).
