Metadata-Version: 2.5
Name: otb-cbso-webservice-pyclient
Version: 2.1.1
Summary: Download annual account data from the Belgian National Bank's Central Balance Sheet Office webservices
Project-URL: Homepage, https://github.com/openthebox/cbso-webservice-pyclient
Project-URL: Issues, https://github.com/openthebox/cbso-webservice-pyclient/issues
Project-URL: Changelog, https://github.com/openthebox/cbso-webservice-pyclient/releases
Author: openthebox
License: Proprietary
Keywords: annual accounts,belgium,cbso,nbb,xbrl
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: fsspec>=2024.6
Requires-Dist: urllib3>=2.0
Provides-Extra: dev
Requires-Dist: coverage[toml]>=7.4; extra == 'dev'
Requires-Dist: mypy>=1.9; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: s3
Requires-Dist: s3fs>=2024.6; extra == 's3'
Description-Content-Type: text/markdown

# otb-cbso-webservice-pyclient

Download Belgian annual accounts from the National Bank's Central Balance Sheet Office (CBSO).
Use the Python package or the `cbso-fetch` command.

The package downloads selected batches and files accounts by enterprise.
Validation and repair run as separate operations. Downloads and repairs run sequentially.
The package can also download missing PDFs for PDF-only filings and collect consolidated accounts.

## Installation

Use Python 3.10 or later.

```console
pip install otb-cbso-webservice-pyclient
```

Install the S3 extra for S3 destinations.

```console
pip install 'otb-cbso-webservice-pyclient[s3]'
```

The core dependencies are `urllib3` and `fsspec`. The S3 extra adds `s3fs` and its dependencies.

## Configure subscriptions

Each CBSO product requires its own subscription key.
Choose the products that match your subscriptions and the data you need.

| Batch product       | Contents                                           | Environment variable         |
| ------------------- | -------------------------------------------------- | ---------------------------- |
| `extracts`          | Daily batches of accounts as filed                 | `CBSO_KEY_EXTRACTS`          |
| `authentic-archive` | Archive batches of accounts as filed               | `CBSO_KEY_AUTHENTIC_ARCHIVE` |
| `improved`          | Daily batches of corrected or converted accounts   | `CBSO_KEY_IMPROVED`          |
| `improved-archive`  | Archive batches of corrected or converted accounts | `CBSO_KEY_IMPROVED_ARCHIVE`  |

Set the key for each product you want to download.

```console
export CBSO_KEY_EXTRACTS='your-extracts-subscription-key'
```

Repair and PDF-only downloads can need individual-access subscriptions.

| Environment variable | Used for                                                       |
| -------------------- | -------------------------------------------------------------- |
| `CBSO_KEY_AUTHENTIC` | Missing enterprise references and individual accounts as filed |
| `CBSO_KEY_IMPROVED`  | Individual improved references and account files               |

A batch request checks its product key. Reusing retained ZIP files requires no CBSO keys.
Repair reports missing keys and continues with work that can use available subscriptions or saved data.
Validation does not require CBSO subscription keys.

## Command line

| Subcommand                     | Purpose                                                  | Changes saved files? |
| ------------------------------ | -------------------------------------------------------- | -------------------- |
| `download`                     | Fetch and extract selected batches                       | Yes                  |
| `validate`                     | Check selected formats against authentic references      | No                   |
| `validate-improved`            | Check improved JSON and expected OCR files               | No                   |
| `repair-improved`              | Restore missing improved JSON across saved references    | Yes                  |
| `repair`                       | Restore missing files and resolve enterprise information | Yes                  |
| `download-pdf-only-accounts`   | Download missing PDFs for PDF-only filings               | Yes                  |
| `filter-consolidated-accounts` | Collect consolidated references and move their PDFs      | Yes                  |

`download` requires dates, `--products`, and `--destination`.
Supply past dates in `YYYY-MM-DD` form. Replace the example dates with the dates you need.
Product availability determines which dates the service can supply.
All other commands accept an input directory and scan files across all dates.
`validate` and `repair` accept `--formats`; the default is `json`.
`validate-improved` and `repair-improved` handle improved JSON only.
`repair-improved`, `download-pdf-only-accounts`, and `filter-consolidated-accounts` also support `--dry-run`.

### Download batches

`download` fetches references and accounts for each selected date and product.
References connect deposit numbers to enterprise numbers. The command uses that connection to file accounts into enterprise folders.

Download one date as JSON-XBRL.

```console
cbso-fetch download 2024-08-03 --products extracts --destination ./data
```

Download multiple dates and formats, with copies of the original ZIP files.

```console
cbso-fetch download 2024-08-03 2024-08-04 \
  --products extracts \
  --destination ./data \
  --formats json xbrl pdf \
  --keep-zip
```

Select archive products directly to request archive batches.
This example stores both authentic and improved JSON-XBRL accounts in S3.

```console
cbso-fetch download 2024-08-03 \
  --products authentic-archive improved-archive \
  --destination s3://my-bucket/cbso
```

| Option                    | Behaviour                                                            |
| ------------------------- | -------------------------------------------------------------------- |
| `--formats json xbrl pdf` | Select account formats. The default is `json`.                       |
| `--keep-zip`              | Retain original reference and account ZIP files. The default is off. |

`json` selects JSON-XBRL account data. Reference files use ordinary JSON.
Authentic products support all three formats. Improved products support only `json`.
A command containing improved products must select only JSON.
Use separate commands when you need other formats from authentic products.

Existing ZIP files under `archives/{product}/{date}/` are reused without a batch request.
Every run extracts the selected ZIP files, including retained copies. Existing output files are overwritten.
Without retained ZIP files, repeated runs download the batches again. ZIP retention is off by default.
Dates are processed in the supplied order.
Entries without usable enterprise information remain under `pending/` for inspection or repair.
Downloading does not request individual accounts or automatically repair missing files.

### Validate saved files

`validate` scans JSON files under `references/` and checks the selected account formats.
It uses reference contents to determine enterprise numbers, deposit numbers, and expected file paths.
The scan includes all dates. It does not require download history.

```console
cbso-fetch validate ./data --formats xbrl pdf
```

Omit `--formats` to check JSON only.
Missing authentic JSON and XBRL for PDF-only accounts appear under `expected_unavailable`.
Pending reference files and pending accounts in selected formats appear under `unresolved` in Python reports.

Validation checks file presence. It does not inspect account contents or validate XBRL schemas.
It makes no CBSO requests and leaves stored files unchanged.
An S3 destination still requires access to its stored objects.
A missing reference directory is an error unless matching pending files exist.

A reference scan cannot detect a deposit when both its reference and account files are absent.
An empty reference directory produces an empty report. This does not confirm that all service data was downloaded.
Run `download` again to restore missing references from batches.

### Validate improved files

`validate-improved` scans JSON files under `improved-references/`.
It checks the improvement named by each reference and reports unresolved pending improvements.
For PDF-based deposits, it also checks OCR JSON, even without a separate OCR reference.

```console
cbso-fetch validate-improved ./data
cbso-fetch validate-improved s3://my-bucket/annual-accounts
```

Improved accounts support JSON only, so this command has no `--formats` option.
It makes no CBSO requests and changes no files.
Run `repair-improved` to restore missing improved files.

### Repair missing files

`repair` scans authentic references across all dates and restores selected account formats.

```console
cbso-fetch repair ./data --formats xbrl pdf
```

It first resolves damaged references and missing enterprise numbers through individual reference requests.
It then files pending accounts using saved or recovered references.
The pending directory supplies each file's product and format. The filename supplies its deposit number.
Repair preserves existing accounts and removes pending copies only after the destination file exists.
It requests remaining account gaps individually. These requests require `CBSO_KEY_AUTHENTIC`.

The Python report includes restored paths and any work that remains incomplete. The CLI prints counts instead of paths.
Unreadable references, missing keys, and unavailable accounts remain visible in the Python report and logs.
Repair does not request authentic JSON or XBRL for PDF-only accounts.

Repair does not download or extract batch ZIP files.
Run `download` to extract retained ZIP files or download batches whose ZIP files were not retained.
Use `repair-improved` for improved references and accounts.

### Repair improved accounts across all dates

```console
export CBSO_KEY_IMPROVED='your-improved-subscription-key'
cbso-fetch repair-improved ./data
cbso-fetch repair-improved s3://my-bucket/annual-accounts
```

`repair-improved` scans all JSON files beneath `improved-references/`, including enterprise subdirectories.
It uses no date or product filters.
It accepts camelCase and PascalCase reference fields.

It checks the improvement identified by each reference:

| Reference improvement     | Output filename                        |
| ------------------------- | -------------------------------------- |
| Annual-account correction | `{reference}-aa-corrected.json`        |
| Social-balance correction | `{reference}-sb-corrected.json`        |
| OCR extraction            | `{reference}-ocr-authentic.json`       |
| Euro conversion           | `{reference}-euro-authentic.json`      |

For models ending in `-p`, it also requests missing OCR JSON, even without a matching OCR reference.
It does not infer additional corrections or euro conversions without their reference metadata.

Files go into `improved-jsons/{enterprise}/`. Existing files remain unchanged.
Duplicate references cause at most one request per output path during a run.
Processing is sequential. Individual requests use `CBSO_KEY_IMPROVED`.
Use `--env test` for UAT2 and the same environment as the source references.
S3 requires the optional `s3` installation and storage credentials.

The command first resolves pending improvements and missing enterprise information.
Individual improved-reference requests must match both the deposit number and improvement label.
Recovered metadata goes into `improved-references/{enterprise}/`.
Repair does not substitute authentic metadata for missing improved metadata.

Inferred OCR downloads create account files only. They do not create reference metadata or invent publication dates.
Future validation and repair scans check those OCR files through the original PDF-based references.
The command does not scan authentic `references/`.

Preview downloads without CBSO requests or file changes:

```console
cbso-fetch repair-improved ./data --dry-run
```

The preview prints readable `DOWNLOAD`, `EXISTS`, and `INVALID` lines.
Paths are relative to the input directory. No CBSO key is needed for a preview.
Pending files appear as `INVALID`. The preview does not plan pending-file moves or metadata recovery.
Readable pending references can produce planned account downloads, even when matching pending account files exist.

Use the same operation from Python:

```python
from cbso_webservice_client import repair_improved

report = repair_improved("./data")
print(report.to_dict())

preview = repair_improved("./data", dry_run=True)
print(preview.planned)
```

The function also accepts `environment="test"` and a caller-owned `client`.
Normal runs create and close a client when none is supplied. Dry runs create no client.

The Python report lists restored, existing, planned, missing, unavailable, and unresolved paths.
Invalid references, missing enterprise numbers, missing keys, and HTTP 404 responses remain visible; processing continues.
Normal CLI runs print JSON counts. Exit code `1` indicates incomplete work.
Other service or storage failures stop processing and produce exit code `4`.
A missing `improved-references/` directory is a storage failure unless pending improvements exist.

### Download PDFs for PDF-only accounts

```console
export CBSO_KEY_AUTHENTIC='your-authentic-subscription-key'
cbso-fetch download-pdf-only-accounts ./data
cbso-fetch download-pdf-only-accounts s3://my-bucket/annual-accounts
```

Preview the downloads without changing files:

```console
cbso-fetch download-pdf-only-accounts ./data --dry-run
```

Dry runs print readable `DOWNLOAD`, `EXISTS`, and `INVALID` lines with paths relative to the input directory.
They read references and check PDF locations, but make no CBSO requests or file changes.
No CBSO subscription key is needed. S3 still requires storage access.
A planned download does not confirm that CBSO has the PDF or that the destination permits writes.
Invalid references produce exit code `1`; planned downloads alone produce exit code `0`.

This command scans all JSON files beneath `references/`, including enterprise subdirectories.
It selects references whose model code ends in `-p`, which marks a PDF-based filing.
It accepts camelCase and PascalCase fields. It uses no date or product filters.

Each missing PDF is requested individually through the authentic service and saved under `pdfs/{enterpriseNumber}/{referenceNumber}.pdf`.
This is the same path used by PDF batch downloads.
Existing PDFs remain unchanged. Duplicate references cause at most one request per output path during a run.
The operation makes no batch requests and leaves references unchanged.
Processing is sequential.

Use `--env test` for UAT2. Use the same environment as the source references.
S3 requires the optional `s3` installation and storage credentials.
The authentic subscription key is needed only when a PDF request is necessary.

Use the library function with the same input directory:

```python
from cbso_webservice_client import download_pdf_only_accounts

report = download_pdf_only_accounts("./data")
print(report.to_dict())
```

Pass `dry_run=True` to preview downloads from Python:

```python
report = download_pdf_only_accounts("./data", dry_run=True)
print(report.planned)
```

Dry runs populate `planned` with output paths and leave `written` empty.
Normal runs leave `planned` empty.
The function also accepts `environment="test"` and an optional `client=Client(...)`.
The caller owns an explicitly supplied client. Otherwise, normal runs create and close their own client. Dry runs create no client.

The returned `Report` uses these fields:

- `written`: saved PDFs.
- `skipped`: PDFs already present.
- `unresolved`: invalid references or PDF-based references without a valid enterprise number.
- `missing`: PDFs that could not be downloaded because of missing keys or HTTP 404 responses.
- `unavailable`: PDFs for which CBSO returned HTTP 404.
- `missing_keys`: required subscription keys that were absent.

Invalid references, missing keys, and HTTP 404 responses do not stop processing other references.
The CLI prints JSON counts. Exit code `1` indicates incomplete work; exit code `0` indicates completion.
Other service or storage failures stop processing and produce exit code `4`.
A missing `references/` directory is a storage failure.

Run this command before `filter-consolidated-accounts`.
PDF-only downloads also check `consolidated_accounts/BE{enterpriseNumber}_{referenceNumber}.pdf`.
If filtering has moved a PDF there, later downloads and dry runs report that path as skipped.
Validation uses the formats selected with `--formats`, which defaults to `json`.
Use `--formats pdf` to check PDFs against saved references. Commands do not store validation records.

### Filter consolidated accounts

Run validation and any required repair before filtering.
Filtering moves PDFs out of their original folders. Later validation can report them as missing, and repair can download them again.

```console
cbso-fetch filter-consolidated-accounts ./data
cbso-fetch filter-consolidated-accounts s3://my-bucket/annual-accounts
```

Preview the copies and moves without changing files:

```console
cbso-fetch filter-consolidated-accounts ./data --dry-run
```

Dry runs print readable `COPY` and `MOVE` lines with source and destination paths.
Paths are relative to the input directory shown in the first line.
`EXISTS`, `MISSING`, and `INVALID` lines report existing output PDFs, missing PDFs, and invalid references.
Dry runs read references and check file locations. They do not create directories, write files, delete files, or read PDF contents.
Exit codes follow the normal filtering rules.

This command scans all JSON files beneath `references/`, without date or product filters.
It selects the fixed model codes `m120-f-p`, `m121-f-p`, `m122-f-p`, and `m123-f-p`.
It accepts camelCase and PascalCase reference fields.
S3 requires the optional `s3` installation and storage credentials. CBSO keys are unnecessary.

For each selected reference, the command checks `pdfs/{enterpriseNumber}/{referenceNumber}.pdf`.
When that PDF exists, it writes these files:

```text
consolidated_accounts/BE{enterpriseNumber}_{referenceNumber}-reference.json
consolidated_accounts/BE{enterpriseNumber}_{referenceNumber}.pdf
```

The reference copy preserves the original bytes. Source references remain in place.
Existing output files are overwritten.
The source PDF is deleted only after the output PDF is saved successfully.
Processing is sequential.

If the source PDF is absent but the output PDF exists, the command treats the PDF as complete.
It still refreshes the output reference.
If both PDFs are absent, the reference is copied to this error path:

```text
consolidated_accounts_errors/BE{enterpriseNumber}_{referenceNumber}-reference.json
```

Existing error copies remain after later successful runs.
Invalid JSON, missing required fields, and unsafe identifiers appear in the report. Processing continues with other references.
Storage failures stop processing. A missing `references/` directory is a storage failure.

The CLI summary includes counts for `planned_copies`, `planned_moves`, `copied_references`, and `moved_pdfs`.
It also includes `existing_pdfs`, `missing_pdfs`, `invalid_references`, and `complete`.
All fields contain integer counts except `complete`, which remains a boolean. Python reports retain paths and error reasons.
Missing PDFs alone produce exit code `0`. Invalid references produce exit code `1`; storage failures produce exit code `4`.
For this command, `complete` means no invalid references were found. Missing PDFs do not change that value.

Use the same operation from Python:

```python
from cbso_webservice_client import filter_consolidated_accounts

report = filter_consolidated_accounts("./data")
print(report.to_dict())
```

The function accepts a local path or S3 URL and returns a `ConsolidatedReport`.
Pass `dry_run=True` to preview transfers from Python:

```python
report = filter_consolidated_accounts("./data", dry_run=True)
print(report.planned_copies)
print(report.planned_moves)
```

`planned_copies` and `planned_moves` map source paths to destination paths.
Dry runs leave `copied_references` and `moved_pdfs` empty. Normal runs leave the planned transfer mappings empty.
It does not download files.

### Argument order and help

Options can precede positional dates.
However, `--products` and `--formats` accept multiple values and can consume dates that follow them directly.
Use `--` to separate trailing dates from those options.

```console
cbso-fetch download --destination ./data --products extracts -- 2024-08-03 2024-08-04
```

View command descriptions and options with `--help`.

```console
cbso-fetch --help
cbso-fetch download --help
cbso-fetch validate --help
cbso-fetch validate-improved --help
cbso-fetch repair --help
cbso-fetch repair-improved --help
cbso-fetch download-pdf-only-accounts --help
cbso-fetch filter-consolidated-accounts --help
```

### Environments and logging

Production is the default CBSO environment.
Use `--env test` with `download`, `repair`, `repair-improved`, or `download-pdf-only-accounts` to select UAT2.
Both validation commands and consolidated filtering have no CBSO environment option.

```console
cbso-fetch download 2024-08-03 \
  --products extracts \
  --destination ./test-data \
  --env test
```

Configure subscription keys for the selected environment.
Use separate destinations for production and test data. Use the same environment when repairing a destination.

Add `-v` or `--verbose` to show operation starts, progress, and summaries.
Add `-vv` to show individual downloads, reads, writes, copies, moves, deletions, and archive entries.
Debug logs also show HTTP response status, size, and duration.
Warnings and errors appear by default. Logs go to stderr. Reports and previews go to stdout.
Scans report progress every 100 files and at completion. Downloads report each batch.
Dry-run logs identify planned actions. They do not report those actions as completed.

```console
cbso-fetch download 2026-09-14 --products extracts --destination ./data -vv
```

The library uses Python's standard `logging` module. Configure logging in the calling application.
Library loggers use names beneath `cbso_webservice_client`.
Use `cbso-fetch --version` to show the installed package version.

## Python package

Import operations from `cbso_webservice_client`.
`download` accepts one date string or a sequence of date strings.
Pass products as a sequence of strings or `Product` enum members.
The other operations accept an input directory.
`validate` and `repair` also accept selected formats.
`repair_improved`, `download_pdf_only_accounts`, and `filter_consolidated_accounts` accept an optional `dry_run=True` argument.

### Download from Python

```python
from cbso_webservice_client import Product, download

report = download(
    dates=["2024-08-03", "2024-08-04"],
    products=[Product.EXTRACTS],
    destination="./data",
    formats=["json", "pdf"],
    keep_zip=True,
)

print(report.written)
print(report.unresolved)
```

`formats` defaults to JSON. `keep_zip` defaults to `False`.
`destination` accepts a local path or an S3 URL.
Use `environment="test"` with `download`, `repair`, `repair_improved`, or `download_pdf_only_accounts` for UAT2.

### Validate and repair from Python

These functions scan stored reference files. No earlier download process or saved format selection is required.

```python
from cbso_webservice_client import repair, repair_improved, validate, validate_improved

validation = validate("./data", formats=["xbrl", "pdf"])
print(validation.to_dict())

repaired = repair("./data", formats=["xbrl", "pdf"])
print(repaired.to_dict())

improved_validation = validate_improved("./data")
improved_repaired = repair_improved("./data")
```

Validation never invokes repair. Call repair only when you want to restore files.

### Supply keys directly

Use a `Client` context to pass keys in code and reuse its HTTP connections.
Non-empty keys supplied to the client take precedence over environment variables.

```python
from cbso_webservice_client import Client, download, repair

with Client(
    keys={
        "extracts": "your-extracts-subscription-key",
        "authentic": "your-authentic-subscription-key",
    },
    environment="production",
) as client:
    downloaded = download("2024-08-03", ["extracts"], "./data", client=client)
    repaired = repair("./data", formats=["json"], client=client)
```

An explicitly supplied client determines the request environment.
The caller owns that client; the context closes it after use.
Operations that make CBSO requests create and close their own clients when none is supplied.
Directory dry runs create no CBSO client and do not use a supplied client.

## Reports and errors

Downloads and repairs return `Report`; validation also returns `Report`.
Consolidated filtering returns `ConsolidatedReport`, with the fields described in its command section.
Both types provide `to_dict()` for JSON-compatible output.

| Field                  | Meaning                                                               |
| ---------------------- | --------------------------------------------------------------------- |
| `planned`              | Output paths proposed by a dry run                                    |
| `written`              | Paths stored or restored during the operation                         |
| `skipped`              | Reused ZIP files or accounts already present                          |
| `missing`              | Expected files that are absent                                        |
| `missing_batches`      | Batch ZIP paths awaiting publication during download                  |
| `unresolved`           | Entries or reference data that cannot be processed or filed           |
| `unavailable`          | Files for which CBSO returned HTTP 404                                |
| `expected_unavailable` | Absent authentic JSON or XBRL for PDF-only filings                    |
| `missing_keys`         | Missing subscription keys for individual requests                     |
| `complete`             | Whether the operation has no outstanding work                         |

`complete` describes the operation performed, not the entire destination.
For dry runs, planned downloads do not make the report incomplete or confirm service availability.
Report paths are relative to the input directory or destination.
Use `validate` to check file presence, including files removed after an earlier download.
A repair report can list a file as both missing and unavailable.

Library failures raise exceptions. These include `MissingKeyError`, `HttpError`, `ValueError`, storage errors, and `urllib3` connection errors.
Missing keys during individual repair and PDF-only requests appear in the report instead.

Completed CLI commands print JSON counts, including any incomplete work.
Python reports and their `to_dict()` methods retain file paths and error details.

Each path field in a standard CLI report contains these count categories, including zero counts:

- `json`, `xbrl`, and `pdf`: account files, including improved JSON.
- `references`: JSON files beneath `references/` or `improved-references/`, including pending references.
- `zip`: archive files, including reference archives.
- `other`: entries with other extensions or no extension.

Counts include each report entry. `missing_keys` contains the number of reported missing keys.
Consolidated filtering uses one integer per field because its field names identify the file types.
CLI summaries omit `missing_batches`, `unavailable`, and `unresolved`. Python reports retain these fields.
HTTP 404 responses for missing accounts remain included in `missing` counts.
Other failures remain in logs and Python reports. They can make `complete` false even when displayed counts contain no failures.
`expected_unavailable` counts absent authentic JSON and XBRL files for PDF-only filings.
These entries do not count as missing or skipped and do not make `complete` false.
`complete` remains a boolean. A complete report produces exit code `0`; an incomplete report produces exit code `1`.

For example, a `missing` field can contain:

```json
{"json": 0, "xbrl": 120, "pdf": 0, "references": 0, "zip": 0, "other": 0}
```

Consolidated filtering, PDF-only downloads, and improved repair scans with `--dry-run` print readable text instead.
The filtering command treats missing PDFs as informational.
See [Filter consolidated accounts](#filter-consolidated-accounts) for its exit behaviour.
Command failures write an error message to standard error.

| Exit code | Meaning                                                      |
| --------- | ------------------------------------------------------------ |
| `0`       | The operation completed without outstanding work.            |
| `1`       | Work remains incomplete, or stored data cannot be processed. |
| `2`       | Command arguments are invalid.                               |
| `3`       | A selected batch subscription key is missing.                |
| `4`       | A service request or storage operation failed.               |

## Storage and repeated runs

Account and reference paths include the enterprise number and deposit number.
Improved filenames also identify the improvement type.

```text
references/0203201340/2024-123.json
jsons/0203201340/2024-123.json
pdfs/0203201340/2024-123.pdf
improved-jsons/0203201340/2024-123-aa-corrected.json
archives/extracts/2024-08-03/json.zip
pending/extracts/2024-08-03/json/2024-456.json
```

Commands do not read or write `_state/`. Existing contents can remain; they have no effect on operations.
References and pending files provide the metadata needed for validation and repair.

A repeated download reuses each ZIP file that exists at its expected archive path.
It extracts retained ZIP files again, restoring their references and accounts.
Remove a retained ZIP file if you need to fetch that batch again from CBSO.
Without `--keep-zip`, newly fetched ZIP files are not saved.

Before 05:00 Belgian time on the following day, HTTP 404 and empty successful responses appear under `missing_batches`.
After that time, the downloader treats either response as an empty batch for that run.
Empty responses are not saved as ZIP files. A later run requests those batches again.
Non-empty invalid ZIP data remains an error. A retained invalid ZIP must be replaced before extraction can succeed.

For S3, permit object reads and writes beneath the destination prefix.
Permit object deletion for pending-file recovery and consolidated PDF moves.
Permit bucket listing for storage inspection and directory scans.
Storage permission failures remain errors; they never count as absent files.

## Development

See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, tests, coverage, and release instructions.

## Licence

Proprietary. © openthebox.
