Metadata-Version: 2.4
Name: dlt-source-aquabyte
Version: 0.1.0
Summary: dlt source package for the Aquabyte API v3
Author: Havbruksdataforeningen
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Havbruksdataforeningen/dlt-sources/tree/main/packages/dlt-source-aquabyte
Project-URL: Repository, https://github.com/Havbruksdataforeningen/dlt-sources
Project-URL: Changelog, https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/CHANGELOG.md
Project-URL: Issues, https://github.com/Havbruksdataforeningen/dlt-sources/issues
Keywords: dlt,aquabyte,aquaculture,salmon,etl,elt,data-ingestion
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: dlt>=1.2
Dynamic: license-file

# dlt-source-aquabyte

An installable [dlt](https://dlthub.com/) source package that loads aquaculture data from the [Aquabyte API v3](https://api.aquabyte.ai/v3/docs) into any dlt destination: sites, biomass, lice counts, welfare scores, behaviour and environmental readings.

**Records land as the API returns them** — nothing renamed, nothing dropped, no invented child tables. What *is* added is named up front rather than discovered later: `_dlt_valid_from` and `_dlt_valid_to` on the versioned `sites` table. Column names are the API's own field names, in dlt's usual snake_case.

Everything else is mechanics: auth, pagination, envelope unwrapping, incremental cursors, and overridable key and write-disposition defaults. Reshaping belongs in your transform layer, where you can change it without waiting for a release. Its only dependency is dlt itself — destination, orchestrator, secrets manager and log routing stay your choices.

## Install

```bash
uv add dlt-source-aquabyte "dlt[duckdb]"   # any dlt destination works; DuckDB is the one below
```

## Quick start

Put the API base URL and your key in a `.dlt/` directory beside the script you are about to run.

`.dlt/config.toml`:

```toml
[sources.aquabyte]
base_url = "https://api.aquabyte.ai/v3/"
initial_date = "2020-01-01"             # first-run start for the date-based cursors
initial_time = "2020-01-01T00:00:00Z"   # first-run start for the time-based cursors
```

`.dlt/secrets.toml`:

```toml
[sources.aquabyte]
api_key = "your-api-key-here"
```

Then load every resource:

```python
import dlt
from dlt_source_aquabyte import aquabyte_source

pipeline = dlt.pipeline(
    pipeline_name="aquabyte",
    destination="duckdb",     # any dlt destination
    dataset_name="aquabyte_data",
)
print(pipeline.run(aquabyte_source()))
```

The two `initial_*` values are the first-run start for the resources that keep an incremental cursor; a run of only `sites` or `environmental_latest` needs neither, and a cursor resource missing one fails with an error naming it. How far back your data goes differs per endpoint and per account, and setting a start earlier than that costs empty requests, not errors.

## What it loads

| Resource | Endpoint | Load strategy | Key |
|---|---|---|---|
| `sites` | `GET /sites`, `GET /sites/{siteId}` | merge, `scd2` | `id` (merge key) |
| `environmental` | `GET /environmental` | merge | `penId`, `fromTime`, `toTime` |
| `environmental_latest` | `GET /environmental/latest` | replace | — |
| `biomass` | `GET /biomass` | merge | `penId`, `date` |
| `harvest_report` | `GET /biomass/harvestReport` | merge | `penId`, `slaughterStartDate`, `mainReport`, `asOfDate` |
| `lice_count` | `GET /liceCount` | merge | `penId`, `date` |
| `behaviour_swim_speed` | `GET /behaviour/swimSpeed` | merge | `penId`, `fromTime`, `toTime` |
| `behaviour_breathing_index` | `GET /behaviour/breathingIndex` | merge | `penId`, `fromTime` |
| `welfare_scores` | `GET /welfareScores` | merge | `penId`, `date` |

`sites` reads every site, and each site record carries its pens the way the API nests them — every pen, active or not. There is no separate pens table, because the API serves no pens endpoint; [where pen history lives, and what the source does and does not promise about a pen or site that stops being reported](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/REFERENCE.md#pens-live-on-the-site-record). `sites` is **versioned rather than replaced**: a row is retired, never deleted, because a pen leaves `/sites` as soon as it is emptied ([what that means for your queries](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/REFERENCE.md#the-site-registry-is-versioned)). Nested objects land as one JSON column each, and `welfare_scores` is not unpivoted ([why, and how to override it](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/REFERENCE.md#nesting)).

## Configuring a resource

Each resource takes its endpoint's params in snake_case, except the window ones, which the incremental cursor drives:

```python
source = aquabyte_source()
source.sites.bind(site_id="site-001")     # switches to GET /sites/{siteId}
source.biomass.bind(pen_id="pen-abc", bucket_size=250)
pipeline.run(source)
```

- **`pen_id`** defaults to `"all"` — the API's own value for "every pen", in one request. Pass one id to read a single pen.
- **`site_id`** is the one path param, not a query param: binding it moves `sites` to the per-site endpoint, and both write the same table.
- **The window** is the incremental cursor's, not a parameter of its own: a daily run resumes where it left off, and a backfill binds the window on the resource's `incremental_*` argument — see [the reference](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/REFERENCE.md#windows-cursors-and-backfilling).
- **`params`** is on every resource and merged into the query string last: the escape hatch for a query param the API grows later, no release needed. It wins over every named param, `penId` included.

Params can also be set in config, per resource:

```toml
[sources.aquabyte.environmental]
period = "15min"
```

### Two params decide the grain of the data itself

`period` and `bucket_size` change what the API computes for you, not which rows you ask for. Both are worth deciding before the first load: a coarse setting is not wrong, but the detail under it never lands, and getting it later means re-loading that history the backfill way.

| Param | Resource | Values | API default | What it decides |
|---|---|---|---|---|
| `period` | `environmental` | `h`, `D`, `15min` | `D` | Row grain: `h` is 24× the rows of `D`, `15min` is 96× |
| `period` | `behaviour_swim_speed` | `h`, `D` | `D` | As above. `15min` here is a `422` — only `environmental` takes it |
| `bucket_size` | `biomass` | integer grams | `1000` | Bucket width of the nested `weightDist` histogram — no extra rows |

⚠️ **Changing `period` later leaves both grains in the table.** The key is `penId` + `fromTime` + `toTime`, so hourly rows do not merge over the daily ones they cover — both sit there. Pick a period per resource and keep it, or re-load the history behind the change.

`bucket_size` adds no rows. `weightDist` covers only the weights observed, so a pen of smolt returns a couple of buckets and a harvest-size pen at 250 g a few dozen — one JSON column either way ([what the arrays hold](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/specs/README.md#api-quirks-worth-knowing)).

The package emits no log records of its own. dlt logs the window each run asked for and every request it made, on its own `dlt` logger, and routing them is dlt's `[runtime]` settings rather than anything here: [what to set](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/REFERENCE.md#logging), and dlt's own [logging documentation](https://dlthub.com/docs/running-in-production/running#set-the-log-level-and-format).

## Examples

One concept each, readable on GitHub. From a checkout, run one with `python examples/<name>.py`.

| Example | The one concept |
|---|---|
| [`quickstart.py`](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/examples/quickstart.py) | Load every resource into DuckDB |
| [`daily_load.py`](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/examples/daily_load.py) | Re-running resumes from the stored cursor |
| [`backfill.py`](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/examples/backfill.py) | Re-load a window, stored cursor untouched |

## Compatibility

| `dlt-source-aquabyte` | Aquabyte API |
|---|---|
| 0.1.x | v3.1 |

The two numbers are unrelated — the package version is ordinary [SemVer](https://semver.org/) and never mirrors the API's. Built against that version's [`specs/openapi.json`](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/specs/README.md) and run against the live API (last on 2026-08-17). A later backwards-compatible version is expected to work and is not verified here; run the suite first.

## Read next

- [**Reference**](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/REFERENCE.md) — the versioned site registry, nesting, backfilling, column types, and what the source deliberately does not expose.
- [**API quirks worth knowing**](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/specs/README.md#api-quirks-worth-knowing) — where the live API departs from its own OpenAPI document, including which identifiers to join on. Some of them change what a correct query looks like, so read it before your first one.
- [**Changelog**](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/CHANGELOG.md) and [**contributing**](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/CONTRIBUTING.md).

## License

[Apache-2.0](https://github.com/Havbruksdataforeningen/dlt-sources/blob/main/packages/dlt-source-aquabyte/LICENSE). `specs/openapi.json` is Aquabyte's own OpenAPI document, included as the spec this package is built against; it is their material, and the licence does not extend to it.
