Metadata-Version: 2.4
Name: caeli-data
Version: 0.6.0
Summary: Python client of the Caeli data repository API (data.caeli.space): load tables, publish rasters, read the catalogue
Author-email: "Caeli B.V." <tim@caeli.nl>
License-Expression: LicenseRef-Proprietary
Project-URL: Homepage, https://data.caeli.space
Project-URL: Catalogue, https://catalog.caeli.space
Keywords: caeli,air quality,STAC,Iceberg,data repository
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: httpx>=0.27
Requires-Dist: pyarrow>=14
Provides-Extra: aws
Requires-Dist: boto3>=1.26; extra == "aws"
Provides-Extra: pandas
Requires-Dist: pandas>=2.0; extra == "pandas"

# caeli-data

The Python client of the Caeli data repository API (https://data.caeli.space). One door for every producer and reader
of datasets: the inventory platform, pipelines and colleagues all use this library; nothing writes to the tables or the
buckets directly.

    pip install caeli-data            # caeli-data[aws] also brings boto3, for Repository.from_secret

    from caeli_data import Repository, TableRejected, IngestFailed

    repo = Repository()               # CAELI_API_URL (default https://data.caeli.space) and CAELI_API_KEY
    repo.whoami()                     # {"name": "tim", "role": "admin", ...}

    # a pandas DataFrame with utc_time, latitude, longitude and the value columns: the platform derives the rest
    # (pt_region, pt_tile, row_hash; the coordinates rounded to 5 decimals), splits per region and month, loads, waits
    jobs = repo.load_dataframe(df, "cdse-sentinel-5p-l2-no2", software={"my-pipeline": "1.0"}, source_ref="download 4711")
    print(jobs[0]["message"])         # "200000 rows appended in cdse_sentinel_5p_l2_no2, ... item N_EU__2026-09__v1 now ..."

    # a source with its own row identity (a station id, an orbit with scanline and pixel) names it
    jobs = repo.load_dataframe(df, "openaq-stations-no2", row_key="location_id")

    # a prepared Parquet file with the six standard columns already in it, any regions and months
    jobs = repo.load_file("prepared.parquet", "cdse-sentinel-5p-l2-no2", software={"my-pipeline": "1.0"})

A complete month delivered again is a new version of that month: `mode="replace", reason="re-import of 2026-09"`.
A service takes its key from Secrets Manager: `Repository.from_secret("caeli/data-platform/api-key-inventory-platform")`.

The rules a table must meet are in Caeli_data_repository_conventions (section 6); the library checks them before
uploading and says which rule is broken (`TableRejected`; for an extra column the message carries the schema change to
make). `IngestFailed` means the platform failed; the same data sent again is the same job, never a double load.
Python 3.9 and later.

A raster (a GeoTIFF on a registered grid) is one item of a raster dataset, area and period:

    from caeli_data import PublishRejected

    job = repo.publish_raster("nh3_2024-05.tif", "caeli-iasi-nh3-monthly", "NL", "2024-05", software={"caeli-maps": "0.1.0"})
    print(job["message"])             # "NL__2024-05__v1 published as s3://caeli-file-store/datasets/.../v1/... (0.1 MB, 60x38 px ...)"
    job = repo.publish_raster("nh3_2024-05.tif", "caeli-iasi-nh3-monthly", "NL", "2024-05", mode="revise", reason="recipe v2")

The worker refuses a raster that is not on the area's grid, has no CRS or nodata, or is not a Cloud-Optimized GeoTIFF;
with rasterio and rio-cogeo installed the library checks the same rules before uploading (`PublishRejected`) and
`convert=True` makes the COG for you. `publish_file` does the same for any file or model; `register_grid` (admin) and
`grids()` manage the grids.

A collection is one recipe: register a raster dataset with the configuration that makes it
(`register_dataset(..., recipe=recipe_dict)`, which records it as `caeli:recipe` with its `fingerprint()`), send the
fingerprint with every publication (`publish_raster(..., recipe_fingerprint=fingerprint(recipe_dict))`), and the
platform refuses a map made with another recipe — that is a different dataset.

Reading goes through the same door (0.6.0, API 0.9.x). One call gives the rows of a table dataset as one file or
one DataFrame, at any size:

    repo.download("cdse-sentinel-5p-l2-no2", "no2_nl_2026-06.csv", area="NL", period="2026-06",
                  columns=["product__nitrogendioxide_tropospheric_column", "product__qa_value"],
                  filters=[("product__qa_value", ">=", 0.75)])
    df = repo.read_table("ads-cams-forecast", regions=["N_EU"], start="2026-06-01", end="2026-06-08", columns=["pm2p5", "lead_hours"])
    repo.plan("cdse-sentinel-5p-l2-no2", regions=["N_EU"], period="2026-06", columns=["product__nitrogendioxide_tropospheric_column"])

The API decides beforehand, from the tables' metadata, whether a request is small enough; a request that is too large
comes back as consecutive windows in time, and `download` and `read_table` fetch them one after another and join
them into one file (`split=False` refuses instead, with `ReadTooLarge` carrying the plan). Every download is pinned to
one table snapshot (`version=` takes an item key or `snapshot:<id>`), filters are typed against the schema
(`("qa_value", ">=", 0.75)`, `("orbit", "in", [1, 2])`, `("no2", "notnull")`), and the three index columns
`idx_utc`, `idx_lat`, `idx_lon` are always part of a download. Without Python: `caeli-data plan ...`,
`caeli-data download ... --to file.csv`, `caeli-data reads`.
