Metadata-Version: 2.4
Name: overture-core
Version: 1.9.0
Summary: Shared Overture business logic — portable, framework-agnostic job classes.
Author: Overture Maps Foundation
License-Expression: MIT
Project-URL: Homepage, https://github.com/OvertureMaps/overture-core
Project-URL: Source, https://github.com/OvertureMaps/overture-core/tree/main/packages/overture_core
Project-URL: Issues, https://github.com/OvertureMaps/overture-core/issues
Keywords: overture,overturemaps,stac,gis
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: overture-serverless>=1.0.0
Requires-Dist: overture-dataset-configs>=1.0.0
Requires-Dist: pystac>=1.8.4
Requires-Dist: boto3>=1.28
Requires-Dist: pydantic>=2.0
Requires-Dist: PyGithub>=2.3
Requires-Dist: requests>=2.31
Requires-Dist: packaging>=23.0
Requires-Dist: rich>=13.0
Requires-Dist: sqlglot>=30.15.0
Provides-Extra: stac
Requires-Dist: overture-stac; extra == "stac"
Provides-Extra: maproulette
Requires-Dist: maproulette>=1.8.1; extra == "maproulette"
Requires-Dist: geojson>=3.3.0; extra == "maproulette"
Requires-Dist: shapely>=2.1.1; extra == "maproulette"
Provides-Extra: sql-spark
Requires-Dist: pyspark>=3.5; extra == "sql-spark"
Provides-Extra: sql-trino
Requires-Dist: trino>=0.328; extra == "sql-trino"
Provides-Extra: azure
Requires-Dist: azure-storage-blob>=12.0; extra == "azure"
Provides-Extra: dev
Requires-Dist: pytest>=9.1.1; extra == "dev"
Requires-Dist: pytest-cov>=7.0.0; extra == "dev"
Requires-Dist: overture-core[azure,maproulette,stac]; extra == "dev"
Requires-Dist: jinja2>=3.0; extra == "dev"
Requires-Dist: duckdb>=1.0.0; extra == "dev"
Requires-Dist: databricks-sdk>=0.20; extra == "dev"
Requires-Dist: moto[s3]>=5.1.5; extra == "dev"
Dynamic: license-file

# overture_core

[![PyPI](https://img.shields.io/pypi/v/overture-core?color=4051CC&labelColor=001A39)](https://pypi.org/project/overture-core/)
[![Python versions](https://img.shields.io/pypi/pyversions/overture-core?color=4051CC&labelColor=001A39)](https://pypi.org/project/overture-core/)
[![uv](https://img.shields.io/badge/managed%20with-uv-0EC1BD?labelColor=001A39)](https://docs.astral.sh/uv/)
[![License: MIT](https://img.shields.io/badge/license-MIT-05A5AF?labelColor=001A39)](LICENSE)

Shared, framework-agnostic business logic for the Overture data platform: helpers and portable job classes built on [`overture-serverless`](../overture_serverless).

## Install

```sh
uv add overture-core
uv add 'overture-core[stac]'  # with an extra
```

Heavier dependencies are optional extras, so consumers on constrained runtimes (for example MWAA's pinned `pyarrow`) stay installable. Quote the extra in shells like zsh that glob brackets. For local development, run `uv sync --extra dev`, then `uv run pytest`.

| Extra | Needed for |
| --- | --- |
| `stac` | `stac.job.PublishStac`, `stac.catalog.build_release_catalog` (`stac.latest_release_job.LatestRelease` works without it) |
| `azure` | `cloud.azure` |
| `maproulette` | `apis.maproulette.MapRouletteClient` |
| `sql-spark`, `sql-trino` | The SQL-engine test lanes only |

`cloud.aws.object.read_parquet_prefix` needs a `pyarrow` of your choosing on the path. Optional provider SDKs stay out of the base dependencies: add an extra and import the SDK lazily inside the helper.

## Modules

Framework-agnostic helpers, each importable on its own without pulling in the job classes below.

<!--
When adding or editing a row: describe the module's scope/boundary (what kind of
helper belongs here), not an inventory of its current classes/functions. This
table should stay accurate as the module grows without needing an edit every
time something is added to it. Put new code in the module whose scope already
covers it; only add a new row/module when nothing existing fits.
-->

| Module | Scope |
| --- | --- |
| `uuids` | Pure Python, Spark-free UUID generation for any UUID version, current or future RFC. |
| `uuids_sql` | SQL-string equivalents of `uuids`, for computing UUID columns server-side in Spark or Trino instead of round-tripping through a Python UDF. |
| `spark_sql` | Spark SQL table-creation strings from caller-owned column definitions, rendered with SQLGlot without importing Spark or Arrow. |
| `strings_sql` | SQL expressions that make string values satisfy the Overture schema's string constraints (e.g. `StrippedString`), computed in-query instead of through a Python UDF. |
| `versioning` | Deriving a consistent dated version string for sources that don't publish explicit version metadata. |
| `iceberg` | Shared Iceberg + Sedona catalog configuration, reusable across platforms/engines. |
| [`cloud`](overture_core/cloud/README.md) | Cloud helpers: provider-agnostic code plus one subpackage per vendor (`aws`, `azure`, `databricks`). |
| `pypi` | Provider-agnostic PyPI package download/publish helpers, usable against any index. |
| `apis` | Clients for third-party HTTP APIs, one module per vendor (`apis.tomtom`, `apis.maproulette`, and `apis.github`, which mints GitHub App installation tokens from a PEM in Secrets Manager; see the [reference doc](https://github.com/OvertureMaps/omf-github-terraform/blob/main/docs/github-app-tokens.md)). Anything that's not a cloud platform but still an external service belongs here. |
| `urls` | Generic URL string utilities not tied to any specific service or cloud provider. |
| `bbox` | Validating and safely rendering an optional bounding-box string param ("min_lon,min_lat,max_lon,max_lat", `''` = full planet), shared by any entry point that accepts one. |
| `stac` | STAC catalog reads/writes and the jobs that publish them (see [Jobs](#jobs)). |
| `data` | Describing a data location and its sync configuration, independent of the mechanism used to move it. |
| `docs` | Automating docs-repo updates for a release, via a GitHub App. |
| `artifacts` | Release artifact types (metadata, license, attribution) and the tree-search/S3 I/O to read and write them. |
| [`dataset`](overture_core/dataset/README.md) | Parsing, validating and rendering provider/resource dataset configs and the license policy, plus the `overture-datasets` CLI. |
| `bundle` | The pipeline data bundle model: an S3 directory of `data/` plus `metadata.json` and a `success` marker under Hive-style partitions, with its path layout, lifecycle and status, partition resolution and retention/cleanup. Storage goes through a pluggable `BundleStore` (S3 by default; an in-memory store for tests and offline tools). Orchestrator concerns (default buckets, namespacing, the metadata writer, catalogue table drops) are passed in by the caller. |

## Jobs

- **`stac.job.PublishStac`**: publishes the Overture STAC catalog. Two modes selected by the presence of the `release` param:
  - *single-release* (`release` + `source_path` + `scratch_bucket`): publish just that release, mirror scoped to `stac/{release}/`. Schema comes from the RC bundle's `metadata.json`.
  - *walk* (no `release`, requires `scratch_bucket`): rebuild the catalog for every release currently in the public bucket, mirror the whole `stac/` prefix with delete-orphans. Schema per release comes from the existing STAC (`schema:version` in each `catalog.json`); on cache miss the job self-heals by reading schema from the released RC bundle in `scratch_bucket` and raises if it can't recover.
- **`stac.latest_release_job.LatestRelease`**: reads the STAC root catalog (`root_href`, defaults to the prod root) and resolves the newest release ID. Read-only, HTTPS-only, no AWS credentials or S3 access needed. Stashes the result on `self.latest_release` for a caller that instantiates the job directly instead of going through `run()`.

## Future work

- **Bundle-owned schema accessor** — move `read_schema_version_from_rc_bundle` onto `bundle.bundle.ReleaseCandidateBundle` as `resolved_schema_version()`; single-release mode's schema read shrinks to one line.
- **Upstream `build_release_catalog`** — file a PR against [OvertureMaps/stac](https://github.com/OvertureMaps/stac) exposing this as a public API (their CLI already does the exact call sequence). When it lands, our wrapper collapses.
