Metadata-Version: 2.4
Name: geoai-datacubes
Version: 1.0.0
Summary: AI-ready multi-mission Earth-observation data cubes for machine-learning and deep-learning workflows.
Author-email: Joachim Moortgat <moortgat.1@osu.edu>
Maintainer-email: Joachim Moortgat <moortgat.1@osu.edu>
License: MIT License
        
        Copyright (c) 2025-2026 The Ohio State University (BuckAI Observatory)
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/buckai-observatory/geoai-datacubes
Project-URL: Repository, https://github.com/buckai-observatory/geoai-datacubes
Project-URL: Documentation, https://github.com/buckai-observatory/geoai-datacubes#readme
Project-URL: Issues, https://github.com/buckai-observatory/geoai-datacubes/issues
Project-URL: Paper, https://github.com/buckai-observatory/geoai-datacubes/blob/main/paper.md
Keywords: remote sensing,earth observation,satellite imagery,data cubes,machine learning,geospatial,STAC,Sentinel,Landsat,NAIP,MODIS,ALOS PALSAR,Copernicus DEM,HLS,PlanetScope
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Topic :: Scientific/Engineering :: Image Processing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=2.0
Requires-Dist: matplotlib>=3.6
Requires-Dist: tqdm>=4.60
Requires-Dist: requests>=2.28
Requires-Dist: pillow>=9.0
Requires-Dist: rasterio>=1.3
Requires-Dist: pystac>=1.10
Requires-Dist: pystac-client>=0.7
Requires-Dist: planetary-computer>=1.0
Requires-Dist: zarr<3,>=2.16
Requires-Dist: lmdb>=1.4
Provides-Extra: ml
Requires-Dist: torch>=2.0; extra == "ml"
Requires-Dist: torchvision>=0.15; extra == "ml"
Requires-Dist: scikit-image>=0.20; extra == "ml"
Requires-Dist: scikit-learn>=1.3; extra == "ml"
Requires-Dist: xgboost>=2.0; extra == "ml"
Requires-Dist: joblib>=1.3; extra == "ml"
Requires-Dist: threadpoolctl>=3.7; extra == "ml"
Requires-Dist: ultralytics>=8.4; extra == "ml"
Requires-Dist: transformers>=4.30; extra == "ml"
Requires-Dist: huggingface_hub>=0.20; extra == "ml"
Provides-Extra: geoai
Requires-Dist: geoai-py>=0.5; extra == "geoai"
Requires-Dist: omniwatermask>=0.4; extra == "geoai"
Provides-Extra: notebooks
Requires-Dist: jupyterlab>=4.0; extra == "notebooks"
Requires-Dist: ipywidgets>=8.0; extra == "notebooks"
Requires-Dist: seaborn>=0.12; extra == "notebooks"
Requires-Dist: geopandas>=0.14; extra == "notebooks"
Requires-Dist: contextily>=1.4; extra == "notebooks"
Provides-Extra: planet
Requires-Dist: python-dotenv>=1.0; extra == "planet"
Requires-Dist: sentinelhub>=3.10; extra == "planet"
Provides-Extra: dev
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pre-commit>=3.5; extra == "dev"
Requires-Dist: quartodoc>=0.10; extra == "dev"
Requires-Dist: griffe<2; extra == "dev"
Requires-Dist: contextily>=1.4; extra == "dev"
Provides-Extra: all
Requires-Dist: geoai-datacubes[geoai,ml,notebooks,planet]; extra == "all"
Dynamic: license-file

# geoai-datacubes

**Turn raw satellite imagery into AI-ready data cubes — pick a place, pick a time, get clean training data.**

[![Docs](https://img.shields.io/badge/docs-Quarto%20site-2563EB.svg?style=flat-square&logo=quarto&logoColor=white)](https://buckai-observatory.github.io/geoai-datacubes/)
[![License: MIT](https://img.shields.io/badge/License-MIT-BA0C2F.svg?style=flat-square)](LICENSE)
[![Python 3.11](https://img.shields.io/badge/Python-3.11-3776AB.svg?style=flat-square&logo=python&logoColor=white)](https://www.python.org/)
[![tests](https://github.com/buckai-observatory/geoai-datacubes/actions/workflows/tests.yml/badge.svg)](https://github.com/buckai-observatory/geoai-datacubes/actions/workflows/tests.yml)
[![notebooks](https://github.com/buckai-observatory/geoai-datacubes/actions/workflows/notebooks.yml/badge.svg)](https://github.com/buckai-observatory/geoai-datacubes/actions/workflows/notebooks.yml)
[![PyPI](https://img.shields.io/pypi/v/geoai-datacubes.svg?style=flat-square&color=3775A9&logo=pypi&logoColor=white)](https://pypi.org/project/geoai-datacubes/)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.20829119.svg)](https://doi.org/10.5281/zenodo.20829119)
[![status](https://joss.theoj.org/papers/41c1ac5fdbfc1a4a4ee3b79cd8e4ee00/status.svg)](https://joss.theoj.org/papers/41c1ac5fdbfc1a4a4ee3b79cd8e4ee00)
[![BuckAI Observatory](https://img.shields.io/badge/BuckAI-Observatory-BA0C2F.svg?style=flat-square)](https://ai-observatory.osu.edu)

**Full documentation:** <https://buckai-observatory.github.io/geoai-datacubes/>

> **New here?** Two ready-to-run paths get you started directly in Colab — no install, no credentials needed:
>
> - **Data acquisition & pre-processing** — the grand-tour notebook walks through every pipeline feature on the data side: every mission (data source), AOI (area of interest) format, cloud masking, NaN handling, tiling, multi-mission fusion, and SLURM submission.
>   <a href="https://colab.research.google.com/github/buckai-observatory/geoai-datacubes/blob/main/notebooks/00_geoai_datacubes_tour.ipynb" target="_blank" rel="noopener noreferrer"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open in Colab"/></a>
>
> - **ML / DL on a data cube** — the classification notebook trains and compares Logistic Regression, Random Forest, XGBoost, and a U-Net on a multi-modal fused cube across three Ohio cities, on any ESA WorldCover class you pick at the top.
>   <a href="https://colab.research.google.com/github/buckai-observatory/geoai-datacubes/blob/main/notebooks/01_classification.ipynb" target="_blank" rel="noopener noreferrer"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open in Colab"/></a>
>
> Also Colab-ready: an `opengeos/geoai` integration notebook ([`03_with_opengeos_geoai.ipynb`](notebooks/03_with_opengeos_geoai.ipynb)). See [Try the notebooks](#try-the-notebooks) for all three. A fourth notebook ([`02_building_detection.ipynb`](notebooks/02_building_detection.ipynb)) is bundled as an in-development scaffold for object detection on NAIP — kept in the repo but not part of the reviewed examples.

---

## What is this?

`geoai-datacubes` is an open-source tool developed at the [**BuckAI Observatory**](https://ai-observatory.osu.edu) at **The Ohio State University**, intended for the worldwide Earth-observation and AI research communities. It gives you ready-to-use pipelines that **download satellite imagery** for any region and time you choose, then **pre-process it into AI-ready "data cubes"** — cloud-filtered, normalised, tiled, augmented, and split into training / validation / test sets that you can feed straight into a machine-learning model.

The repo is designed to lower the entry barrier into Earth-observation ML for anyone — researchers, postdocs, graduate students, undergraduate research assistants, or industry / non-profit practitioners. If you have never touched an HPC or a satellite API before, you can still follow the steps below and produce a usable dataset without first having to stitch together half a dozen vendor SDKs. Tooling the BuckAI Observatory builds is open-sourced under permissive licences so that this kind of accessible AI infrastructure can benefit the broader research community, not just OSU's.

> ### What is an "AI-ready data cube"?
> Raw satellite scenes are huge, messy files: different formats, cloudy pixels, inconsistent value ranges, and far too big to fit on a GPU. A **data cube** is that imagery cleaned up and reshaped into a tidy, stacked array — think of a deck of aligned image layers (the spectral bands) cut into small, equal-sized tiles. Because every tile is the same size, cloud-free, normalised to a common range, and pre-split into train/validation/test groups, you can load it directly into PyTorch or TensorFlow and start training. The cube does the boring, error-prone data prep so you can focus on the science. A *fused* cube is the same thing with layers from more than one mission (say, Sentinel-2 optical plus Sentinel-1 radar plus elevation) resampled onto one grid.
>
> It is called a cube because the stack has three axes: rows x columns x bands (a time axis can be added for time series). Every pixel location carries the same list of values, one per layer.

> ### Key terms
> - **AOI (area of interest)** is the geographic box you want data for. It can be given as a bounding box, a shapefile, a centre point plus a side length, or a point inside a Sentinel-2 tile.
> - **Mission** is one satellite or data product the package can download, for example Sentinel-2 L2A, Landsat, NAIP or ESA WorldCover. Each has its own bands and resolution. Some entries, such as land-cover maps, are derived products rather than a single satellite.
> - **Band** is one layer of an image, such as red or near-infrared reflectance.
> - **Tile (chip)** is a small, equal-sized image crop used as one training sample. **Stride** is how far the tiling window moves between tiles; a stride equal to the tile size means no overlap.
> - **Provider** is the web service that hosts the data (Earth Search, Planetary Computer, and others).

---

## What it does

- **26 missions** in a unified registry — Sentinel-1 / 2, Landsat, NAIP, PlanetScope, MODIS, HLS, ALOS, Copernicus DEM, 3DEP, ESA WorldCover, JRC-GSW, USDA-CDL, Hansen-GFC, and more. Full per-mission band / resolution / value-range reference in [`docs/data_layers.md`](docs/data_layers.md).
- **Four interchangeable providers** (Earth Search, Microsoft Planetary Computer, Planet Orders, Sentinel Hub) plus a `direct_http` path for non-STAC datasets, unified behind one dispatcher. The default `PROVIDER = "auto"` routes each mission to its best free host. See [`docs/providers.md`](docs/providers.md) for the trade-offs and routing table.
- **Declarative per-band metadata.** Every band carries a `kind` and a normalisation recipe; `apply_band_norm` + `get_band_norm` produce ML-ready features in one call without hiding the scale factors.
- **Multi-mission fusion** onto a common UTM grid via `fuse_response_tiffs(...)` with mission-prefixed band descriptions so provenance survives. See [`docs/fusion.md`](docs/fusion.md).
- **Robust pre-processing** — smear-protected reprojection, polygon-aware Sentinel-1 same-day mosaicking, cloud / shadow / haze masking via mission-aware QA bands (Sentinel-2 SCL, Landsat BQA, PlanetScope UDM2).
- **Spatially-aware train / val / test splits** (`random`, `block`, `stripes`, `regions`) selectable as a single argument, closing the leakage hole that random tile splits leave open on autocorrelated imagery.
- **On-the-fly PyTorch tile sampling** via `LazyTileDataset` — sweep tile size, stride, augmentation, NaN handling, and split assignment at training time without materialising tiles to disk.

---

## Two ways to use `geoai-datacubes`

The package supports two typical workflows; you can pick either, and you can move between them freely.

- **Notebook path** — the on-ramp for anyone new to satellite data or ML/DL. The bundled notebooks (`notebooks/00_geoai_datacubes_tour.ipynb` and its siblings, above) are documented, interactive, and easy to adapt. Open one in Colab, swap the AOI and time range to your area of interest, and iterate. This is the recommended starting point for students and researchers coming from adjacent fields.
- **Script / library path** — for users comfortable with remote sensing or ML who want to compose the pipeline into their own code. Import the package as a library, chain the fetch / fuse / tile helpers into your own workflow, embed it in a bigger pipeline, or run headless on an HPC cluster. Notebooks can also be stripped down to a plain Python script for SLURM runs — [`smoke-tests/perf_bench.slurm`](smoke-tests/perf_bench.slurm) is one worked example of that pattern.

Both paths share the same underlying package — the notebooks are just curated call-sites of the same public API a script user would call directly.

---

## Install

The recommended path is a single `mamba` (or `conda`) command from conda-forge plus one `pip install`. `mamba` is a drop-in `conda` replacement that solves environments much faster; if you already have `conda` installed, substitute `conda` for `mamba` in the commands below and everything works the same.

```bash
mamba create -y -n geoai-cubes -c conda-forge \
    python=3.11 \
    geoai-py leafmap torchgeo omniwatermask \
    rasterio gdal libgdal-jp2openjpeg pyproj shapely \
    pystac pystac-client planetary-computer \
    "pytorch>=2.0" "torchvision>=0.15" \
    zarr lmdb scikit-image pillow \
    matplotlib numpy pandas tqdm requests \
    scikit-learn xgboost ultralytics transformers \
    jupyterlab ipywidgets seaborn geopandas contextily
mamba activate geoai-cubes
pip install geoai-datacubes              # or: pip install -e . from a clone
```

`libgdal-jp2openjpeg` is the JPEG-2000 GDAL plugin — needed for Earth Search's Sentinel-2 L1C assets, which are hosted as `.jp2`. Every other mission is COG-native and does not need it; if you skip it, all missions except S2 L1C still work.

For Docker, pip-only, slimmer installs via `[ml]` / `[geoai]` / `[notebooks]` / `[planet]` extras, and the first-run recipe, see **[`docs/install.md`](docs/install.md)**. PlanetScope and Sentinel Hub credential setup is in **[`docs/credentials.md`](docs/credentials.md)**.

---

## Try the notebooks

The repo ships with **three complementary reviewed notebooks** in `notebooks/` plus one in-development scaffold. See [`notebooks/README.md`](notebooks/README.md) for a detailed walkthrough of each.

### 1. The grand tour (start here if you are new)

[`notebooks/00_geoai_datacubes_tour.ipynb`](notebooks/00_geoai_datacubes_tour.ipynb)
<a href="https://colab.research.google.com/github/buckai-observatory/geoai-datacubes/blob/main/notebooks/00_geoai_datacubes_tour.ipynb" target="_blank" rel="noopener noreferrer"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open in Colab"/></a>

A pedagogical walkthrough of every feature on the *data* side of the pipeline — AOI formats, free missions, cloud masking, NaN handling, tiling with/without overlap, the four split strategies, multi-mission fusion, reading metadata back from a tile, augmentation, and submitting jobs to SLURM. Click the Colab badge to launch in your browser — the first cell clones the repo and installs everything, no local Python required.

### 2. Land-cover classification end-to-end (ML / DL)

[`notebooks/01_classification.ipynb`](notebooks/01_classification.ipynb)
<a href="https://colab.research.google.com/github/buckai-observatory/geoai-datacubes/blob/main/notebooks/01_classification.ipynb" target="_blank" rel="noopener noreferrer"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open in Colab"/></a>

A complete machine-learning workflow that picks up where the tour leaves off. Fetches and fuses Sentinel-2 + Sentinel-1 + Copernicus DEM + ESA WorldCover for **Columbus, Cincinnati, and Cleveland**; trains and compares four standard classifiers (Logistic Regression, Random Forest, XGBoost, and a lightweight U-Net) on a binary target — the chosen ESA WorldCover class vs everything else. The class is a user input at the top, with a per-class quality table. Includes conditional NDWI / NDVI baselines, KMeans unsupervised bonus, multi-modal fusion comparison, threshold tuning on validation, and a collapsible binary-classification metrics explainer. **Cached weights for the default water target ship in the repo** so a fresh Colab launch lands in ~5 min instead of ~30.

### 3. Integration with `opengeos/geoai` (segmentation hand-off)

[`notebooks/03_with_opengeos_geoai.ipynb`](notebooks/03_with_opengeos_geoai.ipynb)
<a href="https://colab.research.google.com/github/buckai-observatory/geoai-datacubes/blob/main/notebooks/03_with_opengeos_geoai.ipynb" target="_blank" rel="noopener noreferrer"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open in Colab"/></a>

A worked example of composing `geoai-datacubes` (data-prep front-end) with `opengeos/geoai` (Wu, 2026, JOSS 11(118):9605 — modelling back-end). Builds fused multi-mission cubes for three Ohio cities and hands them off to `geoai-py` in two patterns: pretrained inference via `geoai.segment_water` on a NAIP scene, and custom training via `select_bands` + `geoai.train_segmentation_landcover` + `geoai.semantic_segmentation`. Trains on Cleveland + Cincinnati and **holds Columbus out entirely** — in-distribution F1 reaches ~0.95 while OOD F1 collapses to ~0.05, an honest illustration of the standard remote-sensing-ML reality of training on a handful of AOIs.

### ⚠️ Work in progress: object detection on NAIP

[`notebooks/02_building_detection.ipynb`](notebooks/02_building_detection.ipynb)
<a href="https://colab.research.google.com/github/buckai-observatory/geoai-datacubes/blob/main/notebooks/02_building_detection.ipynb" target="_blank" rel="noopener noreferrer"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open in Colab"/></a>

**In development — not part of the reviewed examples.** This notebook is a scaffold for the first object-detection workflow in the series: YOLOv8 fine-tuned on NAIP 1 m aerial imagery + Microsoft US Building Footprints, with OWLv2 zero-shot and a community-trained YOLO as pretrained baselines. The pipeline runs end-to-end but the trained detector does not converge reliably enough for us to include it as a reviewed example alongside notebooks 00 / 01 / 03. It is kept in the repo as a starting point for object-detection experimentation; contributions and issue reports are welcome.

---

## Documentation

The detailed reference docs live under [`docs/`](docs/):

- **[`install.md`](docs/install.md)** — clone, env, full install matrix, first run, AOI formats.
- **[`providers.md`](docs/providers.md)** — Earth Search / Planetary Computer / Sentinel Hub / Planet / `direct_http` trade-offs, capability matrix, `auto` routing.
- **[`fusion.md`](docs/fusion.md)** — multi-mission fusion onto a common UTM grid.
- **[`data_layers.md`](docs/data_layers.md)** — per-mission band / resolution / value-range / normalisation recipe reference.
- **[`configuration.md`](docs/configuration.md)** — `main.py` parameter knobs + pipeline-script reference.
- **[`credentials.md`](docs/credentials.md)** — Sentinel Hub + Planet credential setup.
- **[`adding_a_mission.md`](docs/adding_a_mission.md)** — how to wire a new mission profile into `MISSION_PROFILES`.
- **[`project_structure.md`](docs/project_structure.md)** — directory tree + per-module summary.
- **[`HPC_QUICKSTART.md`](docs/HPC_QUICKSTART.md)** — short cluster-side install + SLURM training recipe.

For contributing, reporting issues, or getting support, see **[`CONTRIBUTING.md`](CONTRIBUTING.md)** (which also covers the Contributor Covenant code of conduct).

---

## License

Released under the **MIT License** — see [`LICENSE`](LICENSE) for the full text. Copyright © **The Ohio State University / BuckAI Observatory**.

---

## Acknowledgements & contact

Built and maintained by the [**BuckAI Observatory**](https://ai-observatory.osu.edu) at The Ohio State University.

- Website: <https://ai-observatory.osu.edu>
- More tools & tutorials: BuckAI Observatory [resources page](https://buckai-observatory.org/resources.html)

This project was developed over approximately one year by **Jain, Bhavika**; **Radhakrishnan, Aswathnarayan**; **Chowdhury, Satyaki Roy**; **Hsu, Hsiao Jou**; and **Moortgat, Joachim** (principal investigator) at OSU. See [`CHANGELOG.md`](CHANGELOG.md) for the full timeline and [`CONTRIBUTORS.md`](CONTRIBUTORS.md) for a per-area breakdown of who contributed what. Code development since May 2026 was substantially accelerated by [Claude Code](https://claude.com/claude-code), Anthropic's AI coding assistant, used under continuous human direction and review.

If you use `geoai-datacubes` in your research, please cite via the GitHub "Cite this repository" dropdown (sourced from [`CITATION.cff`](CITATION.cff)) or the JOSS paper draft at [`paper.md`](paper.md).

We welcome collaboration. If this tool helps your research, we'd love to hear about it.
