Metadata-Version: 2.4
Name: geodaten
Version: 0.4.0
Summary: Download spatial and temporal subsets of global geospatial datasets
Author-email: Sanjeev Bashyal <sanjeev.bashyal@ufz.de>
License-Expression: GPL-3.0-or-later
Project-URL: Homepage, https://github.com/SanjeevBashyal/geodaten
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: GIS
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: netCDF4>=1.6.5
Requires-Dist: numpy>=1.26
Requires-Dist: pyproj>=3.6
Requires-Dist: rasterio>=1.3.9
Requires-Dist: requests>=2.31
Provides-Extra: zenodo
Requires-Dist: remotezip>=0.12; extra == "zenodo"
Provides-Extra: earthdata
Requires-Dist: earthaccess>=0.18; extra == "earthdata"
Provides-Extra: cdse
Requires-Dist: boto3>=1.34; extra == "cdse"
Provides-Extra: arco
Requires-Dist: fsspec[http]>=2024.2; extra == "arco"
Requires-Dist: xarray>=2025.1; extra == "arco"
Requires-Dist: zarr<4,>=3; extra == "arco"
Provides-Extra: tui
Requires-Dist: textual<9,>=1; extra == "tui"
Provides-Extra: gui
Requires-Dist: qtpy>=2.4; extra == "gui"
Provides-Extra: gui-pyside6
Requires-Dist: qtpy>=2.4; extra == "gui-pyside6"
Requires-Dist: PySide6>=6.7; extra == "gui-pyside6"
Provides-Extra: all
Requires-Dist: boto3>=1.34; extra == "all"
Requires-Dist: earthaccess>=0.18; extra == "all"
Requires-Dist: fsspec[http]>=2024.2; extra == "all"
Requires-Dist: qtpy>=2.4; extra == "all"
Requires-Dist: remotezip>=0.12; extra == "all"
Requires-Dist: textual<9,>=1; extra == "all"
Requires-Dist: xarray>=2025.1; extra == "all"
Requires-Dist: zarr<4,>=3; extra == "all"
Provides-Extra: test
Requires-Dist: build>=1; extra == "test"
Requires-Dist: pytest>=7.4; extra == "test"
Requires-Dist: pytest-cov>=4; extra == "test"
Dynamic: license-file

# geodaten

`geodaten` downloads spatial and temporal subsets of global geospatial datasets through one Python API, CLI, Textual TUI, or QtPy GUI.

The package streams source assets, stages one logical slice at a time, and writes only these final formats:

- `tif`: one single-band Cloud Optimized GeoTIFF for each requested variable/time combination, mosaicked over the complete requested extent.
- `netcdf`: exactly one compressed NetCDF4 file for the complete request, written block by block and marked `CF-1.13`.
- `shp`: one ESRI Shapefile layer for each requested variable of a vector dataset, cropped to the extent in EPSG:4326 with its published attribute table.

Each dataset declares the formats it can produce, so a raster source never offers `shp` and a vector source never offers `tif` or `netcdf`. `geodaten info DATASET` lists them, and `--format` defaults to the first one.

Credentials are always supplied by the user through environment variables, an SDK's own configuration, the Python request, or a masked UI prompt. They are never written to filenames, logs, or manifests. The one exception is opt-in and GUI-only: the saved-inputs file described under [TUI and GUI](#tui-and-gui) stores whatever was typed into its credential fields.

## Supported datasets

| ID | Dataset family | Formats | Access |
| --- | --- | --- | --- |
| `soilgrids` | ISRIC SoilGrids | `tif`, `netcdf` | Public |
| `srtm` | NASA SRTMGL1 v3 via OpenTopography | `tif`, `netcdf` | Public object storage |
| `hydrosheds` | HydroSHEDS v1.1 core terrain | `tif`, `netcdf` | Public download |
| `esa_worldcover` | ESA WorldCover 2021 v200 | `tif`, `netcdf` | Public COG tiles |
| `glc_fcs30d` | GLC_FCS30D v2 land cover | `tif`, `netcdf` | Public Zenodo archive |
| `glim` | GLiM v1.1 global lithological map | `shp` | Public download |
| `gimms_lai4g` | GIMMS LAI4g v1.2 | `tif`, `netcdf` | Public Zenodo archive |
| `modis_lai` | MODIS LAI/FPAR C6.1 | `tif`, `netcdf` | NASA Earthdata account |
| `copernicus_lai` | Copernicus Global Land LAI 300 m v2 | `tif`, `netcdf` | Copernicus Data Space account |
| `era5_land` | ECMWF ERA5-Land hourly ARCO | `tif`, `netcdf` | CDS account and accepted licence |
| `chirps` | CHIRPS v3 daily precipitation | `tif`, `netcdf` | Public Zarr mirror |
| `chirts_era5` | CHIRTS-ERA5 daily temperature | `tif`, `netcdf` | Public Zarr mirror |
| `nex_gddp_cmip6` | NASA NEX-GDDP-CMIP6 | `tif`, `netcdf` | Public cloud archive |
| `sentinel2` | Sentinel-2 MSI Level-2A scenes | `tif` | Public Planetary Computer STAC |
| `landsat9` | Landsat 9 Collection 2 Level-2 scenes | `tif` | Public Planetary Computer STAC |

Run `geodaten info DATASET` for its variables, provider options, credential environment names, source, citation, and license.

## Installation

Python 3.12 or newer and the GDAL command-line programs (`gdalinfo`, `gdal_translate`, `gdalbuildvrt`, and `gdalwarp`) are required. Install the core package from this checkout with:

```bash
python -m pip install -e .
```

Install only the adapters and interfaces you use:

```bash
python -m pip install -e ".[zenodo]"       # GLC_FCS30D and GIMMS
python -m pip install -e ".[earthdata]"    # MODIS
python -m pip install -e ".[cdse]"         # Copernicus LAI
python -m pip install -e ".[arco]"         # ERA5-Land, CHIRPS, and CHIRTS Zarr
python -m pip install -e ".[tui]"
python -m pip install -e ".[gui-pyside6]"
```

For every optional adapter and interface, use `python -m pip install -e ".[all,gui-pyside6]"`. Check the current machine with `geodaten doctor`.

## CLI

Discover a dataset before starting a large request:

```bash
geodaten datasets
geodaten info srtm
geodaten plan srtm --bbox 5 47 15 55 -v elevation --format tif
geodaten plan hydrosheds --bbox 5 47 15 55 -v flow_direction --format tif
```

Then run the same request as a download:

```bash
geodaten download srtm \
  --bbox 5 47 15 55 \
  -v elevation \
  --format tif \
  --output ./data
```

A static vector dataset is requested the same way, and its extent crop is written as a shapefile:

```bash
geodaten download glim \
  --bbox 79 26 90 31 \
  -v lithology \
  --output ./data
```

Temporal datasets additionally accept inclusive `--start` and `--end` values. Repeat `-v` for multiple variables and pass adapter-specific settings as `-O KEY=VALUE`; comma-separate values for a multi-value option. `WEST > EAST` represents an antimeridian-crossing bounding box where the selected source supports such a crop; `plan` reports a clear error otherwise.

Credentials are normally read from the environment or the source SDK's user configuration. Add `--prompt-credentials` to enter missing values without echoing secrets:

```bash
geodaten download modis_lai \
  --bbox 5 47 15 55 \
  --start 2024-01-01 --end 2024-01-31 \
  -v lai --prompt-credentials
```

ERA5-Land reads the caller's own CDS token from `CDSAPI_KEY`, `~/.cdsapirc`, a masked prompt, or `credentials={"cds_api_key": "..."}` in Python. Accept the ERA5-Land licence in the CDS before access. Its `hours` option selects UTC hours and `chunking=auto|geo|time` controls the ARCO layout; `auto` uses the time-chunked stores because retrieval is performed month by month. The ARCO precipitation and radiation values are already hourly de-accumulated and are written without another conversion.

CHIRPS and CHIRTS-ERA5 require no account or API key. Planning resolves each public Zarr mirror's current immutable commit, which is reused for the matching download and recorded in the manifest. CHIRPS exposes `pr`; CHIRTS-ERA5 exposes `tasmax` and `tasmin`.

`sentinel2` and `landsat9` write **one GeoTIFF per variable per acquisition date**: every scene covering the extent on that date is reprojected and mosaicked into a single raster, so an extent spanning several MGRS tiles or WRS rows still yields one file rather than fragments. Grouping is by date because Landsat numbers adjacent rows of one pass seconds apart. Both offer an `rgb` variable that stacks red/green/blue into one three-band image with the matching colour interpretation, for loading straight into GIS; values stay native reflectance with the documented scale and offset, so the viewer applies its own stretch. `sentinel2` filters at 20% cloud by default while `landsat9` keeps every scene, since its 16-day revisit makes a discarded acquisition expensive; set `-O max_cloud_cover=` on either to change that.

Sentinel-2 scenes from processing baselines that require different reflectance offsets are not merged into one raster; the request stops with the affected date and baseline identifiers instead of producing radiometrically inconsistent values.

Use `--allow-partial` only when an incomplete source coverage is acceptable. Existing validated outputs are reused unless `--overwrite` is set. A temporal request containing literal `latest` deliberately refreshes discovery and outputs; use explicit dates for reproducible resume. `--workers` is a provider concurrency hint, currently used mainly to size connection pools; a provider may still run safely in sequence.

## Python API

```python
from pathlib import Path

from geodaten import BBox, DownloadRequest, DownloadService, OutputFormat

request = DownloadRequest(
    dataset="srtm",
    bbox=BBox(west=5, south=47, east=15, north=55),
    variables=("elevation",),
    output_format=OutputFormat.TIF,
    output_dir=Path("data"),
)

service = DownloadService()
plan = service.plan(request)
result = service.download(request)
print(result.outputs, result.manifest)
```

For a credentialed adapter, pass only your own values in `credentials={...}`. The accepted keys are listed by `geodaten info DATASET`. The exported `CancellationToken` and `ProgressCallback` can be supplied to `download()` by long-running applications.

## TUI and GUI

```bash
geodaten-tui
geodaten-gui
```

Both interfaces collect the dataset, variables, WGS84 extent, dates, output format/directory, provider options, and user credentials. The CLI and TUI also expose the provider worker hint; the GUI uses the safe package default. The GUI controller uses QtPy and loads the packaged Qt Designer file at `geodaten/gui/download.ui`; the `gui-pyside6` extra supplies the default Qt binding.

The GUI remembers its inputs. Starting a plan or download and closing the window both write `geodaten.json` to `$XDG_CONFIG_HOME/geodaten/` (`~/.config/geodaten/` by default, or wherever `GEODATEN_STATE` points), and the next launch restores the group, dataset, variables, extent, dates, format, output directory, and provider options. Variables, options, and credentials are kept per dataset, so switching datasets and coming back preserves each one's selections.

**This file stores credentials in clear text.** It is written with `0600` permissions and kept in the configuration directory rather than the working directory so it cannot be committed by accident, but anyone who can read the file can read the keys. Leave the credential fields empty to keep using environment variables or an SDK's own configuration instead; empty fields are not written. Delete the file to clear everything it holds.

## Output layout and large requests

Each request is stored below `OUTPUT/DATASET/FINGERPRINT/` with a credential-free `manifest.json`. For explicit reproducible requests, the folder fingerprint combines the request with an immutable source identity when the provider can supply one, so changing source revisions or scene inventories cannot be mixed or mislabeled during resume. A literal `latest` keeps one refreshable root instead of accumulating a new large directory for every source update. Credentials, worker hints, cache locations, and output paths remain excluded. Files are first written to a temporary path and atomically moved into place only after validation.

Raster transfers and conversions use streaming I/O and GDAL virtual rasters. NetCDF data are compressed and appended in bounded raster windows, so memory use is based on a processing block rather than the complete spatial/temporal cube. ERA5-Land, CHIRPS, and CHIRTS-ERA5 fetch one variable and spatial subset per calendar-month block. The NetCDF writer appends that whole block in one pass, while TIFF output expands its bands into the required time slices; exceptionally large extents are split below a 512 MiB decoded-data ceiling. `--cache` selects a staging location; SRTM, HydroSHEDS, and MODIS reuse validated source tiles or assets, while request scratch data and incomplete NetCDF files are deliberately removed. A retry reuses an exact, validated complete TIFF set without restaging provider data. Without `--overwrite`, a partial TIFF retry may repeat provider staging but never overwrites already validated completed slices. Source services may still impose request-size, rate, date-range, account, or beta-service limits, so use `plan` before large jobs.

A `shp` request writes the whole `.shp`/`.shx`/`.dbf`/`.prj`/`.cpg` set for each layer, all of which are listed in the manifest; the set is built beside the target and moved in together, and a retry reuses a validated complete set without restaging. Vector sources that ship as one large archive are cropped locally, so the source is cached below `--cache` and reused by later extents; `geodaten info glim` documents the options that relocate, reuse, or discard it.

## Development

```bash
python -m pytest -q
python -m build
```

Tests mock remote catalogs and authentication. Live downloads are intentionally not required, because users must provide and authorize their own service accounts.
