Metadata-Version: 2.4
Name: geoextract-gee
Version: 0.2.1
Summary: Google Earth Engine backend for geoprepare (geoextract)
Author: Bruno Cocitto
License-Expression: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: GIS
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: geoprepare>=0.6.331
Requires-Dist: earthengine-api
Requires-Dist: google-auth
Requires-Dist: google-cloud-storage
Requires-Dist: geopandas
Requires-Dist: pandas
Requires-Dist: shapely
Requires-Dist: arrow
Requires-Dist: tenacity
Requires-Dist: tqdm
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# geoextract_gee

Google Earth Engine backend extension for `geoprepare` (geoextract). 

This package provides an optional Google Earth Engine (GEE) execution path for Earth Observation (EO) data extraction. It orchestrates task combinations, processes them using GEE, polls the tasks, and downloads the resulting CSVs from Google Cloud Storage (GCS).

## Prerequisites

- **Google Earth Engine Account**: Ensure you have access and a project set up.
- **Google Cloud Storage**: A GCS bucket is required to export the results before downloading.
- **Environment**: It is recommended to refer to [installer](https://github.com/ritviksahajpal/installer) for setting up the environment required for `geocif` and `geoprepare`.

## Installation

```bash
pip install geoextract-gee
```

This also installs `geoprepare`, which it extends. geoprepare switches to this backend when `use_gee = True` (see Configuration).

## Authentication

Set `key` in `[GEE]` to a service account JSON key, as a path relative to geoprepare's metadata folder. Without it (or with `key = none`), the standard Earth Engine sign-in (`ee.Authenticate()`) runs the first time; follow its instructions.

## Configuration

Turn the backend on in `[DEFAULT]` of your geoprepare config, and put its settings in a `[GEE]` section:

```ini
[DEFAULT]
use_gee = True
redo = False

[GEE]
project = your-gcp-project-name
bucket = your-gcs-bucket-name

# Optional
# Service account key, relative to the metadata folder. Default: interactive sign-in.
key = gee_key.json
# Compute extra counts and sums per region, for checking results. Costs more EECU. Default False.
audit_stats = False
# One GEE task per country instead of one per region (see Architecture). Default False.
parallel_regions = True
# Price of one EECU-hour, in dollars, for the cost shown while polling. Default 0.4.
eecu_h_pricing = 0.4
# Extract variables that have no GEE handler with the local tiles backend
# (see "Variables not in Earth Engine" below). Default False.
fallback_local = False
# Write GEE outputs to dir_output/gee/ instead of dir_output/, for comparing the
# two backends. A debugging aid; geomerge reads dir_output. Default False.
use_subdir = False
```

Configs from before version 0.2.0 have these settings in `[DEFAULT]` with a `gee_` prefix (`gee_project`, `gee_bucket`, `gee_key`, `gee_audit_stats`, `gee_parallel_regions`, `gee_eecu_h_pricing`, `gee_fallback_local`) and `use_gee_subdir`. They still work, with a warning naming the new setting.

## Supported Datasets

The GEE backend currently supports extraction for the following `geoprepare` datasets:

| Dataset | Description | Notes |
|---------|-------------|-------|
| **AEF** (`aef`) | AlphaEarth Foundations satellite embeddings | Extracted as a multi-year static average. The GEE dataset has different scaling and quantization compared to the raw files from Source Cooperative. |
| **AgERA5** (`agera5_*`) | Agrometeorological indicators | Extracted from the Climate Engine Community Catalog. Supports `agera5_snow_thickness_lwe` and `agera5_dewpoint` (converted from Kelvin to °C, as in the tiles backend). Only variables the tiles backend also extracts are added, so both backends give the same columns. |
| **CHIRPS** (`chirps`) | Precipitation data | CHIRPS v3, IMERG-based daily (`UCSB-CHC/CHIRPS/V3/DAILY_SAT`). Arrives in Earth Engine weeks after the CHC server: on 2026-10-11 the last image was 2026-08-31. |
| **CPC** (`cpc_*`) | Temperature and precipitation | Supports `cpc_tmax`, `cpc_tmin` and `cpc_precip`. Earth Engine stores precipitation in 0.1 mm; it is divided by 10 to give mm. Its current version starts in 2006, so earlier years are empty. |
| **ESI** (`esi`) | Evaporative Stress Index | ESI data in GEE often has a processing lag. The pipeline extracts the last available data for the given period. Supports `esi_4wk` and `esi_12wk`. |
| **NDVI** (`ndvi`) | Vegetation index (MODIS) | Because the GEE pipeline pulls daily observations rather than the 8-day composite blocks used by the legacy `tiles` backend, the timing and number of missing observations at the start or end of a time-series will differ. `geomerge` naturally handles linear interpolation of these gaps across year boundaries. |
| **NSIDC** (`nsidc`) | SMAP L4 soil moisture (`NASA/SMAP/SPL4SMGP/008`) | Supports `nsidc_surface` and `nsidc_rootzone`. |
| **SoilGrids** (`soilgrids`) | Soil properties | Extracted as a multi-year static dataset. |
| **VIIRS** (`viirs`) | High-res VIIRS NDVI data | |

## Variables not in Earth Engine

Some geoprepare variables have no copy in Earth Engine or its community catalog, so this backend cannot extract them:

| Variable | Why |
|----------|-----|
| `chirts_era5_tmax`, `chirts_era5_tmin` | Earth Engine has only CHIRTS-daily v1, which ends in 2016. |
| `chirps_gefs` | CHIRPS-GEFS is not in Earth Engine. |
| `etref` | NCEP RefET is not in Earth Engine. AgERA5's FAO-56 reference ET is, but it is a different product and geoprepare has no variable for it, so it is not offered as a replacement. |

Other variables without a handler (for example `lst`, `fpar`, `ndvi_cdr`) are treated the same way.

By default (`fallback_local = False`) the run stops with an error listing these variables, before any GEE task is submitted. With `fallback_local = True`, each one is logged with a warning and extracted by geoprepare's local tiles backend while the GEE tasks run. The local backend reads files that geodownload has already written to `dir_intermed`; geoextract does not download them. If they are missing, the run stops before submitting any task and names the geodownload dataset to run (for example `CHIRTS-ERA5`). Fallback variables are written to `dir_output`, not to `dir_output/gee/`, even with `use_subdir = True`. They are extracted one combination at a time.

## Architecture

1.  **Task Submission**: The pipeline extracts data using one of two modes, controlled by `parallel_regions` in `[GEE]`:
    - **Sequential Mode** (`parallel_regions = False`): For each region and variable combination, a separate GEE task is created. 
    - **Parallel Batch Mode** (`parallel_regions = True`): All regions within a country are bundled into a single `ee.FeatureCollection` and processed simultaneously by GEE via `reduceRegion` mapping. This submits a single "bulk" GEE task per year/variable, reducing the task queue size and may also avoid quota limits.
      1. **Note:** For variables requiring heavy reductions (e.g., `nsidc` converting 3-hourly images to daily means), Parallel Batch Mode may cause GEE tasks to take more time to run, time out or run out of memory. In case of error, using Sequential Mode (`parallel_regions = False`) is recommended for these variables.
      2. **Note:** In Parallel Batch Mode, incremental updates for the current year do not generate partial files like Sequential Mode. Instead, the pipeline automatically re-extracts the entire year and overwrites the local CSVs to ensure seamless data coverage without complex local file merging and different time windows required by each variable.
2.  **Polling**: The process tracks the submitted tasks and polls Earth Engine until all tasks are marked as `COMPLETED` or `FAILED`. To prevent unexpected billing, polling monitors EECU usage and cancels runaway tasks exceeding defined limits.
3.  **Download and Format**: Once tasks are complete, the resulting CSVs are downloaded concurrently from Google Cloud Storage using `transfer_manager`. If Parallel Batch Mode was used, the bulk CSV is automatically parsed and split locally into the individual region CSVs expected by `geomerge`.
