Metadata-Version: 2.4
Name: defair
Version: 0.4.0rc2
Summary: Destination Earth Framework for preparing AI-Ready data
Keywords: eumetsat,defair,destine,destination earth,machine learning,earth observation
Author: Dusan Figala, Erwan Fassot
Author-email: Dusan Figala <dusan@oxidian.com>, Erwan Fassot <erwan@oxidian.com>
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Dist: defair-ops
Requires-Dist: defair-data
Requires-Dist: typer>=0.19.2
Requires-Dist: loguru>=0.7.3
Requires-Dist: h5netcdf>=1.6.4
Requires-Dist: pydantic>=2.12.3
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: psutil>=7.1.1
Requires-Dist: bokeh>=3.8.2
Requires-Dist: urllib3>=2.6.3
Requires-Dist: aiohttp>=3.13.3
Requires-Dist: dask-gateway>=2023.1.1
Requires-Dist: satpy>=0.60 ; python_full_version >= '3.11' and extra == 'benchmark'
Requires-Dist: pyresample>=1.28 ; extra == 'benchmark'
Requires-Dist: matplotlib>=3.10,<3.11 ; extra == 'benchmark'
Requires-Dist: python-geotiepoints>=1.7.5 ; extra == 'benchmark'
Requires-Dist: cartopy>=0.25.0 ; extra == 'visualization'
Requires-Python: >=3.10
Provides-Extra: benchmark
Provides-Extra: visualization
Description-Content-Type: text/markdown

# DEFAIR

DEFAIR (Destination Earth Framework for preparing AI-Ready data) reads,
transforms, aligns, and writes meteorological and Earth-observation data as
Dask-backed xarray datasets. Its plugin system supports EUMETSAT product
readers, local and S3-compatible storage, geospatial transformations, and
NetCDF/Zarr output.

## Installation

DEFAIR requires Python 3.10 or later. Install a supplied wheel bundle into a
virtual environment:

```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install /path/to/dist/*.whl
```

For development from a source checkout, install the uv-managed workspace:

```bash
make install
```

DEFAIR releases frequently. Pin the exact version in reproducible pipelines
and check the release notes, which list the resolved ticket numbers, before
upgrading.

## First dataset

`Dataset.from_source()` accepts a local path, an S3 URI, a glob, or an explicit
list of paths. The reader and source plugins are auto-detected when possible.

```python
from defair_data import Dataset

dataset = Dataset.from_source(
    "/data/MSG3-SEVI-MSG15-product.nat",
    reader="msg15nat",
    channels=["IR_108"],
    calibration="brightness_temperature",
    use_channel_names=True,
)

print(dataset.data)
dataset.to_file("output/msg.zarr", writer="zarrv2")
```

Reader results are Dask-backed by default. Do not pass the removed `use_dask`
argument.

## S3 source selection

For an `s3://` URI, omitting `source` auto-selects `source="s3"`. This is the
s3fs/fsspec-backed plugin and is the normal choice for xarray and Dask
workflows. `source="s3boto"` selects the direct boto3 plugin for source-level
operations such as range-based file access, listing, and globbing.

```python
dataset = Dataset.from_source(
    "s3://bucket/path/product.nc",
    source="s3",  # optional: this is the auto-detected S3 default
    source_kwargs={
        "endpoint_url": "https://s3.example.com",
        "aws_access_key_id": "...",
        "aws_secret_access_key": "...",
    },
    reader="mtg_fci_l1c_nc",
)
```

Both plugins pass the same read settings to xarray. `s3boto` changes how DEFAIR
lists files and performs direct reads, but xarray readers may still use s3fs.
See [Data loading, S3, and streaming](docs/data-loading.md) for the differences,
credential options, and performance settings.

## Large multi-file datasets

Normal multi-file loading builds one `Dataset`, while reading array data only
when it is needed. For a large collection of files, use `streaming=True`.
DEFAIR processes a fixed number of scene groups at a time and appends each
group to Zarr:

```python
cube = Dataset.from_source(
    "s3://bucket/archive/*.nc",
    reader="mtg_fci_l1c_nc",
    use_channel_names=True,
    streaming=True,
    stream_batch_size=4,
)

cube = cube.transform("content_filter", include_vars=["vis_06", "ir_105"])
cube.to_file("output/archive.zarr", writer="zarrv2")
```

A streaming plan does not provide `.data`. It records supported transformations
and writes only to Zarr. Reload the Zarr to inspect the result. Use
`streaming=False` when you need the complete dataset or an operation that
combines all files.

## Discovering capabilities and reader options

List registered plugins and inspect the selected reader's constructor and
`read()` signatures. Options from both signatures can be passed through
`Dataset.from_source()`. IDE completion on `from_source()` shows only its
common parameters because reader-specific options travel through `**kwargs`:

```python
from inspect import signature

from defair.plugin_manager import list_readers, list_sources, load_reader

print(list_sources())
print(list_readers())

reader = load_reader("msg15nat")
print(signature(type(reader)))  # calibration, chunks, use_channel_names, ...
print(signature(reader.read))   # channels, lazy_coordinates, ...
```

To resolve a reader short name to its product name and EUMETSAT collection ID, use
`describe_readers()` (or `defair readers` on the command line):

```python
from defair_data.readers import describe_readers, get_reader_info

print(describe_readers())
print(get_reader_info("mtg_fci_l2_amv"))  # EO:EUM:DAT:0676
```

The full table is in `docs/reader-catalogue.md`.

The current API supports `use_channel_names`, but not `use_channel_times`.
Each reader creates the time coordinates required by its product. Reader
arguments are validated against the reader's declared parameters, so an
unrecognised option raises and lists the accepted arguments rather than being
ignored. This applies to third-party reader plugins too: a `**kwargs` catch-all
on `read()` no longer widens what a reader accepts, so declare every supported
option by name.

## Documentation

- [Getting started](docs/what-is-defair.md): what DEFAIR is, installation, a first product and a first workflow
- [Data loading, S3, streaming, and reader options](docs/data-loading.md)
- [Workflow examples](docs/workflow-examples.md)
- [Dask cluster recommendations](docs/dask-cluster-recommendations.md)
- [Performance benchmarks](docs/benchmarks.md)
- [Demonstration notebooks](docs/demos/)
- [Collocation AI use case](packages/defair-ai-demo/msg_mtg_era5_collocation_ai_usecase.ipynb),
  the notebook that puts MSG, MTG and ERA5 on one grid and trains a cloud-mask model on it
- [Reader](docs/reader-plugins.md), [writer](docs/writer-plugins.md), and
  [transformation](docs/transformation-plugins.md) plugin development

The command-line entry point also lists the installed commands and options:

```bash
defair --help
```

## Logging

Set logging with CLI flags or environment variables:

```bash
defair --log-level DEBUG version
defair --log-json version
defair --log-module-level "defair_data:DEBUG,defair_ops:WARNING" version

export DEFAIR_LOG_LEVEL=INFO
export DEFAIR_LOG_JSON=true
```

Python callers can configure logging directly:

```python
from defair.logging import setup_logging

setup_logging(log_level="INFO", json_output=False)
```

## Development

The uv workspace contains three publishable packages:

- `defair` (`src/defair/`): CLI and workflow orchestration
- `defair-data` (`packages/defair-data/`): data sources, readers, and writers
- `defair-ops` (`packages/defair-ops/`): transformations and Dask management

These are mutually dependent rather than strictly layered: each imports the other two, so each
declares the other two, and installing any one of them pulls in the rest. The coupling exists
because the shared kernel — the entry-point plugin manager, `CFHistoryMixin`, kwargs validation
and logging — lives in `defair` rather than in a leaf package, so the lower-level packages call
back up into it. A new cross-package import therefore has to be declared in that package's
`pyproject.toml`, even though a workspace `uv sync` satisfies it either way.

Common development commands:

```bash
make install
make check
make test
make docs
```

Add a dependency to a workspace package with:

```bash
uv add --package <defair|defair-data|defair-ops> <dependency_name>
```

All Python source files require Apache 2.0 license headers. Run
`make headers-check` to validate them or `make headers-add` to add missing
headers.

## Release maintenance

Build all workspace packages with:

```bash
uv build --all-packages
```

Version commands use uv's workspace support:

```bash
uv version --package <defair|defair-data|defair-ops> --bump <major|minor|patch|rc|stable>
```

For a patch to an older release, merge the fix to `main`, create a release
branch from the target tag, cherry-pick the fix, update package versions, and
tag the patched release.

## Building the documentation

Build the local Material for MkDocs site with:

```bash
make docs
```

To generate the DEDL ReadTheDocs/Sphinx source tree, run:

```bash
uv run python scripts/generate_docs.py path/to/target/directory
```

[![coverage report](https://gitlab.eumetsat.int/defair/defair-core/badges/main/coverage.svg)](https://gitlab.eumetsat.int/defair/defair-core/-/commits/main)
