Metadata-Version: 2.4
Name: rex3handler
Version: 0.3.0
Summary: Download, extract, and convert the REX3 MRIO database to Zarr.
Author-email: Yanfei Shan <yanfei.shan@tum.de>
License: MIT
Project-URL: Homepage, https://github.com/FayeShan/REX3_handler
Project-URL: Issues, https://github.com/FayeShan/REX3_handler/issues
Project-URL: Dataset, https://zenodo.org/records/10354283
Keywords: MRIO,REX3,EXIOBASE,input-output,zarr
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.31
Requires-Dist: tqdm>=4.66
Requires-Dist: numpy>=1.24
Requires-Dist: h5py>=3.9
Requires-Dist: xarray>=2023.0
Requires-Dist: zarr<3,>=2.16
Requires-Dist: pandas>=2.0
Requires-Dist: openpyxl>=3.1
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# REX3Handler

**REX3Handler** downloads, extracts and converts the **REX3 MRIO database** (Resolved EXIOBASE 3) from Zenodo, MATLAB `.mat` files into analysis-ready, chunked **Zarr** stores with labelled `xarray` coordinates.

[![CI](https://github.com/FayeShan/REX3_handler/actions/workflows/ci.yml/badge.svg)](https://github.com/FayeShan/REX3_handler/actions)
[![PyPI](https://img.shields.io/pypi/v/rex3handler)](https://pypi.org/project/rex3handler/)

---

## About REX3

**REX3** (Resolved EXIOBASE version 3) resolves EXIOBASE 3 to **189 countries × 163 sectors** for **1995–2022** and carries 18 environmental and socio-economic satellite extensions (climate, PM health, blue water, water stress, land use, biodiversity, value added, workforce, ...). It is developed by Livia Cabernard, Stephan Pfister and Stefanie Hellweg (ETH Zürich / TUM).

**Please cite the database in your papers:**

> Cabernard, L., Pfister, S., Hellweg, S. (2024). *Resolved Exiobase version 3 (REX3)* [Data set]. Zenodo. https://doi.org/10.5281/zenodo.10354283
> Method: Cabernard, L. & Pfister, S. (2021). A highly resolved MRIO database for analyzing environmental footprints and Green Economy Progress. *Science of The Total Environment* 755, 142587. https://doi.org/10.1016/j.scitotenv.2020.142587

Each `REX3_<year>.zip` (3.3–4.4 GB) contains `T_REX3.mat` (transaction matrix), `Y_REX3.mat` (final demand matrix), `Q_REX3.mat` / `Q_Y_REX3.mat` (satellite extensions of production / of final demand). `REX3_Labels.zip` holds the country, sector and satellite extension labels.

---

## Install

```bash
pip install rex3handler        # Python >= 3.10
```

## Quick start (CLI)

```bash
rex3 list                                  # files in the Zenodo record (name, size, md5)
rex3 download single --year 2022           # one year (+ labels), ~4 GB
rex3 download range --start 2018 --end 2022
rex3 download all                          # whole record (~117 GB)
rex3 extract                               # downloads/ -> unzipped/
rex3 convert                               # unzipped/ -> REX3ZARR/<year>.zarr
```

Downloads are **sequential, resumable and md5-verified**: an interrupted transfer continues from its `.part` file on the next run, every file is checked against the Zenodo checksum before it is kept, and a `manifest.csv` records name / size / md5 / status. Rate limits (`429`/`503`) are honoured. There is deliberately no parallel-download option — the archives are bandwidth-bound and parallel requests only raise the risk of being blocked.

## Python API

```python
from rex3handler import download_files, unzip_dir, convert_years_to_zarr, open_year

download_files("single", year=2022)                 # returns list of failed files ([] on success)
unzip_dir()
convert_years_to_zarr(years=[2022])                 # labels picked up from unzipped/REX3_Labels

ds = open_year(2022)                                # lazy xarray.Dataset
ds["T"].sel(region="China", sector="Cultivation of wheat").sum(("region_to", "sector_to"))
```

## Aggregation and basic Leontief accounting

```python
from rex3handler import (load_grouping_xlsx, make_grouping, aggregate, total_output,
                         coefficients, leontief_inverse, footprint,
                         production_based, consumption_based, check_balance)

reg, sec = load_grouping_xlsx("examples/REX3_grouping.xlsx", ds)   # or make_grouping(ds.region.values, {...})
agg = aggregate(ds, reg, sec)              # same grouping applied to T, Y, Q, Q_Y; x recomputed after
L   = leontief_inverse(coefficients(agg))  # dense (I - A)^-1 on the aggregated system
fp  = footprint(agg, "Blue water consumption [Mio. m3]", L)   # producer (region, sector) -> consumer region_fd
production_based(agg, "Value added [Mio Euro]"), consumption_based(agg, "Value added [Mio Euro]", L)
check_balance(agg)                         # supply/use identity + PBA == CBA closure
```

Aggregation streams `T` one supplier region at a time (a full year in ~6 s, < 1 GB). Dense inversion is
intended for aggregated systems (a few thousand country-sectors); the native 30,807-dimension system needs
an iterative solver, which is out of scope here. See [`examples/quickstart.ipynb`](examples/quickstart.ipynb)
for a worked tutorial on REX3 2022.

## Data layout

| Variable | Dimensions                               | Meaning                                                                        |
| -------- | ---------------------------------------- | ------------------------------------------------------------------------------ |
| `T`    | `region, sector, region_to, sector_to` | flow from supplier`(region, sector)` to purchaser `(region_to, sector_to)` |
| `Y`    | `region, sector, region_fd`            | final demand of`region_fd` for the output of supplier `(region, sector)`   |
| `Q`    | `indicator, region, sector`            | extension of production (18 indicators)                                        |
| `Q_Y`  | `indicator, region_fd`                 | direct extension of final demand                                               |

Values are `float32`; monetary units are Mio EUR (current prices of the data year); extension units are in the `indicator` labels. `T` is chunked one supplier region per chunk (`1 × 163 × 189 × 163`) so slicing by producing country is cheap.

**Naming.** Rows (the producing / supplying side) keep the plain `region, sector` names used by pymrio and EXIOBASE, so `T` rows, `Y` rows, `Q` columns and total output all share one coordinate. The receiving side of `T` carries the `_to` suffix ("trade from region *i* to region *j*"), and the final-demand region is `region_fd` — the same `FDreg` used in the REX3 authors' own MATLAB code (rows there are `Preg/Psec`, the *production perspective*).

**Orientation.** MATLAB v7.3 files store arrays transposed in HDF5; the converter restores MATLAB orientation, in which `T[i, j]` is the flow from supplier *i* to purchaser *j*. This was verified on REX3 2022 through the supply/use identity (row-sum + final demand vs column-sum + value added match to ~1e-4 relative; the flipped orientation is off by two orders of magnitude).

## Resources

- Conversion streams `T` in blocks: peak memory ≈ 0.9 GB per year, ~1 minute per year on a laptop.
- Disk: ~4 GB per raw ZIP, ~2.2 GB per Zarr year.

## Development

```bash
pip install -e ".[dev]"
pytest -q
```

## License

MIT — see `LICENSE`. The REX3 data itself is distributed by its authors under the license stated on Zenodo (CC-BY 4.0).
