Metadata-Version: 2.4
Name: nemdatatools
Version: 0.2.0
Summary: Tools for accessing and preprocessing AEMO data for the National Electricity Market
Author-email: Zhipeng He <zhipeng.he@hdr.qut.edu.au>
License: MIT
Project-URL: Homepage, https://github.com/ZhipengHe/nemdatatools
Project-URL: Documentation, https://zhipenghe.me/nemdatatools/
Project-URL: Bug Tracker, https://github.com/ZhipengHe/nemdatatools/issues
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.25.0
Requires-Dist: pandas>=1.3.0
Requires-Dist: numpy>=1.20.0
Requires-Dist: pyarrow>=19.0.0
Requires-Dist: tqdm>=4.0.0
Requires-Dist: beautifulsoup4>=4.9.0
Provides-Extra: dev
Requires-Dist: pytest>=6.0; extra == "dev"
Requires-Dist: pytest-cov>=2.12; extra == "dev"
Requires-Dist: black>=25.0.0; extra == "dev"
Requires-Dist: mypy>=1.14.0; extra == "dev"
Requires-Dist: types-requests>=2.32.0; extra == "dev"
Requires-Dist: isort>=6.0.0; extra == "dev"
Requires-Dist: pre-commit>=3.3.2; extra == "dev"
Requires-Dist: ruff>=0.5.0; extra == "dev"
Requires-Dist: commitizen>=4.4.0; extra == "dev"
Requires-Dist: twine>=6.0.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx>=8.0.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=3.0.0; extra == "docs"
Requires-Dist: myst-parser>=4.0.0; extra == "docs"
Dynamic: license-file

# NEMDataTools

An MIT-licensed Python package for accessing and preprocessing Australian
Energy Market Operator (AEMO) data for the National Electricity Market (NEM).

## How it works

AEMO publishes the same market data at three ages, and NEMDataTools models
that system directly instead of hard-coding one source per table:

| Tier | Location | Granularity | Retention |
|------|----------|-------------|-----------|
| Reports CURRENT | `nemweb.com.au/Reports/Current/` | one file per event | rolling days (varies per package) |
| Reports ARCHIVE | `nemweb.com.au/Reports/Archive/` | daily bundles | ~13 months |
| MMSDM Data Archive | `nemweb.com.au/Data_Archive/` | monthly snapshots | 2009 → ~6 weeks ago |

One `fetch()` call stitches whichever tiers a date range needs. Remote files
are discovered by reading directory listings and pattern-matching — never by
constructing filenames — so AEMO's filename-format changes (such as the
August 2024 `PUBLIC_DVD_*` → `PUBLIC_ARCHIVE#*` switch) and multi-part
archives are handled transparently, including all `FILEnn` parts of large
tables. Known holes in a table's history (for example the bid tables removed
at the 2021 five-minute-settlement transition) raise a clear error naming
the substitute table instead of returning silently partial data.

## Installation

```bash
pip install nemdatatools
```

Requires Python 3.11+. Dependencies: pandas, pyarrow, requests,
beautifulsoup4.

## Quick start

```python
import nemdatatools as ndt

# One call, any range — tiers are stitched automatically. Start small:
# a week is a quick download; multi-year ranges (e.g. 2020 -> today)
# work the same way but fetch months of archive files on first run.
prices = ndt.fetch(
    "DISPATCHPRICE",
    "2026/06/01",
    "2026/06/07",
    regions=["QLD1"],
)

# Interval-ending-aware resampling: the 00:30 bucket aggregates the six
# 5-minute rows stamped 00:05..00:30, matching AEMO's own convention.
half_hourly = ndt.resample(prices[["RRP"]], "30min")

# Frames holding several regions/units must be grouped explicitly:
all_regions = ndt.fetch("DISPATCHPRICE", "2026/06/01", "2026/06/07")
daily = ndt.resample(all_regions, "1D", by="REGIONID", trading_day=True)

# Aggregated price+demand CSVs (aemo.com.au visualisation service):
pd_data = ndt.fetch_price_and_demand("2024/01/01", "2024/12/31", ["NSW1"])

# Discovery:
ndt.tables()                        # curated table names
ndt.availability("BIDPEROFFER_D")   # tier locations + known gaps

# Escape hatch: any of the ~236 MMSDM tables, era-aware, all parts:
gencon = ndt.fetch_mmsdm_table("GENCONDATA", "2026/05/01", "2026/05/31")
```

All datetimes are naive **NEM time** (fixed UTC+10, no daylight saving);
timezone-aware datetimes are rejected rather than silently converted. AEMO
timestamps mark the **end** of the interval they describe.

## Curated tables

`fetch()` accepts curated tables spanning four families; each is wired to
its locations in every tier and era:

- **Prices & demand** — `DISPATCHPRICE`, `TRADINGPRICE`,
  `DISPATCHREGIONSUM`, `TRADINGINTERCONNECT`, `DISPATCHINTERCONNECTORRES`
- **Generation & SCADA** — `DISPATCH_UNIT_SCADA`, `DISPATCHLOAD`,
  `ROOFTOP_PV_ACTUAL`
- **Forecasts** — `P5MIN_REGIONSOLUTION`, `P5MIN_INTERCONNECTORSOLN`,
  `PREDISPATCHPRICE`, `PREDISPATCHREGIONSUM`, `PREDISPATCHLOAD`
- **Bids & offers** — `BIDDAYOFFER_D`, `BIDPEROFFER_D`, `BIDDAYOFFER`,
  `BIDPEROFFER` (plus pre-2021 `TRADINGREGIONSUM` for history)

Every other MMSDM table is reachable through `fetch_mmsdm_table()`.

## Caching

Downloads land under `~/.nemdatatools/` (override with
`ndt.Cache("path")` passed as `cache=`):

- `raw/` mirrors nemweb paths — full provenance, a parser fix never forces
  a re-download;
- `parquet/` stores parsed per-table frames, so repeat reads skip zip
  extraction and CSV parsing entirely.

## Data attribution

Data is © AEMO and provided under AEMO's terms; this package downloads
publicly available files and does not redistribute data.

## License

MIT — see [LICENSE](https://github.com/ZhipengHe/nemdatatools/blob/master/LICENSE).
