Metadata-Version: 2.4
Name: usdata
Version: 0.8.0
Summary: Unified Python SDK and CLI for discovering, fetching, and tracking provenance of U.S. public scientific data
Keywords: noaa,usgs,nasa,open-data,scientific-data,provenance
Author: Jake Van Slyke
Author-email: Jake Van Slyke <jakervanslyke@gmail.com>
License-Expression: Apache-2.0
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Requires-Dist: httpx>=0.28.1
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6.0
Requires-Dist: typer>=0.12
Requires-Dist: xarray>=2025.1 ; extra == 'netcdf'
Requires-Dist: h5netcdf[h5py]>=1.8.1 ; extra == 'netcdf'
Requires-Dist: pandas>=3.0 ; extra == 'pandas'
Requires-Dist: xradar>=0.12.0 ; extra == 'radar'
Requires-Python: >=3.11
Project-URL: Homepage, https://github.com/jakeryderv/usdata
Project-URL: Repository, https://github.com/jakeryderv/usdata
Provides-Extra: netcdf
Provides-Extra: pandas
Provides-Extra: radar
Description-Content-Type: text/markdown

# usdata

Unified Python SDK and CLI for discovering, fetching, and tracking the
provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).

> Status: pre-alpha. v0.8 supports GHCN-Daily, GSOM monthly summaries,
> NEXRAD Level II, GOES ABI CONUS imagery, Storm Events annual archives,
> USGS daily values, and CoastWatch SST subsets with provenance. It includes
> optional CSV, radar, and NetCDF4 readers, six executed notebooks, Census
> state/county lookup, and terminal download progress. Other datasets are planned.
> See [docs/roadmap.md](docs/roadmap.md).

## Providers

<!-- registry:start -->
| Provider | Available | Stub | Planned | Next up (unassigned) | Datasets |
|---|---:|---:|---:|---|---|
| [NOAA](docs/providers/noaa.md) | 6 | 0 | 23 | — | `ghcn-daily`, `gsom`, `storm-events`, `nexrad-level2`, `goes-abi`, `coastwatch-sst`, +23 planned |
| [USGS](docs/providers/usgs.md) | 1 | 0 | 2 | — | `water-daily`, +2 planned |
| [Census Bureau](docs/providers/census.md) | 0 | 0 | 1 | — | +1 planned |
| [EPA](docs/providers/epa.md) | 0 | 0 | 1 | — | +1 planned |
| [FEMA](docs/providers/fema.md) | 0 | 0 | 1 | — | +1 planned |
| [NASA](docs/providers/nasa.md) | 0 | 0 | 1 | — | +1 planned |
| [USDA](docs/providers/usda.md) | 0 | 0 | 1 | — | +1 planned |

Available datasets are in `code`, stubs in _italics_; planned ones are counted. Available means implemented in this source checkout; consult the [releases](https://github.com/jakeryderv/usdata/releases) for published support. Each provider page has access notes and full dataset details; [docs/roadmap.md](docs/roadmap.md) lists datasets by target version.
<!-- registry:end -->

## Install

```sh
pip install usdata        # or: uv add usdata
```

## Usage

```python
from usdata import build_query, get, search
from usdata.fetch import fetch

for r in search("precipitation", location="Oklahoma"):
    print(r.dataset.id, r.dataset.title)

ds = get("noaa:ghcn-daily")
query = build_query(
    lat=35.39,
    lon=-97.60,
    radius_km=15,
    start="2024-05-06",
    end="2024-05-07",
    variables=["PRCP", "TMAX"],
)
for item in fetch(ds, query):
    print(item.path, item.provenance.checksum)
```

```sh
usdata search "tornado radar" --state OK
usdata search precipitation --location "Cleveland County, OK"
usdata info noaa:ghcn-daily
usdata fetch noaa:ghcn-daily --lat 35.39 --lon -97.60 --radius-km 15 \
    --start 2024-05-06 --end 2024-05-07 --vars PRCP,TMAX
usdata fetch noaa:ghcn-daily -p stations=USW00013967 --start 2024-01-01 --end 2024-12-31
usdata fetch noaa:nexrad-level2 --lat 35.47 --lon -97.52 \
    --start 2024-05-06T20:00 --end 2024-05-06T23:00        # nearest radar (KTLX)
usdata fetch noaa:nexrad-level2 -p site=KTLX --start 2024-05-06T20:00 --end 2024-05-06T20:30 --dry-run
usdata fetch usgs:water-daily -p sites=07164500 --vars 00060 \
    --start 2024-05-06 --end 2024-05-07
usdata fetch noaa:coastwatch-sst --bbox=-80.08,30.02,-80.02,30.08 \
    --start 2024-05-06T12:00Z --end 2024-05-06T12:00Z    # four grid cells
usdata pull dataset.yaml            # resolve, fetch, write dataset.lock.json
usdata verify dataset.yaml          # exit 1 if any cached input drifted
```

Storm Events bulk access is available since v0.8. Dates select complete
annual details archives; filter rows locally after opening the gzip CSV. For example:

```sh
usdata fetch noaa:storm-events --start 2024-05-01 --end 2024-05-31 --dry-run
```

This lists the entire 2024 archive. Location and variable filters are rejected;
see the [executed Storm Events notebook](examples/storm-events/example.ipynb)
for local filtering and reporting limitations.

Fetched files land in `~/.cache/usdata/<provider>/<dataset>/` (override with
`USDATA_CACHE_DIR` or `--cache-dir`), each with a `.provenance.json` sidecar
recording source URL, retrieval time, checksum, size, and license.

Locations accept state names/postal codes, county/state names, and quoted FIPS
codes. These select bounding rectangles; see [place lookup](docs/reference/places.md)
for coverage, ambiguity, and antimeridian limits. CoastWatch CSV includes a
second header row containing units; see its [access notes](docs/providers/noaa.md#coastwatch-sst).

Manifest and source fields are validated strictly; unknown fields are errors.
Provider-specific options belong under `params`.

A manifest declares every input a project needs. `pull` resolves each source,
fetches it, and writes `dataset.lock.json` pinning every asset with its checksum
and provenance. A second `pull` restores exactly what the lockfile pins without
re-querying upstream, so the inputs stay reproducible even if the source
changes. `verify` checks the manifest checksum and re-hashes cached files
against the lockfile. Editing the manifest after locking requires `pull --force`
to re-resolve. A required source matching no assets fails the pull; set
`allow_empty: true` on a source only when an empty result is intentional.

Checksums detect upstream changes; they cannot recover historical bytes that
are no longer available. Preserve the cache for long-lived reproducibility.
See the [manifest reference](docs/reference/manifests.md) and the small
[NOAA/USGS example](examples/weather-and-streamflow/README.md).

```yaml
name: tornado-environment
sources:
  - dataset: noaa:nexrad-level2
    location: oklahoma
    start: 2024-05-06
    end: 2024-05-07
  - dataset: noaa:ghcn-daily
    location: oklahoma
    start: 2024-05-01
    end: 2024-05-31
```

Terminal progress is available since v0.7. On a terminal, `fetch` and
`pull` show progress on stderr: resolved asset counts,
known bytes and unknown sizes, HTTP download bytes for the current attempt, and
validated cache hits. `fetch --dry-run` also summarizes known sizes. Asset totals
include possible cache hits; each manifest source is resolved separately. Bytes
from a failed HTTP attempt reset on retry; encoded responses have unknown decoded
size. Adapters that assemble files from metadata requests show asset-level progress.
Use `--no-progress` to disable it. Progress is automatically disabled when either
stdout or stderr is redirected; existing output lines and exit codes are unchanged.

For single-channel GOES CONUS imagery (available since v0.8):

```sh
usdata fetch noaa:goes-abi --start 2024-05-06T12:01:18.1Z --end 2024-05-06T12:01:18.1Z -p satellite=18 -p channel=6
```

The download is a whole NetCDF scene. See [GOES access notes](docs/providers/noaa.md#goes-abi-conus-imagery)
for supported selectors and scan-start time semantics.

## Opening CSV data

`FetchedAsset.open()` is available since v0.6 with the optional pandas
extra (`pip install "usdata[pandas]"`). It reads cached CSV into a DataFrame,
preserves identifier strings, and keeps CoastWatch units as metadata.
See the [reader reference](docs/reference/readers.md)
and [fetch → open → analyze example](examples/sst-analysis/README.md).

The [examples directory](examples/README.md) contains executed Jupyter notebooks
with saved data previews, small plots, and source provenance. Start with weather
and streamflow for manifest workflows, SST for gridded CSV reading, or monthly
climate for GSOM observations.

NetCDF4 scene opening is available since v0.8 with `usdata[netcdf]`.
See the executed [GOES infrared notebook](examples/goes-imagery/example.ipynb).

## Development

Requires [uv](https://docs.astral.sh/uv/) and [just](https://just.systems/).

`just setup` uses the tested Python 3.14.7 pin in `.python-version`. Older Linux
uv Python 3.14 builds can crash during NumPy array operations; see
[the upstream fix](https://github.com/astral-sh/python-build-standalone/issues/991).

```sh
git clone https://github.com/jakeryderv/usdata && cd usdata
just setup     # install toolchain and dependencies
just test      # unit tests
just check     # format, lint, typecheck, offline tests, generated docs, release notices
just check-pandas  # install the CSV extra and run the same checks
just check-radar   # install the radar extra and run the same checks
just check-netcdf  # install the NetCDF4 extra and run the same checks
just notebooks    # launch the optional Jupyter examples environment
just run-notebooks # execute notebooks live in fresh kernels and temporary caches
just build     # build wheel and sdist
just smoke     # exercise core and pandas wheel installations outside the checkout
just run search radar
```

Unit tests mechanically block network connections. Integration tests that hit
live services run with `just test-integration`. CI checks Python 3.11 and 3.14 on
Linux with core-only, pandas, radar, and NetCDF dependency profiles. Installed-wheel
checks cover all four profiles on Linux, macOS, and Windows. The full unit and
live-service suites run on Linux. `just setup` restores a core-only development
environment; the `check-pandas`, `check-radar`, and `check-netcdf` commands install
their respective extras.

Releases: `just release minor` opens a version-bump PR; merging it publishes
to PyPI and creates the tag and GitHub release. See
[docs/versioning.md](docs/versioning.md).

See [docs/providers/](docs/providers/) for per-provider access notes,
[docs/architecture.md](docs/architecture.md) for how the pieces fit,
[docs/adr/](docs/adr/) for why, and [CONTRIBUTING.md](CONTRIBUTING.md) to add
a dataset.

## License

Apache-2.0
