Metadata-Version: 2.4
Name: mostlyright-data
Version: 0.17.1
Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
Project-URL: Homepage, https://mostlyright.md/
Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/
Requires-Python: <3.14,>=3.13
Provides-Extra: build
Requires-Dist: hatchling==1.28.0; extra == 'build'
Requires-Dist: uv-build==0.11.3; extra == 'build'
Provides-Extra: dev
Requires-Dist: antlr4-python3-runtime==4.9.3; extra == 'dev'
Requires-Dist: astropy==8.0.1; extra == 'dev'
Requires-Dist: cffi==2.1.0; extra == 'dev'
Requires-Dist: cmarkgfm>=2025.10.22; extra == 'dev'
Requires-Dist: colorlog==6.12.0; extra == 'dev'
Requires-Dist: croniter==6.2.4; extra == 'dev'
Requires-Dist: cryptography==50.0.0; extra == 'dev'
Requires-Dist: duckdb==1.5.5; extra == 'dev'
Requires-Dist: eccodes==2.48.0; extra == 'dev'
Requires-Dist: eccodeslib==2.48.0.26; (sys_platform != 'win32') and extra == 'dev'
Requires-Dist: et-xmlfile==2.0.0; extra == 'dev'
Requires-Dist: flatbuffers==25.12.19; extra == 'dev'
Requires-Dist: h5py==3.16.0; extra == 'dev'
Requires-Dist: idna==3.18; extra == 'dev'
Requires-Dist: jsonschema[format]<5,>=4.23; extra == 'dev'
Requires-Dist: lxml==6.1.2; extra == 'dev'
Requires-Dist: narwhals==2.25.0; extra == 'dev'
Requires-Dist: netcdf4==1.7.4; extra == 'dev'
Requires-Dist: numpy==2.5.1; extra == 'dev'
Requires-Dist: omegaconf==2.3.1; extra == 'dev'
Requires-Dist: onnxruntime==1.28.0; extra == 'dev'
Requires-Dist: opencv-python-headless==5.0.0.93; extra == 'dev'
Requires-Dist: openpyxl==3.1.5; extra == 'dev'
Requires-Dist: packaging==26.3; extra == 'dev'
Requires-Dist: pandas==3.0.3; extra == 'dev'
Requires-Dist: pikepdf==10.10.0; extra == 'dev'
Requires-Dist: pillow==12.3.0; extra == 'dev'
Requires-Dist: polars-runtime-32==1.42.1; extra == 'dev'
Requires-Dist: polars==1.42.1; extra == 'dev'
Requires-Dist: protobuf==7.35.1; extra == 'dev'
Requires-Dist: pyarrow==23.0.1; extra == 'dev'
Requires-Dist: pyclipper==1.4.0; extra == 'dev'
Requires-Dist: pycparser==3.0; extra == 'dev'
Requires-Dist: pyhdf==0.11.7; ((sys_platform == 'linux' and platform_machine == 'x86_64') or (sys_platform == 'darwin' and platform_machine == 'arm64')) and extra == 'dev'
Requires-Dist: pypdfium2==5.13.0; extra == 'dev'
Requires-Dist: pypublicdecompwt==2.8.1.6; extra == 'dev'
Requires-Dist: pyreadstat==1.3.6; extra == 'dev'
Requires-Dist: pytest-xdist>=3.6; extra == 'dev'
Requires-Dist: pytest>=8.3; extra == 'dev'
Requires-Dist: python-dateutil==2.9.0.post0; extra == 'dev'
Requires-Dist: pyyaml==6.0.3; extra == 'dev'
Requires-Dist: rasterio==1.5.1; extra == 'dev'
Requires-Dist: referencing<1,>=0.35; extra == 'dev'
Requires-Dist: ruff>=0.9; extra == 'dev'
Requires-Dist: shapely==2.1.2; extra == 'dev'
Requires-Dist: six==1.17.0; extra == 'dev'
Requires-Dist: zarr==3.3.0; extra == 'dev'
Provides-Extra: hosted
Requires-Dist: mostlyright-studio-client-v3==3.0.0; extra == 'hosted'
Provides-Extra: local
Requires-Dist: antlr4-python3-runtime==4.9.3; extra == 'local'
Requires-Dist: cffi==2.1.0; extra == 'local'
Requires-Dist: colorlog==6.12.0; extra == 'local'
Requires-Dist: croniter==6.2.4; extra == 'local'
Requires-Dist: cryptography==50.0.0; extra == 'local'
Requires-Dist: duckdb==1.5.5; extra == 'local'
Requires-Dist: et-xmlfile==2.0.0; extra == 'local'
Requires-Dist: flatbuffers==25.12.19; extra == 'local'
Requires-Dist: idna==3.18; extra == 'local'
Requires-Dist: lxml==6.1.2; extra == 'local'
Requires-Dist: narwhals==2.25.0; extra == 'local'
Requires-Dist: numpy==2.5.1; extra == 'local'
Requires-Dist: omegaconf==2.3.1; extra == 'local'
Requires-Dist: onnxruntime==1.28.0; extra == 'local'
Requires-Dist: opencv-python-headless==5.0.0.93; extra == 'local'
Requires-Dist: openpyxl==3.1.5; extra == 'local'
Requires-Dist: packaging==26.3; extra == 'local'
Requires-Dist: pandas==3.0.3; extra == 'local'
Requires-Dist: pikepdf==10.10.0; extra == 'local'
Requires-Dist: pillow==12.3.0; extra == 'local'
Requires-Dist: polars-runtime-32==1.42.1; extra == 'local'
Requires-Dist: polars==1.42.1; extra == 'local'
Requires-Dist: protobuf==7.35.1; extra == 'local'
Requires-Dist: pyarrow==23.0.1; extra == 'local'
Requires-Dist: pyclipper==1.4.0; extra == 'local'
Requires-Dist: pycparser==3.0; extra == 'local'
Requires-Dist: pypdfium2==5.13.0; extra == 'local'
Requires-Dist: pyreadstat==1.3.6; extra == 'local'
Requires-Dist: python-dateutil==2.9.0.post0; extra == 'local'
Requires-Dist: pyyaml==6.0.3; extra == 'local'
Requires-Dist: shapely==2.1.2; extra == 'local'
Requires-Dist: six==1.17.0; extra == 'local'
Provides-Extra: local-search-neural
Requires-Dist: anyio==4.14.2; extra == 'local-search-neural'
Requires-Dist: certifi==2026.7.22; extra == 'local-search-neural'
Requires-Dist: click==8.4.2; extra == 'local-search-neural'
Requires-Dist: filelock==3.32.2; extra == 'local-search-neural'
Requires-Dist: flatbuffers==25.12.19; extra == 'local-search-neural'
Requires-Dist: fsspec==2026.7.0; extra == 'local-search-neural'
Requires-Dist: h11==0.16.0; extra == 'local-search-neural'
Requires-Dist: hf-xet==1.6.0; extra == 'local-search-neural'
Requires-Dist: httpcore==1.0.9; extra == 'local-search-neural'
Requires-Dist: httpx==0.28.1; extra == 'local-search-neural'
Requires-Dist: huggingface-hub==1.27.0; extra == 'local-search-neural'
Requires-Dist: idna==3.18; extra == 'local-search-neural'
Requires-Dist: numpy==2.5.1; extra == 'local-search-neural'
Requires-Dist: onnxruntime==1.28.0; extra == 'local-search-neural'
Requires-Dist: packaging==26.3; extra == 'local-search-neural'
Requires-Dist: protobuf==7.35.1; extra == 'local-search-neural'
Requires-Dist: pyyaml==6.0.3; extra == 'local-search-neural'
Requires-Dist: tokenizers==0.23.1; extra == 'local-search-neural'
Requires-Dist: tqdm==4.70.0; extra == 'local-search-neural'
Requires-Dist: typing-extensions==4.16.0; extra == 'local-search-neural'
Provides-Extra: recorder
Requires-Dist: cryptography==50.0.0; extra == 'recorder'
Provides-Extra: release
Requires-Dist: antlr4-python3-runtime==4.9.3; extra == 'release'
Requires-Dist: cffi==2.1.0; extra == 'release'
Requires-Dist: colorlog==6.12.0; extra == 'release'
Requires-Dist: croniter==6.2.4; extra == 'release'
Requires-Dist: cryptography==50.0.0; extra == 'release'
Requires-Dist: duckdb==1.5.5; extra == 'release'
Requires-Dist: et-xmlfile==2.0.0; extra == 'release'
Requires-Dist: flatbuffers==25.12.19; extra == 'release'
Requires-Dist: idna==3.18; extra == 'release'
Requires-Dist: jsonschema[format]<5,>=4.23; extra == 'release'
Requires-Dist: lxml==6.1.2; extra == 'release'
Requires-Dist: narwhals==2.25.0; extra == 'release'
Requires-Dist: numpy==2.5.1; extra == 'release'
Requires-Dist: omegaconf==2.3.1; extra == 'release'
Requires-Dist: onnxruntime==1.28.0; extra == 'release'
Requires-Dist: opencv-python-headless==5.0.0.93; extra == 'release'
Requires-Dist: openpyxl==3.1.5; extra == 'release'
Requires-Dist: packaging==26.3; extra == 'release'
Requires-Dist: pandas==3.0.3; extra == 'release'
Requires-Dist: pikepdf==10.10.0; extra == 'release'
Requires-Dist: pillow==12.3.0; extra == 'release'
Requires-Dist: polars-runtime-32==1.42.1; extra == 'release'
Requires-Dist: polars==1.42.1; extra == 'release'
Requires-Dist: protobuf==7.35.1; extra == 'release'
Requires-Dist: pyarrow==23.0.1; extra == 'release'
Requires-Dist: pyclipper==1.4.0; extra == 'release'
Requires-Dist: pycparser==3.0; extra == 'release'
Requires-Dist: pypdfium2==5.13.0; extra == 'release'
Requires-Dist: pyreadstat==1.3.6; extra == 'release'
Requires-Dist: python-dateutil==2.9.0.post0; extra == 'release'
Requires-Dist: pyyaml==6.0.3; extra == 'release'
Requires-Dist: shapely==2.1.2; extra == 'release'
Requires-Dist: six==1.17.0; extra == 'release'
Provides-Extra: scientific
Requires-Dist: astropy==8.0.1; extra == 'scientific'
Requires-Dist: eccodes==2.48.0; extra == 'scientific'
Requires-Dist: eccodeslib==2.48.0.26; (sys_platform != 'win32') and extra == 'scientific'
Requires-Dist: h5py==3.16.0; extra == 'scientific'
Requires-Dist: netcdf4==1.7.4; extra == 'scientific'
Requires-Dist: pillow==12.3.0; extra == 'scientific'
Requires-Dist: pyhdf==0.11.7; ((sys_platform == 'linux' and platform_machine == 'x86_64') or (sys_platform == 'darwin' and platform_machine == 'arm64')) and extra == 'scientific'
Requires-Dist: pypublicdecompwt==2.8.1.6; extra == 'scientific'
Requires-Dist: rasterio==1.5.1; extra == 'scientific'
Requires-Dist: zarr==3.3.0; extra == 'scientific'
Description-Content-Type: text/markdown

# mostlyright-data

Browse datasets built by our community or create your own with agents that find the sources and
configure the pipeline. Mostly Right treats a successful run as one readable dataset version;
publishing that version does not by itself prove that catch-up or scheduled refresh is running.

`mostlyright-data` is the command-line client. It probes sources, registers one recipe document,
starts and follows hosted dataset runs, reads what they delivered, and downloads the datasets after
they finish.

## Requirements

Use CPython 3.13.

## Install

```bash
python -m pip install mostlyright-data
```

That is the whole install. There is no second profile and no lane to choose: every `mr-data`
command sends its work to the backend, and it works on Linux, macOS, and Windows.

It also carries the `mr-data-build` agent skill, which Claude Code and Codex read to author a
recipe without rediscovering the contract a run at a time. The first `mr-data` command you run
places it in your agent's skill directory; there is nothing to install by hand. A skill you have
edited yourself is never replaced, and `MOSTLYRIGHT_SKILL_AUTOINSTALL=0` switches the placement
off.

## Sign in

```bash
mr-data login
mr-data whoami
```

`login` stores a device key on the current machine. `whoami` checks that key.

## Check the installation

```bash
mr-data --version
mr-data --help
```

## Build a dataset

Register a recipe document with the sources, table schema, transformations and checks:

```bash
mr-data recipe recipe.json --json
mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full
mr-data status RUN_ID
mr-data checks RUN_ID
mr-data download RUN_ID --output ./out
```

Use the identifiers returned by registration and run submission. `mr-data dataset create`
creates a dataset page before a recipe is ready. `mr-data watch RUN_ID` follows a submitted run.

The hosted engine executes sample, full and refresh runs. A refresh uses each source's declared
update behavior: bounded source windows merge with retained history; snapshot sources are
revalidated or acquired again. The table's transformations and checks run over the resulting
source relations. URL windows may use declared query parameters or path placeholders; a fixed
date URL does not advance automatically. Configure a correction lookback where the publisher
can revise earlier observations.

A source exposing only its current snapshot cannot supply a historical delta. It remains a
supported source, with snapshot comparison and recomputation rather than a claim that only new
rows were fetched. Recorded streams use their registered capture and continuation semantics.

See [Recipe documents](docs/RECIPE-DOCUMENT.md) for source-window and bootstrap contracts.
`mr-data promote TABLE_ID` requests publication; claim a table is live and current only when
Studio returns that evidence.

## There is no local execution

There used to be. A second product lane built datasets on your own machine, and `mr-data` carried
about fifty commands for making, running, serving, indexing, checking and signing a build here.
That lane is gone: the engine it ran now ships only inside the backend's worker images.

`mr-data --help` lists every command this product has. A name that is not on it is not a command
and says so. The distribution still publishes a `[local]` extra — that is the engine's dependency
closure, which the backend's worker image installs — and installing it makes no command run that
does not run without it.

## Documentation

- [CLI guide](https://mostlyright.md/docs/guides/cli/)
- [Hosted research sessions](https://mostlyright.md/docs/guides/hosted-research/)
- [Credentials](https://mostlyright.md/docs/guides/credentials/)
- [Public datasets](https://mostlyright.md/docs/guides/use-public-datasets/)
- [Certified document extraction](docs/DOCUMENT-EXTRACTION.md)

Use `mr-data --help` for the full command list and options. Commands that support
`--json` write one JSON object.

`--output` may use a relative path.

## Behavior and limits

- Weather Reader versions 1 and 2 accept one extracted GRIB2 record and refuse a multi-record GFS
  or HRRR file. Version 3 reads a bounded collection and selects exactly one sealed record.
- A single-address credential-free public HTTPS source uses the reviewed hosted route. Bounded
  collections have no route at all; the retired local lane was their only route.
- A recipe supports up to 256 declared sources. Source count is separate from bounded request concurrency and resource limits. Existing source adapters and Readers retain their own format, pagination, byte, row and credential contracts; a URL collection is not an implicit permission to crawl arbitrary links.
- A keyed stream venue is named by a credential reference the backend holds, never by a value. No command flag takes a secret as an argument. Venues reached over an actual WebSocket are covered; a venue whose credential comes from an interactive human login, or that is not WebSocket at all, is not.

## Verification

```bash
make test-fast
```

`make test-fast` is non-authoritative and is useful while editing. `make verify` is required
before review or merge.

## Development

```bash
uv sync --locked --extra dev --extra local-search-neural
```

The test suite drives the execution engine in-process, and requires Linux for the worker-boundary
tests. The client itself is tested on Linux, macOS, and Windows.
