Metadata-Version: 2.4
Name: mostlyright-data
Version: 0.21.2
Summary: Mostly Right hosted CLI for reviewed datasets
Project-URL: Homepage, https://mostlyright.md/
Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/
Requires-Python: >=3.11
Provides-Extra: build
Requires-Dist: hatchling==1.28.0; extra == 'build'
Requires-Dist: uv-build==0.11.3; extra == 'build'
Description-Content-Type: text/markdown

# mostlyright-data

Browse datasets built by our community or create your own with agents that find the sources and
configure the pipeline. Mostly Right treats a successful run as one readable dataset version;
publishing that version does not by itself prove that catch-up or scheduled refresh is running.

`mostlyright-data` is the command-line client. It probes sources, registers one recipe document,
starts and follows hosted dataset runs, reads what they delivered, and downloads the datasets after
they finish.

## Requirements

Use CPython 3.11 or newer.

## Install

```bash
python -m pip install mostlyright-data
```

That is the whole install. There is no second profile and no lane to choose: every `mr-data`
command sends its work to the backend, and it works on Linux, macOS, and Windows.

It also carries the `mr-data-build` agent skill, which Claude Code and Codex read to author a
recipe without rediscovering the contract a run at a time. The first `mr-data` command you run
places it in your agent's skill directory; there is nothing to install by hand. A skill you have
edited yourself is never replaced, and `MOSTLYRIGHT_SKILL_AUTOINSTALL=0` switches the placement
off.

## Sign in

```bash
mr-data login
mr-data whoami
```

`login` stores a device key on the current machine. `whoami` checks that key.

New macOS/Linux logins use `~/.mostlyright/credentials.secure`, an unencrypted file restricted to
your OS account (0600 inside a 0700 directory). Processes running as you and backups can read it.
Older file-based logins migrate to this file without requesting Keychain access. Windows keeps
DPAPI encryption as its default.
Existing Keychain, DPAPI, or Secret Service logins keep their selected store; an unavailable store
never causes an automatic switch to file storage.

If an existing macOS login requests Keychain access, it is requesting the `md.mostlyright.cli`
device credential. To switch that login to file storage, restore access to its Keychain item,
run `mr-data auth logout` successfully, then run `mr-data login`. The `--credential-store secure-file`
option does not move an existing native-store login.

## Check the installation

```bash
mr-data --version
mr-data --help
```

## Build a dataset

Create the dataset, choose its primary category from the fixed vocabulary, then register a recipe document with the sources, table schema, transformations and checks. Use the created dataset ID in the recipe:

```bash
mr-data dataset create --name "Dataset title" --json
mr-data dataset categories --json
mr-data dataset set DATASET_ID --category climate-environment
mr-data recipe recipe.json --json
mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full
mr-data status RUN_ID
mr-data checks RUN_ID
mr-data download RUN_ID --output ./out
```

Use the identifiers returned by registration and run submission. `mr-data dataset create`
creates a dataset page before a recipe is ready. `mr-data watch RUN_ID` follows a submitted run.

To request an offline replay of retained inputs with a registered revision:

```bash
mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --mode replay --sources-from RUN_ID
```

Studio must have replay enabled and the named successful run must belong to the same table with
its raw inputs still retained. Replay compares against that run and never becomes the live version.
Studio returns a typed refusal when replay is unavailable; the CLI does not fetch sources locally.

The hosted engine executes sample, full and refresh runs. A refresh uses each source's declared
update behavior: bounded source windows merge with retained history; snapshot sources are
revalidated or acquired again. The table's transformations and checks run over the resulting
source relations. URL windows may use declared query parameters or path placeholders; a fixed
date URL does not advance automatically. Configure a correction lookback where the publisher
can revise earlier observations.

A source exposing only its current snapshot cannot supply a historical delta. It remains a
supported source, with snapshot comparison and recomputation rather than a claim that only new
rows were fetched. Recorded streams use their registered capture and continuation semantics.

See [Recipe documents](docs/RECIPE-DOCUMENT.md) for source-window and bootstrap contracts.
A table's first succeeded run goes live on its own, whatever mode it was; `mr-data promote
TABLE_ID` records how often it refreshes and why, and puts back a table that was withdrawn. Claim
a table is live and current only when Studio returns that evidence.

## There is no local execution

There used to be. A second product lane built datasets on your own machine, and `mr-data` carried
about fifty commands for making, running, serving, indexing, checking and signing a build here.
That lane is gone: the engine it ran now ships only inside the backend's worker images.

`mr-data --help` lists every command this product has. The execution engine and its dependency
closure live in Mostly Right Studio's `apps/worker`; this repository ships no engine extras,
worker executables or image publisher.

## Documentation

- [Install and sign in](https://mostlyright.md/docs/start/install/)
- [Build your first dataset](https://mostlyright.md/docs/start/first-dataset/)
- [CLI reference](https://mostlyright.md/docs/reference/cli/)
- [Recipe reference](https://mostlyright.md/docs/reference/recipe/)
- [Recipe examples](https://mostlyright.md/docs/recipes/)
- [Join market settlements](https://mostlyright.md/docs/recipes/market-settlement-join/)
- [Compare a forecast with observations](https://mostlyright.md/docs/recipes/forecast-vs-observation/)
- [Replace a snapshot on refresh](https://mostlyright.md/docs/recipes/snapshot-window/)
- [Union many weather stations](https://mostlyright.md/docs/recipes/many-station-weather/)
- [Aggregate a stream into bars](https://mostlyright.md/docs/recipes/stream-to-bars/)
- [Use a public dataset](https://mostlyright.md/docs/guides/use-public-datasets/)
- [Certified document extraction](docs/DOCUMENT-EXTRACTION.md)
- [Use a table to drive a stream's market roster](docs/TABLE-DRIVEN-DISCOVERY.md)

Use `mr-data --help` for the full command list and options. Commands that support
`--json` write one JSON object.

`--output` may use a relative path.

## Behavior and limits

- Weather Reader versions 1 and 2 accept one extracted GRIB2 record and refuse a multi-record GFS
  or HRRR file. Version 3 reads a bounded collection and selects exactly one sealed record.
- A single-address credential-free public HTTPS source uses the reviewed hosted route. Bounded
  collections have no route at all; the retired local lane was their only route.
- A recipe supports up to 256 declared sources. Source count is separate from bounded request concurrency and resource limits. Existing source adapters and Readers retain their own format, pagination, byte, row and credential contracts; a URL collection is not an implicit permission to crawl arbitrary links.
- A keyed stream venue is named by a credential reference the backend holds, never by a value. No command flag takes a secret as an argument. Venues reached over an actual WebSocket are covered; a venue whose credential comes from an interactive human login, or that is not WebSocket at all, is not.

## Verification

```bash
make test
```

`make test` runs the thin client suite while editing. `make verify` is required
before review or merge.

## Development

```bash
uv sync --locked
```

There is no `dev` extra to ask for -- `uv run --extra dev` answers
``error: Extra `dev` is not defined in the project's `optional-dependencies` table``. The test
tooling is the default dependency group, which `uv sync` installs.

`make lint` is the repository's one lint set (`uv run ruff check .` plus `uv lock --check`); it is
the first thing `make verify` runs and the same target the Linux leg of `thin-client-smoke.yml`
runs, so the two cannot disagree about what is linted.

The test suite checks client behavior, contract compatibility and thin package ownership. Backend
execution and worker-boundary tests run in Studio. The client supports Linux, macOS and Windows.

To update the public Studio contract pin, run `make repin STUDIO_GIT=/path/to/studio STUDIO_REF=<commit>`. It reads committed OpenAPI and JSON Schema bytes, records their hashes and source commit in `vendor/studio-contracts/client-v4-pin.json`, and updates the thin client. Run the cross-repository tests against that Studio revision before release. Worker/client generation and backend job-contract repins now belong to Studio.
