Metadata-Version: 2.4
Name: mostlyright-data
Version: 0.24.0
Summary: Mostly Right hosted CLI for reviewed datasets
Project-URL: Homepage, https://mostlyright.md/
Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/
Requires-Python: >=3.11
Provides-Extra: build
Requires-Dist: hatchling==1.28.0; extra == 'build'
Requires-Dist: uv-build==0.11.3; extra == 'build'
Description-Content-Type: text/markdown

# mostlyright-data

Browse datasets built by our community or create your own with agents that find the sources and
configure the pipeline. Mostly Right treats a successful run as one readable dataset version;
publishing that version does not by itself prove that catch-up or scheduled refresh is running.

`mostlyright-data` is the command-line client. It probes sources, registers one recipe document,
starts and follows hosted dataset runs, reads what they delivered, and downloads the datasets after
they finish.

## Requirements

Use CPython 3.11 or newer.

## Install

```bash
python -m pip install mostlyright-data
```

That is the whole install. There is no second profile and no lane to choose: every `mr-data`
command sends its work to the backend, and it works on Linux, macOS, and Windows.

It also carries the `mr-data-build` agent skill, which Claude Code and Codex read to author a
recipe without rediscovering the contract a run at a time. The first `mr-data` command you run
places it in your agent's skill directory; there is nothing to install by hand. A skill you have
edited yourself is never replaced, and `MOSTLYRIGHT_SKILL_AUTOINSTALL=0` switches the placement
off.

## Sign in

```bash
mr-data login
mr-data whoami
```

`login` stores a device key on the current machine. `whoami` checks that key.

New macOS/Linux logins use `~/.mostlyright/credentials.secure`, an unencrypted file restricted to
your OS account (0600 inside a 0700 directory). Processes running as you and backups can read it.
Older file-based logins migrate to this file without requesting Keychain access. Windows keeps
DPAPI encryption as its default.
Existing Keychain, DPAPI, or Secret Service logins keep their selected store; an unavailable store
never causes an automatic switch to file storage.

If an existing macOS login requests Keychain access, it is requesting the `md.mostlyright.cli`
device credential. To switch that login to file storage, restore access to its Keychain item,
run `mr-data auth logout` successfully, then run `mr-data login`. The `--credential-store secure-file`
option does not move an existing native-store login.

## Check the installation

```bash
mr-data --version
mr-data --help
```

## Build a dataset

Create the dataset, choose its primary category from the fixed vocabulary, then register a recipe document with the sources, table schema, transformations and checks. Use the created dataset ID in the recipe:

```bash
mr-data dataset create --name "Dataset title" --json
mr-data dataset categories --json
mr-data dataset set DATASET_ID --category climate-environment
mr-data recipe recipe.json --json
mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full
mr-data status RUN_ID
mr-data checks RUN_ID
mr-data download RUN_ID --output ./out
```

Run submission is idempotent: sending identical recipe, mode and bounds returns the same run,
including a previous failure. The CLI reports a non-queued result as `run_returned` and exits
with code 2 when that run failed. Read its failure before retrying; `mr-data run --retry RUN_ID`
can reissue supported platform execution failures without changing the recipe.

Use the identifiers returned by registration and run submission. `mr-data dataset create`
creates a dataset page before a recipe is ready. `mr-data watch RUN_ID` follows a submitted run.
A scheduled refresh runs only Studio's persisted bounded source-action plan; it never silently
falls back to a mutable whole-source fetch. `mr-data table resync TABLE_ID --request-id UUID`
explicitly requests a full source reread when that is needed. Choose and retain the UUID before
submitting: if the response is lost, repeat that exact command with the same UUID rather than
starting another reread. A persisted spend hold is still an accepted resync: its receipt names
the run, projection and exact `mr-data run --confirm-held RUN_ID` action. If it is a sample-first
preview, confirm that preview first, then release its linked full
only after the preview succeeds. `mr-data recipe readiness` reads Studio's paginated,
immutable-revision inventory before a production-wide schedule sweep: predecessor,
incremental-materialization and differential-proof readiness, together with each source's next
action. It reports Studio's persisted facts; it does not authorize a schedule.

To request an offline replay of retained inputs with a registered revision:

```bash
mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --mode replay --sources-from RUN_ID
```

Studio must have replay enabled and the named successful run must belong to the same table with
its raw inputs still retained. Replay compares against that run and never becomes the live version.
Studio returns a typed refusal when replay is unavailable; the CLI does not fetch sources locally.

The hosted engine executes sample, full and refresh runs. Studio's action classifier can name
closed-source reuse, request-window or collection acquisition, and recorded stream input, but the
current worker has only two executable normal-refresh materializers: an exact single direct
partition request-window plan with a compatible predecessor, and an all-recorded-stream plan.
Closed-only, collection, and mixed plans return `RESYNC_REQUIRED` before acquisition, as does a
mutable snapshot or other unwindowed source. Nothing is revalidated or re-acquired whole as an
implicit fallback. Use `mr-data table resync TABLE_ID --request-id UUID` when a deliberate full
source reread is intended, retaining that UUID through an uncertain response. URL windows may use
declared query parameters or path placeholders; a fixed date URL does not advance automatically.
Configure a correction lookback where the publisher can revise earlier observations.

A source exposing only its current snapshot cannot supply a historical delta. It is supported only
through an explicit table resync; refresh does not turn its whole current result into an
incremental update. Recorded streams use their registered capture and continuation semantics.

See [Recipe documents](docs/RECIPE-DOCUMENT.md) for source-window and bootstrap contracts.
A table's first succeeded run goes live on its own, whatever mode it was; `mr-data promote
TABLE_ID` records how often it refreshes and why, and puts back a table that was withdrawn. Claim
a table is live and current only when Studio returns that evidence.

## There is no local execution

There used to be. A second product lane built datasets on your own machine, and `mr-data` carried
about fifty commands for making, running, serving, indexing, checking and signing a build here.
That lane is gone: the engine it ran now ships only inside the backend's worker images.

`mr-data --help` lists every command this product has. The execution engine and its dependency
closure live in Mostly Right Studio's `apps/worker`; this repository ships no engine extras,
worker executables or image publisher.

## Documentation

- [Install and sign in](https://mostlyright.md/docs/start/install/)
- [Build your first dataset](https://mostlyright.md/docs/start/first-dataset/)
- [CLI reference](https://mostlyright.md/docs/reference/cli/)
- [Recipe reference](https://mostlyright.md/docs/reference/recipe/)
- [Recipe examples](https://mostlyright.md/docs/recipes/)
- [Join market settlements](https://mostlyright.md/docs/recipes/market-settlement-join/)
- [Compare a forecast with observations](https://mostlyright.md/docs/recipes/forecast-vs-observation/)
- [Model a snapshot source](https://mostlyright.md/docs/recipes/snapshot-window/) — normal refresh
  is unavailable until its bounded materializer ships; use explicit table resync today
- [Union many weather stations](https://mostlyright.md/docs/recipes/many-station-weather/)
- [Aggregate a stream into bars](https://mostlyright.md/docs/recipes/stream-to-bars/)
- [Use a public dataset](https://mostlyright.md/docs/guides/use-public-datasets/)
- [Certified document extraction](docs/DOCUMENT-EXTRACTION.md)
- [Use a table to drive a stream's market roster](docs/TABLE-DRIVEN-DISCOVERY.md)

Use `mr-data --help` for the full command list and options. Commands that support
`--json` write one JSON object.

`--output` may use a relative path.

## Behavior and limits

- Weather Reader versions 1 and 2 accept one extracted GRIB2 record and refuse a multi-record GFS
  or HRRR file. Version 3 reads a bounded collection and selects exactly one sealed record.
- A single-address credential-free public HTTPS source uses the reviewed hosted route. Bounded
  collections have no route at all; the retired local lane was their only route.
- A recipe supports up to 256 declared sources. Source count is separate from bounded request concurrency and resource limits. Existing source adapters and Readers retain their own format, pagination, byte, row and credential contracts; a URL collection is not an implicit permission to crawl arbitrary links.
- A keyed stream venue is named by a credential reference the backend holds, never by a value. No command flag takes a secret as an argument. Venues reached over an actual WebSocket are covered; a venue whose credential comes from an interactive human login, or that is not WebSocket at all, is not.

## Verification

```bash
make test
```

`make test` runs the thin client suite while editing. `make verify` is required
before review or merge.

## Development

```bash
uv sync --locked
```

There is no `dev` extra to ask for -- `uv run --extra dev` answers
``error: Extra `dev` is not defined in the project's `optional-dependencies` table``. The test
tooling is the default dependency group, which `uv sync` installs.

`make lint` is the repository's one lint set (`uv run ruff check .` plus `uv lock --check`); it is
the first thing `make verify` runs and the same target the Linux leg of `thin-client-smoke.yml`
runs, so the two cannot disagree about what is linted.

The test suite checks client behavior, contract compatibility and thin package ownership. Backend
execution and worker-boundary tests run in Studio. The client supports Linux, macOS and Windows.

To update the public Studio contract pin, run `make repin STUDIO_GIT=/path/to/studio STUDIO_REF=<commit>`. It reads committed OpenAPI and JSON Schema bytes, records their hashes and source commit in `vendor/studio-contracts/client-v4-pin.json`, and updates the thin client. Run the cross-repository tests against that Studio revision before release. Worker/client generation and backend job-contract repins now belong to Studio.
