Metadata-Version: 2.4
Name: arisole-eval
Version: 0.1.5
Summary: Post-process LeRobot evaluation artifacts into Arisole MotionBench reliability comparisons
Author-email: Arisole <move@arisole.com>
License: Apache-2.0
Project-URL: Homepage, https://www.arisole.com/developers/lerobot
Project-URL: Documentation, https://www.arisole.com/developers/lerobot
Project-URL: Repository, https://github.com/stealthg0dd/strideiq
Project-URL: Issues, https://github.com/stealthg0dd/strideiq/issues
Project-URL: ReliabilityIndex, https://www.arisole.com/reliability-index
Project-URL: MotionBench, https://www.arisole.com/motionbench
Keywords: lerobot,robotics,motionbench,reliability,evaluation,arisole
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: jsonschema<5,>=4.22
Requires-Dist: numpy<3,>=1.26
Provides-Extra: ci
Requires-Dist: PyYAML<7,>=6.0.1; extra == "ci"
Provides-Extra: dev
Requires-Dist: pytest<9,>=8.2; extra == "dev"
Requires-Dist: PyYAML<7,>=6.0.1; extra == "dev"
Provides-Extra: pypi
Requires-Dist: build<2,>=1.2; extra == "pypi"
Requires-Dist: twine<7,>=5.1; extra == "pypi"
Dynamic: license-file

# arisole-eval

**Reliability Engineering for Physical AI** — MotionBench Reliability Suite postprocessor for LeRobot evaluation artifacts.

Compare baseline vs candidate checkpoints from `lerobot-eval` outputs (`eval_info.json`).  
Local evaluation needs **no account, no GPU, no robot**. Cloud MotionBench upload is optional.

Apache-2.0 · Not an official LeRobot / Hugging Face partnership.

## Five-minute quickstart

```bash
pip install arisole-eval==0.1.5
arisole quickstart --out-dir ./arisole-quickstart
```

This materializes an educational CloseFridge **OBSERVED_SIMULATION** sample (n=1, FAIL_TO_FAIL) and runs evaluate → interpretation → gate. Expect **INSUFFICIENT_DATA** — that is the correct scientific outcome for weak-n evidence.

Docs: https://www.arisole.com/developers/lerobot  
Hub: https://huggingface.co/stealthgod/arisole-eval  
PyPI: https://pypi.org/project/arisole-eval/

## Install

Requires Python ≥ 3.11.

```bash
pip install arisole-eval==0.1.5
# YAML reliability configs in CI:
pip install 'arisole-eval[ci]'
```

## Bring your own LeRobot runs

Minimal local compare (no account / GPU / robot). Each directory needs `eval_info.json`.

```bash
arisole evaluate --baseline ./runs/v1 --candidate ./runs/v2 --out result.json
arisole report result.json
arisole gate result.json --config .arisole/reliability.yml
```

**Identity (task / environment):** when every `per_task[].task_group` in a run is the same value (typical PushT / LIBERO suite exports), arisole-eval infers `task` and `environment` from that observed field. It never invents checkpoint IDs or episode seeds. If identity is still missing (heterogeneous task groups, non-standard layouts), pass flags or write `run_meta.json`:

```bash
arisole evaluate --baseline ./runs/v1 --candidate ./runs/v2 \
  --task libero_10 --environment libero_10 \
  --baseline-checkpoint ck20 --candidate-checkpoint ck60 \
  --out result.json
```

Optional cloud:

```bash
export ARISOLE_API_URL=...
export ARISOLE_API_KEY=...
arisole upload result.json
```

## LeRobot input layout

```text
runs/v1/
├── eval_info.json      # required
├── run_meta.json       # optional when task/env can be inferred or passed via CLI
└── telemetry.npz       # optional
```

Success labels come only from `eval_info.json`. Missing success → `insufficient_data`. Labels, task/env/checkpoint identity, and episode pairing are never invented. Multi-task metric lists without seeds stay in **unpaired** mode.

## Telemetry (off by default)

Anonymous CLI usage events are **disabled** unless you pass `--telemetry-consent` **and** set `ARISOLE_TELEMETRY=1`. No eval artifacts, paths, or secrets are sent.

## Metrics (Suite v0.1 / package 0.1.5)

| Metric | Source |
| --- | --- |
| Task success rates / delta | Episode `success` / multi-task `per_task[].metrics.successes` when present |
| Checkpoint regression (P2F / F2P / …) | **Seed-matched pairs only** (demonstrable episode identity) |
| Unpaired rates | When seeds are absent (common multi-task `lerobot-eval` output): binomial rates + SE; no fake pairs |
| Gate | Evidence uses **fully labelled** pairs only; missing labels / weak n → `INSUFFICIENT_DATA` (CI exit 2 by default) |
| Repeatability | Episode reward std |
| Trajectory / action deviation | Optional NPZ channels; else `insufficient_data` |

See `docs/arisole-eval-success-vs-reliability.md` in the monorepo.

## Tests

Gate YAML fixtures used by the suite ship inside the sdist at `tests/fixtures/configs/` (and as package data under `arisole_eval/data/configs/`). Paths resolve from the test package, not the process working directory or a private monorepo checkout.

```bash
pip install -e '.[dev]'
pytest -q
```
