Metadata-Version: 2.4
Name: arisole-eval
Version: 0.1.3
Summary: Post-process LeRobot evaluation artifacts into Arisole MotionBench reliability comparisons
Author-email: Arisole <move@arisole.com>
License: Apache-2.0
Project-URL: Homepage, https://www.arisole.com/developers/lerobot
Project-URL: Documentation, https://www.arisole.com/developers/lerobot
Project-URL: Repository, https://github.com/stealthg0dd/strideiq
Project-URL: Issues, https://github.com/stealthg0dd/strideiq/issues
Project-URL: ReliabilityIndex, https://www.arisole.com/reliability-index
Project-URL: MotionBench, https://www.arisole.com/motionbench
Keywords: lerobot,robotics,motionbench,reliability,evaluation,arisole
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: jsonschema<5,>=4.22
Requires-Dist: numpy<3,>=1.26
Provides-Extra: ci
Requires-Dist: PyYAML<7,>=6.0.1; extra == "ci"
Provides-Extra: dev
Requires-Dist: pytest<9,>=8.2; extra == "dev"
Requires-Dist: PyYAML<7,>=6.0.1; extra == "dev"
Provides-Extra: pypi
Requires-Dist: build<2,>=1.2; extra == "pypi"
Requires-Dist: twine<7,>=5.1; extra == "pypi"
Dynamic: license-file

# arisole-eval

**Reliability Engineering for Physical AI** — MotionBench Reliability Suite postprocessor for LeRobot evaluation artifacts.

Compare baseline vs candidate checkpoints from `lerobot-eval` outputs (`eval_info.json`).  
Local evaluation needs **no account, no GPU, no robot**. Cloud MotionBench upload is optional.

Apache-2.0 · Not an official LeRobot / Hugging Face partnership.

## Five-minute quickstart

```bash
pip install arisole-eval==0.1.3
arisole quickstart --out-dir ./arisole-quickstart
```

This materializes an educational CloseFridge **OBSERVED_SIMULATION** sample (n=1, FAIL_TO_FAIL) and runs evaluate → interpretation → gate. Expect **INSUFFICIENT_DATA** — that is the correct scientific outcome for weak-n evidence.

Docs: https://www.arisole.com/developers/lerobot  
Hub: https://huggingface.co/stealthgod/arisole-eval  
PyPI: https://pypi.org/project/arisole-eval/

## Install

Requires Python ≥ 3.11.

```bash
pip install arisole-eval==0.1.3
# YAML reliability configs in CI:
pip install 'arisole-eval[ci]'
```

## Bring your own LeRobot runs

```bash
arisole evaluate --baseline ./runs/v1 --candidate ./runs/v2 --out result.json
arisole report result.json
arisole gate result.json --config .arisole/reliability.yml
```

Optional cloud:

```bash
export ARISOLE_API_URL=...
export ARISOLE_API_KEY=...
arisole upload result.json
```

## LeRobot input layout

```text
runs/v1/
├── eval_info.json      # required
├── run_meta.json       # recommended
└── telemetry.npz       # optional
```

Success labels come only from `eval_info.json`. Missing success → `insufficient_data`. Labels are never invented.

## Telemetry (off by default)

Anonymous CLI usage events are **disabled** unless you pass `--telemetry-consent` **and** set `ARISOLE_TELEMETRY=1`. No eval artifacts, paths, or secrets are sent.

## Metrics (Suite v0.1 / package 0.1.3)

| Metric | Source |
| --- | --- |
| Task success rates / delta | Episode `success` / multi-task `per_task[].metrics.successes` when present |
| Checkpoint regression (P2F / F2P / …) | **Seed-matched pairs only** (demonstrable episode identity) |
| Unpaired rates | When seeds are absent (common multi-task `lerobot-eval` output): binomial rates + SE; no fake pairs |
| Gate | Evidence uses **fully labelled** pairs only; missing labels / weak n → `INSUFFICIENT_DATA` (CI exit 2 by default) |
| Repeatability | Episode reward std |
| Trajectory / action deviation | Optional NPZ channels; else `insufficient_data` |

See `docs/arisole-eval-success-vs-reliability.md` in the monorepo.

## Tests

```bash
pip install -e '.[dev]'
pytest -q
```
