Metadata-Version: 2.3
Name: gauntlet-robotics
Version: 0.4.0
Summary: An evaluation harness for learned robot policies.
Keywords: robotics,evaluation,policy,mujoco,regression-testing
Author: Muhammad Hussain Ahmad
License: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: mujoco>=3.2,<5
Requires-Dist: gymnasium>=1.0,<2
Requires-Dist: pydantic>=2.7,<3
Requires-Dist: typer>=0.12,<1
Requires-Dist: numpy>=1.26,<3
Requires-Dist: pandas>=2.2,<4
Requires-Dist: jinja2>=3.1,<4
Requires-Dist: pyyaml>=6.0,<7
Requires-Dist: rich>=13.7,<17
Requires-Dist: transformers>=4.40,<5 ; extra == 'dt'
Requires-Dist: torch>=2.2,<3 ; extra == 'dt'
Requires-Dist: huggingface-hub>=0.20,<1 ; extra == 'dt'
Requires-Dist: genesis-world>=0.4,<0.5 ; extra == 'genesis'
Requires-Dist: torch>=2.2,<3 ; extra == 'genesis'
Requires-Dist: lerobot>=0.4,<1 ; extra == 'groot'
Requires-Dist: transformers>=4.40,<5 ; extra == 'groot'
Requires-Dist: pillow>=10.0,<12 ; extra == 'groot'
Requires-Dist: torch>=2.2,<3 ; extra == 'hf'
Requires-Dist: transformers>=4.40,<5 ; extra == 'hf'
Requires-Dist: timm>=0.9.10,<2 ; extra == 'hf'
Requires-Dist: tokenizers>=0.19,<1 ; extra == 'hf'
Requires-Dist: pillow>=10.0,<12 ; extra == 'hf'
Requires-Dist: isaacsim>=5.0,<6 ; extra == 'isaac'
Requires-Dist: lerobot[smolvla]>=0.4,<1 ; extra == 'lerobot'
Requires-Dist: transformers>=4.40,<5 ; extra == 'lerobot'
Requires-Dist: pillow>=10.0,<12 ; extra == 'lerobot'
Requires-Dist: mlflow>=2.10,<4 ; extra == 'mlflow'
Requires-Dist: torch>=2.2,<3 ; extra == 'monitor'
Requires-Dist: pillow>=10.0,<12 ; extra == 'monitor'
Requires-Dist: pyarrow>=15.0,<22 ; extra == 'parquet'
Requires-Dist: lerobot[pi]>=0.4,<1 ; extra == 'pi0'
Requires-Dist: transformers>=4.40,<5 ; extra == 'pi0'
Requires-Dist: pillow>=10.0,<12 ; extra == 'pi0'
Requires-Dist: pybullet>=3.2,<4 ; extra == 'pybullet'
Requires-Dist: torch>=2.2,<3 ; extra == 'rdt'
Requires-Dist: transformers>=4.40,<5 ; extra == 'rdt'
Requires-Dist: huggingface-hub>=0.20,<1 ; extra == 'rdt'
Requires-Dist: pillow>=10.0,<12 ; extra == 'rdt'
Requires-Dist: torch>=2.0,<3 ; extra == 'realsim-gsplat'
Requires-Dist: pillow>=10.0,<12 ; extra == 'realsim-gsplat'
Requires-Dist: gsplat>=1.0,<2 ; extra == 'realsim-gsplat'
Requires-Dist: dtw-python>=1.4,<2 ; extra == 'trajectory-taxonomy'
Requires-Dist: imageio[ffmpeg]>=2.34,<3 ; extra == 'video'
Requires-Dist: wandb>=0.16,<1 ; extra == 'wandb'
Requires-Python: >=3.11
Project-URL: Changelog, https://github.com/mhussainahmad/gauntlet/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/mhussainahmad/gauntlet/blob/main/README.md
Project-URL: Homepage, https://github.com/mhussainahmad/gauntlet
Project-URL: Issues, https://github.com/mhussainahmad/gauntlet/issues
Project-URL: Repository, https://github.com/mhussainahmad/gauntlet
Provides-Extra: dt
Provides-Extra: genesis
Provides-Extra: groot
Provides-Extra: hf
Provides-Extra: isaac
Provides-Extra: lerobot
Provides-Extra: mlflow
Provides-Extra: monitor
Provides-Extra: parquet
Provides-Extra: pi0
Provides-Extra: pybullet
Provides-Extra: rdt
Provides-Extra: realsim-gsplat
Provides-Extra: ros2
Provides-Extra: trajectory-taxonomy
Provides-Extra: video
Provides-Extra: wandb
Description-Content-Type: text/markdown

# Gauntlet

[![CI](https://github.com/mhussainahmad/gauntlet/actions/workflows/ci.yml/badge.svg)](https://github.com/mhussainahmad/gauntlet/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/gauntlet-robotics.svg)](https://pypi.org/project/gauntlet-robotics/)
[![Python](https://img.shields.io/pypi/pyversions/gauntlet-robotics.svg)](https://pypi.org/project/gauntlet-robotics/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/mhussainahmad/gauntlet/blob/main/LICENSE)

**Regression testing and failure analysis for learned robot policies.**

Gauntlet answers one question for VLA, diffusion, and scripted policies:

> *How does this policy fail, and has the new checkpoint regressed against the last one?*

It wraps any policy behind a small adapter, runs it across a seeded grid
of simulator perturbations (lighting, camera pose, clutter, object pose,
actuation latency, sensor corruption, instruction paraphrase), and
writes a report that **breaks failures down by condition** instead of
averaging them away.

**55-second overview** (real rollouts, report and CLI output):

https://github.com/user-attachments/assets/7a55c70a-6c0d-42f4-8004-991032c3ac87

![Gauntlet report: failure clusters, per-axis sensitivity and success rates](https://raw.githubusercontent.com/mhussainahmad/gauntlet/main/docs/assets/report-field-conditions.png)

<sub>Report from the [field-conditions example](https://github.com/mhussainahmad/gauntlet/blob/main/docs/field-robustness.md):
the baseline is perfect up to 100 ms of control latency and falls to
47% at 200 ms. Every failure cluster is a latency cluster.</sub>

## Why

A success rate is a mean, and a mean hides the thing you care about.
Two checkpoints at 80% can fail in completely different places, and a
fine-tune that gains two points on the clean scene can lose its entire
safety margin on one axis. Gauntlet makes the per-condition picture the
default output:

- **Failure clusters first.** Axis combinations whose failure rate is
  well above baseline, with Wilson CIs, ranked by lift.
- **Paired comparisons.** `gauntlet compare` / `gauntlet diff` line two
  runs up cell by cell (common random numbers + McNemar when seeds
  match) and flag regressions, not just a headline delta.
- **Sensitivity indices.** Per-axis first- and total-order Sobol
  indices, corrected for rollout seed noise.
- **Reproducible.** Every episode is determined by
  `(suite, cell, seed)`; `gauntlet replay` re-simulates any one
  bit-for-bit, optionally with one axis nudged.

## See a real report

Every [GitHub Release](https://github.com/mhussainahmad/gauntlet/releases/latest)
ships `reference-benchmark.zip`: two policies (a closed-loop controller
and a degraded copy of it) on the smoke suite and the
[field-conditions suite](https://github.com/mhussainahmad/gauntlet/blob/main/docs/field-robustness.md), with the
`compare` / `diff` deltas. Unzip and open any `report.html`.

## Install

```bash
pip install gauntlet-robotics            # core: MuJoCo, torch-free
pip install 'gauntlet-robotics[hf]'      # + OpenVLA / HuggingFace adapter
pip install 'gauntlet-robotics[lerobot]' # + SmolVLA / pi0 / diffusion adapters
```

Other extras: `pybullet`, `genesis`, `isaac` (backends), `monitor`
(drift detection), `video`, `ros2`. Python ≥ 3.11.

## Quickstart

```bash
git clone https://github.com/mhussainahmad/gauntlet && cd gauntlet
uv sync

# 1. One policy, one suite: 24 rollouts, a few seconds.
uv run gauntlet run examples/suites/tabletop-smoke.yaml --policy random --out out/smoke
xdg-open out/smoke/report.html        # macOS: open

# 2. Baseline vs. regressed checkpoint on field-style conditions: 2 x 720 rollouts, ~15 s.
uv run python scripts/generate_reference_benchmark.py \
  --suite examples/suites/tabletop-field-conditions.yaml --out out/field
cat out/field/diff.txt
```

Each run writes `episodes.json` (one record per rollout), `report.json`
(the analysis) and a self-contained `report.html`.

### Bring your own policy

A policy is anything with `act(obs) -> action`:

```python
# my_policy.py
import numpy as np

class MyPolicy:
    def __init__(self) -> None:
        self.model = load_my_checkpoint()          # your code

    def act(self, obs: dict[str, np.ndarray]) -> np.ndarray:
        # obs: ee_pos, cube_pos, target_pos, ... (+ "image" with render_in_obs=True)
        return self.model(obs["ee_pos"], obs["cube_pos"])  # 7-D EE twist + gripper

def make_policy() -> MyPolicy:
    return MyPolicy()
```

```bash
uv run gauntlet run examples/suites/tabletop-basic-v1.yaml \
  --policy my_policy:make_policy --out out/mine
uv run gauntlet compare out/last_week/episodes.json out/mine/episodes.json
```

Third-party packages can also register policies and envs through entry
points; see [`docs/plugin-development.md`](https://github.com/mhussainahmad/gauntlet/blob/main/docs/plugin-development.md).

## What's in the box

| | |
|---|---|
| **Tasks** | Tabletop pick-and-place (`tabletop`), planar pushing (`tabletop-push`: no grasp, success only once the cube settles in the target), three-cube stacking with per-subtask credit (`tabletop-stack`) and a mobile-base variant (`tabletop-mobile`). |
| **Backends** | MuJoCo (core; every task), PyBullet (pick-and-place, state and images) and Genesis (pick-and-place; state tested in CI, image rendering not exercised in CI). Isaac Sim adapter written against the Isaac Sim 5.0 API, CI-tested against a mocked runtime but **not yet run on hardware**. Same action/observation spaces; `compare` refuses cross-backend diffs unless asked. |
| **Perturbation axes** | Lighting, camera offset and full extrinsics, object texture / pose / class swap, distractors, OOD initial state, actuation latency, image corruption, colour shift, instruction paraphrase. All of them run straight from a suite YAML. |
| **Sampling** | Cartesian grids, Latin hypercube, Sobol, worst-case search; `gauntlet suite plan` sizes episodes-per-cell for a target effect. |
| **Policy adapters** | Random, scripted pick-and-place and push controllers, OpenVLA (HF), SmolVLA / pi0 / diffusion (LeRobot), GR00T, RDT, Decision Transformer. |
| **Analysis** | Failure clusters, Wilson CIs, paired compare, per-cell diff, Sobol indices, behavioural metrics (time, path length, jerk), safety counters, `gauntlet bisect` across checkpoints. Non-finite observations or actions end the episode as a flagged failure instead of silently corrupting it. |
| **Operations** | Rollout caching, MP4 recording, runtime drift detection, ROS 2 publish/record, fleet aggregation, static dashboard, real-to-sim scene ingestion with a nearest-frame renderer and a smoke-test gaussian-splat fit (gsplat CUDA or pure PyTorch). |

Details for each are in the **[user guide](https://github.com/mhussainahmad/gauntlet/blob/main/docs/guide.md)**. Design
decisions are recorded as RFCs and design notes under [`docs/`](https://github.com/mhussainahmad/gauntlet/tree/main/docs/).

## Stability

`0.4.x` is on PyPI as `gauntlet-robotics`. The public API, on-disk
schemas and CLI flags follow [Semantic Versioning](https://semver.org/);
the contract is in [`docs/stability.md`](https://github.com/mhussainahmad/gauntlet/blob/main/docs/stability.md). Pin
`gauntlet-robotics>=0.4,<0.5`.

## Development

```bash
uv sync
uv run ruff check . && uv run ruff format --check .
uv run mypy                 # --strict
uv run pytest               # ~1,800 torch-free tests; extras run in their own CI jobs
```

See [`CONTRIBUTING.md`](https://github.com/mhussainahmad/gauntlet/blob/main/CONTRIBUTING.md) and the
[property-test notes](https://github.com/mhussainahmad/gauntlet/blob/main/docs/guide.md#property-tests).

## Project layout

```
src/gauntlet/
  policy/      # Policy adapter protocol + reference wrappers (Random, Scripted, HF, LeRobot)
  env/         # Parameterized envs — MuJoCo (core) + PyBullet/Genesis/Isaac (extras)
  suite/       # YAML-defined perturbation grid suites (cartesian / LHS / Sobol)
  runner/      # Parallel rollout orchestration + seed management + cache
  report/      # Per-run failure analysis + HTML/JSON generation
  monitor/     # Runtime drift detection + action-entropy ([monitor] extra)
  replay/      # Single-episode replay with axis overrides
  ros2/        # ROS 2 publisher + recorder ([ros2] extra; rclpy via apt/Docker)
  diff/        # Structured per-axis report deltas powering `gauntlet diff`
  aggregate/   # Fleet-wide failure-mode clustering across many runs
  dashboard/   # Self-contained static SPA indexing every report.json
  realsim/     # Real-to-sim scene ingestion + renderers (nearest-frame, gaussian splat)
  plugins.py   # Entry-point discovery for third-party policies / envs
  cli.py       # gauntlet run / report / compare / diff / aggregate /
               # dashboard / realsim / monitor / replay / ros2
```

## License

MIT, see [LICENSE](https://github.com/mhussainahmad/gauntlet/blob/main/LICENSE).
