Metadata-Version: 2.5
Name: openral-dataset
Version: 0.4.0
Summary: OpenRAL dataset bridge — RolloutRecorder + LeRobotDataset v3 / rosbag2 sinks for every skill execution
License: Apache-2.0
Requires-Python: <3.13,>=3.12
Requires-Dist: mcap>=1.2
Requires-Dist: numpy>=1.26
Requires-Dist: openral-core==0.4.0
Requires-Dist: openral-observability==0.4.0
Requires-Dist: structlog>=24.1
Provides-Extra: lerobot
Requires-Dist: lerobot[dataset]>=0.6.1; extra == 'lerobot'
Requires-Dist: pyarrow>=17; extra == 'lerobot'
Provides-Extra: ros
Requires-Dist: mcap-ros2-support>=0.5; extra == 'ros'
Requires-Dist: mcap>=1.2; extra == 'ros'
Description-Content-Type: text/markdown

# openral-dataset

rosbag2 ↔ LeRobotDataset v3 bridge. Every skill execution (sim or hardware)
becomes a row in a LeRobotDataset v3.0 (`codebase_version="3.0"`,
`lerobot[dataset]>=0.6.0`). Successful and failed episodes are both persisted;
consumers filter on the per-row `next.success` flag and the per-dataset
`meta/info.json["metadata"]["dataset_success_rate"]`.

## Public API

```python
from openral_core import RobotDescription
from openral_dataset import LeRobotDatasetSink, RolloutRecorder

robot = RobotDescription.from_yaml("robots/so100_follower/robot.yaml")
sink = LeRobotDatasetSink(
    root="/tmp/ds",
    robot=robot,
    fps=30.0,
    repo_id="openral/dataset-pick-cube",
    license="CC-BY-4.0",
)
with RolloutRecorder(
    robot=robot,
    task_string="pick the cube",
    fps=30.0,
    sinks=[sink],
    repo_id="openral/dataset-pick-cube",
) as rec:
    rec.episode_start()
    # for each tick:
    rec.record_frame(
        observation_state=state_vec,    # (state_dim,) float32
        images={"camera1": rgb_frame},  # camera_key → (H, W, 3) uint8
        action=action_vec,              # (action_dim,) float32
        reward=0.0,
        terminated=False,
        truncated=False,
    )
    rec.episode_end(success=True)
```

## Components

- **`RolloutRecorder`** — in-memory per-rollout accumulator with multi-sink
  fan-out. Writes the `openral.dataset.repo_id` / `episode_idx` / `frame_idx`
  OTel attributes on the active `rskill.tick` span so the Jaeger trace can be
  joined to the on-disk frame.
- **`DatasetSink` Protocol** — every sink (online / offline) implements
  `open_episode` / `write_frame` / `close_episode` / `finalize`.
- **`LeRobotDatasetSink`** — wraps the real
  `lerobot.datasets.LeRobotDataset.create / add_frame / save_episode /
  finalize`. Lazy-imports lerobot; raises `ROSConfigError` with an install
  hint when lerobot is absent.
- **`features_from_robot`** — pure `RobotDescription` → LeRobot v3 features
  dict mapping. No I/O, no lerobot import.

Dataset-bridge PR series status:

- **PR2** *(partial)* — `SensorRosPublisher` in `python/sensors/`
  (`ros_publisher.py`) ships and is tested; the wrapping
  `packages/openral_sensors_ros/` lifecycle node from the dataset-bridge design
  is not built. The deploy recorder does not need it: `DatasetRecorderBridge`
  reads camera frames from the aggregator snapshot the runner feeds the policy.
- **PR3** ✅ — `Rosbag2Sink` (mcap, daemon writer thread) + `openral_msgs/Tick`
  / `openral_msgs/Episode` IDLs + explicit `episode_start` / `episode_end`
  API on `DeployRunner` (`bag.py`).
- **PR4** ✅ — `Rosbag2ToLeRobotConverter.from_bag` + `openral dataset from-bag`
  CLI subcommand (`converter.py`).
- **PR5** ✅ — `openral dataset push` with consent prompt + `_hf_publish` shared
  helper de-duped from `tools/rskill_publisher.py`.

## CLI surface

**Sim path** — `openral sim run` accepts three flags (PR1):

- `--dataset-out PATH` — write a LeRobotDataset v3 to `PATH` as the sim runs.
  Path must not pre-exist; lerobot v3 refuses to write into a populated root.
- `--dataset-repo-id STR` — repo id stored in `meta/info.json`. Defaults to
  `openral/dataset-<robot_id>`. Not pushed by `openral sim run`;
  `openral dataset push` owns publishing.
- `--dataset-license SPDX` — defaults to `CC-BY-4.0` (the LeRobot
  convention). PII-bearing datasets must set a more restrictive license; the
  `openral dataset push` consent prompt enforces this.

**Hardware / rosbag path** — record on a live ROS 2 graph, then convert
offline:

- `openral record --profile slim|full --out DIR` — wrap `ros2 bag record`
  with curated topic profiles (mcap storage). `--dry-run` prints the composed
  argv without a sourced ROS 2 install.
- `openral dataset from-bag BAG --robot robot.yaml --output DS` — replay a
  `Rosbag2Sink` mcap into a LeRobotDataset v3 (`--repo-id` / `--license` /
  `--fps` optional). PR4.
- `openral dataset push DS [--repo-id ID] [--yes] [--dry-run]` — consent-gated
  upload of a local dataset root to the HF Hub. PR5.

**Deploy path** — `openral deploy sim|run --dataset-out PATH` attaches a
`DatasetRecorderBridge` to the live graph (`compose_runtime`), writing the same
`Rosbag2Sink` mcap for offline `from-bag` conversion. Note:

- `PATH` is a single **`.mcap` file**, not a rosbag2 bag *directory*. Its parent
  directory must already exist and the file must not (mcap refuses to
  overwrite); either violation fails the launch at compose time.
- Recording is segmented by `/openral/episode` PHASE_START/PHASE_END markers,
  which only an **executing** rSkill publishes. A session where every dispatch
  is rejected at goal acceptance opens no episode and writes no file. The
  bridge logs `dataset_recorder.armed` at startup and, at shutdown, either
  `dataset_recorder.summary` (episode + frame counts) or
  `dataset_recorder.nothing_recorded`, so the zero case is never silent.

## Verification

```bash
uv run pytest python/dataset/tests/ -v
uv run mypy --strict -p openral_dataset
uv run ruff check python/dataset/
```

End-to-end against a real `lerobot.datasets.LeRobotDataset` v3.0 round-trip:

```bash
uv run pytest python/dataset/tests/test_sink_lerobot.py -v
```

Per CLAUDE.md §1.11 (no mocks), every test loads the real SO-100
`RobotDescription` (`robots/so100_follower/robot.yaml`) and the real
`lerobot.datasets.LeRobotDataset` writer, and `pytest.skip`s with a typed reason
on hosts without `lerobot[dataset]>=0.6.0` (installed by
`just sync --group dataset`).

### Coverage scope

- **Sim path** is covered end-to-end: `tests/sim/test_dataset_emission.py`
  drives `openral sim run --dataset-out` against `aloha_transfer_cube.yaml`
  with real ACT weights + gym-aloha physics + SVT-AV1 video encoding, then
  re-opens the produced v3 dataset (CUDA + `lerobot` + `gym_aloha` gated; skips
  on CPU/CI).
- **Hardware / rosbag path**: `tests/unit/test_deploy_runner_dataset_recording.py`
  drives `DeployRunner` + `Rosbag2Sink` directly, plus the converter / CLI unit
  tests. The deploy graph's own recorder — `DatasetRecorderBridge`, attached by
  `compose_runtime(dataset_out=...)` — is covered against a real aggregator,
  real `Rosbag2Sink` and real bag read-back in
  `tests/integration/test_dataset_recorder_bridge.py` (including the
  nothing-recorded reporting path), but not yet through a full live
  `ros2 launch` graph.
