Metadata-Version: 2.4
Name: physicalai-mujoco-plugin
Version: 0.4.0
Summary: MuJoCo simulation plugin for PhysicalAI Runtime
Project-URL: Homepage, https://github.com/openvinotoolkit/physicalai
Project-URL: Repository, https://github.com/openvinotoolkit/physicalai
Project-URL: Issues, https://github.com/openvinotoolkit/physicalai/issues
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Requires-Python: >=3.12
Requires-Dist: defusedxml>=0.7.1
Requires-Dist: fastapi>=0.115
Requires-Dist: loguru>=0.7
Requires-Dist: mjviser<0.1,>=0.0.14
Requires-Dist: mujoco-menagerie==2026.9.4
Requires-Dist: mujoco>=3.6.0
Requires-Dist: numpy>=1.24
Requires-Dist: opencv-python-headless>=4.10
Requires-Dist: physicalai-studio-plugin>=0.4.0
Requires-Dist: physicalai[transport]<1.0,>=0.2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: uvicorn>=0.30
Requires-Dist: websockets>=13.0
Provides-Extra: tests
Requires-Dist: httpx>=0.27; extra == 'tests'
Requires-Dist: pytest; extra == 'tests'
Requires-Dist: pytest-asyncio; extra == 'tests'
Description-Content-Type: text/markdown

# Physical AI MuJoCo Plugin

Run single-arm or bimanual SO-101 simulations, and other [MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie) robots, through [PhysicalAI Runtime](https://github.com/openvinotoolkit/physicalai). The plugin provides a browser viewer, camera streams, task scenes, and follower catalog entries for [Physical AI Studio](https://github.com/open-edge-platform/physical-ai-studio).

## Features

- Run a virtual SO-101 as a PhysicalAI transport owner
- Viser browser viewer with a Simulation tab (scene switching, reset, arm homing, seeded resets, episode auto-reset, conveyor belt controls, object dragging, camera previews, confirmed shutdown) and a Camera tab to follow an object, the target, or a gripper
- Camera streams over HTTP (MJPEG), used in Studio as IP cameras
- REST control server with the same controls as the viewer
- Built-in task scenes
- Automatic cube respawn after a configurable success dwell (five seconds by default) in `single_pick_place`
- A conveyor-belt sorting cell (`conveyor_sort`) with an adjustable belt speed, a continuous item feed, and per-episode scoring

## What this is for

- Teleoperating a virtual SO-101 from PhysicalAI Studio.
- Testing robot setup, task logic, and control flows without physical hardware.
- Playing back policy/inference outputs in a simulated scene while monitoring robot + camera streams.
- Developing workflows where Studio, transport, and robot APIs stay identical between sim and real deployments.

## Quick start

Requires Python 3.12 or newer. From the Runtime repository root:

```bash
git lfs pull
uv sync --package physicalai-mujoco-plugin
uv run --package physicalai-mujoco-plugin physicalai-mujoco start
```

On a headless Linux machine (neither `DISPLAY` nor `WAYLAND_DISPLAY` set, for example over SSH), `start` renders the cameras with EGL: it sets `MUJOCO_GL=egl` (and `PYOPENGL_PLATFORM=egl`) when `MUJOCO_GL` is unset and an EGL library is installed, and logs it. If MuJoCo still cannot render, `start` runs without cameras and logs a warning. An explicit `MUJOCO_GL` always wins; use `MUJOCO_GL=osmesa` on a machine without a GPU driver (slower, needs OSMesa installed):

```bash
MUJOCO_GL=osmesa uv run --package physicalai-mujoco-plugin physicalai-mujoco start
```

The first `start` on a machine downloads the SO-101 model from [MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie) (about 4 MB) into a per-user cache before the simulation starts. To download it ahead of time, for example while building a container image, run:

```bash
uv run --package physicalai-mujoco-plugin physicalai-mujoco prefetch
```

On a machine without internet access, run `prefetch` on a connected machine and copy the cache (set its location with `MENAGERIE_CACHE_DIR`), or set `MENAGERIE_ROOT` to a `mujoco_menagerie` checkout. A simulation owner started some other way, such as `examples/run_mujoco_owner.py`, downloads the model during its startup instead, so run `prefetch` first on a slow connection.

By default this starts:

- A MuJoCo SO-101 owner named `mujoco-so101-follow`
- Browser viewer at `http://127.0.0.1:9090`
- Camera streams served over HTTP (MJPEG + REST control server on port `8080`)
- Control rate `50 Hz`
- Substeps `10`

To drive the simulation from Physical AI Studio, see [Use with Physical AI Studio](#use-with-physical-ai-studio).

### Browser viewer controls

The viewer opens on its **Simulation** tab, which controls the simulation. The **Camera** tab controls whether the view follows a body. The **Visualization** and **Groups** tabs come from mjviser and control what is drawn.

- **Scene**: pick another scene for the running robot; every tabletop scene runs one or two arms, and a switch keeps the arm count. **Reset Scene** respawns the scene's objects while keeping the target fixed. **Home Arm** puts the arm joints and their position targets at the scene's home pose. An active policy or teleop session will drive the arm away again on its next action. During a replay ([`POST /replay`](#replay-recorded-joint-positions)), the folder shows the replayed frame and a **Stop Replay** button. **Overview camera** moves the `overview` camera between **Shoulder** and **Front** ([Overview camera styles](#overview-camera-styles)); it is disabled while the conveyor's automatic Studio recording runs.
- **Robot**: the profile, its tier, the joint units, the overview camera style, and each floating base's height, tilt and hold.
- **Performance**: the simulation speed as a multiple of real time, the control loop rate, and each camera's frame rate, over the last two seconds. Below 0.95x real time, the belt, arm and physics all run slower than the wall clock, while Studio records and policies run at wall-clock rates. Keep it at 1.00x when recording or evaluating.
- **Randomization**: tick **Fixed seed** to reseed before every reset and scene switch, so object layouts repeat. Untick it to go back to random layouts.
- **Episode** (`single_pick_place` only): the cube respawns after it rests on the target for the success dwell. Turn **Auto-reset** off or change the dwell here. The panel shows the countdown and the number of completed episodes.
- **Conveyor** (`conveyor_sort` only, in place of **Episode**): **Belt running** pauses or resumes the belt and the item feed. **Belt speed** sets the belt surface speed from 0 to 10 cm/s. The panel shows the items fed so far, the current and last episode's score, and the sorting rule.
- **Objects**: tick **Drag objects** to show a handle on each free object. While you drag, the object stays at the handle's pose. When you let go, it falls from there with zero velocity. The target stays fixed.
- **Cameras**: tick **Show previews** to see low-rate thumbnails of the rendered cameras. Previews start off because every open viewer receives them.
- **Shutdown** asks for confirmation before stopping the owner (hidden with `--viewer-theme studio` or `compact`).

On the **Camera** tab, **Follow** chooses a body for the view to stay on: a free object, the target, or a gripper. Orbiting and zooming keep the view on it. Panning away stops following, and **Follow** goes back to **None**. **None**, the default, is a free camera that nothing moves. While you drag the followed object, the view holds still, then glides back onto it when you let go. **FOV** and **Reset View** apply to every open viewer. The follow choice is shared too. Following moves the viewer cameras; the rendered scene itself never shifts.

Switching scenes rebuilds the whole viewer for the new model. The follow, drag and preview settings carry over to the new scene; following an object the new scene lacks falls back to **None**. Changes made over HTTP show up in the panel within about half a second. Camera settings are viewer-only and have no HTTP endpoint.

### Bimanual simulation

`--bimanual` runs two arms in the chosen scene, any tabletop scene, or the profile's default scene. Start one alongside a single-arm owner using separate server ports:

```bash
uv run --package physicalai-mujoco-plugin physicalai-mujoco start --bimanual --http-port 8081 --viser-port 9091
uv run --package physicalai-mujoco-plugin physicalai-mujoco start --bimanual --http-port 8082 --viser-port 9092 --name so101-bimanual
```

The arms are named `left_…` and `right_…`; the left arm is on the left seen from behind the arms. Connect Studio's `MuJoCo SO-101 Bimanual Follower` to `mujoco-so101-bimanual-follow`. Its camera names are `left_wrist`, `right_wrist`, and `overview`, served on the selected HTTP port.

Each scene places the two arms itself: `conveyor_sort` puts them behind its bins; the other tabletop scenes place them side by side at the single arm's table edge, 20 cm apart for the SO-101, with the block's spawn arc 2 cm closer. ALOHA (already two arms in one model) and floating-base robots run alone, so `--bimanual` refuses them. The conveyor autopilot drives a single SO-101; with two arms the belt runs and the autopilot stays off.

### Stop an owner

```bash
uv run --package physicalai-mujoco-plugin physicalai-mujoco stop --name mujoco-so101-follow
uv run --package physicalai-mujoco-plugin physicalai-mujoco stop --name mujoco-so101-bimanual-follow --http-port 8081
```

The stop command selects a local owner by name. Its HTTP fallback checks that the server reports the requested name.

### CLI teleoperation (self-relay)

With the owner running, you can also relay the simulation back to itself using
the [PhysicalAI CLI](https://github.com/openvinotoolkit/physicalai):

```bash
uv run --package physicalai-mujoco-plugin physicalai run --config packages/physicalai-mujoco-plugin/examples/runtime/teleop.yaml
```

Press `Ctrl+C` to stop.

View the camera streams in a browser or with `curl`:

```bash
curl http://127.0.0.1:8080/health
```

MJPEG stream URLs are `http://127.0.0.1:8080/cameras/<name>/mjpeg` (e.g. open
`http://127.0.0.1:8080/cameras/overview/mjpeg` in a browser, or play it in VLC).

## Use with Physical AI Studio

Studio sees the simulation as an ordinary follower robot plus IP cameras, so teleoperation, dataset recording and policy inference work the same way they do with a real SO-101. The simulation runs as its own process; Studio attaches to it by name over the local Physical AI transport.

### 1. Install the plugin into Studio

The plugin is not on Studio's curated **Plugins** page, so install it into Studio's backend environment. From the Studio repository:

```bash
cd application/backend
uv add physicalai-mujoco-plugin
uv sync
```

The PyPI release may lag behind this repository. To use the version in this checkout, including the viewer controls described above, run `git lfs pull` in this repository (the robot meshes are stored in Git LFS), then install it as an editable path instead:

```bash
cd application/backend
uv add --editable /path/to/physicalai/packages/physicalai-mujoco-plugin
uv sync
```

Restart Studio. The robot picker then offers **MuJoCo SO-101 Follower** and **MuJoCo SO-101 Bimanual Follower**, one MuJoCo follower for each other supported robot, and bimanual followers for the other twins, the WidowX AI and the reBot B601 (see [Other robots](#other-robots)). See Studio's robot plugin guide for Docker and other install options.

### 2. Start the simulation

Start the simulation before you add or open the robot in Studio. Studio checks that the robot is online by connecting to it. Run it from Studio's backend environment, so the simulation and Studio use the same `physicalai` version:

```bash
cd application/backend
uv run physicalai-mujoco start
```

The simulation registers under the name `mujoco-so101-follow` (`mujoco-so101-bimanual-follow` with `--bimanual`). These are also the default robot names that Studio fills in for the MuJoCo followers, so you can keep both unchanged. To run under another name, pass `--name` and enter the same name when you add the robot in Studio:

```bash
uv run physicalai-mujoco start --name my-sim
```

Keep this terminal open. Open the viewer at `http://127.0.0.1:9090` to watch the scene and use the **Simulation** tab.

Studio and the simulation must run on the same machine. The transport only accepts local connections unless both sides allow remote ones (`--allow-remote` here, and the remote-connection option in the robot's advanced settings in Studio). A Studio instance running in Docker does not share the host's localhost.

### 3. Add the robot and cameras

Studio hides the **IP Camera** type behind a feature flag that is off by default. Enable it once in the browser: open Studio, run the following in the developer console, then reload the page.

```js
setFeatureFlag("ipcam", true);
```

Then, in your Studio project:

1. Open **Robots** and select **Configure new robot**. Choose **MuJoCo SO-101 Follower** and keep the name `mujoco-so101-follow`. The name must match `--name` if you changed it.
2. Add a leader for teleoperation. The simulation provides only the follower. Use a real **SO101 Leader** arm connected over USB.
3. Open **Cameras**, select **Configure new camera**, and add an **IP Camera** for each simulated camera, with these stream URLs:

   - `http://127.0.0.1:8080/cameras/wrist/mjpeg`
   - `http://127.0.0.1:8080/cameras/overview/mjpeg`

   The cameras render 640×480 at 30 fps, which matches the IP Camera defaults.

4. Open **Environments**, select **Configure new environment**, and add the MuJoCo follower, the leader, and both cameras.

For the bimanual simulation, start it with `--bimanual` (in any tabletop scene, chosen with `--scene`), choose **MuJoCo SO-101 Bimanual Follower** with the name `mujoco-so101-bimanual-follow`, and add the `left_wrist`, `right_wrist` and `overview` cameras from its HTTP port. Every tabletop scene streams the same cameras for a robot and arm count, so a dataset keeps its camera features when you switch scenes. Its leader is a bimanual SO-101 leader, for example from the Bimanual SO-101 plugin.

Other robots work the same way: start the simulation with `--profile` and choose the matching follower, for example **MuJoCo WidowX AI Follower** for `--profile trossen_wxai`, whose default name is `mujoco-trossen_wxai-follow` (`mujoco-<profile>-follow`, or `mujoco-<profile>-bimanual-follow` for two robots). A twin takes its real robot's leader, for example a WidowX AI leader arm. Each follower's advanced settings also hold the simulation's HTTP address (`http_url`, default `http://127.0.0.1:8080`), where its cameras and its 3D viewer (`/viewer`) are served.

### 4. Teleoperate and record datasets

Record from the dataset page as with a real robot (**Add episode**, **Start episode**, then **Accept** or **Discard**). Use the viewer in place of resetting a physical scene:

- **Reset Scene** before each episode respawns the objects. The target stays fixed.
- **Home Arm** puts the simulated arm at its home pose. The next teleop action moves it again, so use it while the leader is still or before starting teleoperation.
- In `single_pick_place`, auto-reset respawns the cube five seconds after it rests on the target, which can happen in the middle of a recording. Turn **Auto-reset** off in the **Episode** section if you prefer to reset by hand.
- Tick **Fixed seed** to get the same object layouts in the same order, for example to compare runs.

The same controls are available over HTTP (see [REST control API](#rest-control-api)) if you want to script resets between episodes.

#### Record the conveyor autopilot, with automatic episodes

In `conveyor_sort`, the scripted demonstrator can act as the leader arm, and the simulation can save each conveyor episode to the open dataset. No one has to teleoperate, and no one has to press **Accept** or **Start episode**.

1. In Studio, add a robot of type **MuJoCo SO-101 Virtual Leader**. Its port is the simulation's `--http-port` (default `8080`). Use it as the leader of the MuJoCo environment.
2. In the viewer's **Autopilot** folder, set **Autopilot** to **Virtual leader (Studio teleop)**. The demonstrator now only publishes its joint targets (`GET /leader`); the arm moves once Studio sends them back as actions.
3. In Studio, open the dataset's recording view and start teleoperation. The demonstrator's targets become the recorded actions.
4. In the viewer, set **Task** and **Keep**, then tick **Record in Studio**. The simulation attaches to Studio's running session. For each episode it clears the belt, starts a recording with the task, and holds the belt when the tenth item is scored. Then it saves the episode, or discards it if **Keep** is **Perfect episodes only** and the episode had a wrong or missed item.
5. Untick **Record in Studio** to stop, or set **Episodes** to stop after that many saved episodes. Pressing stop in Studio also stops the automation.

**Record in Studio** also works with a physical leader arm. You teleoperate, and the simulation saves an episode each time the belt's ten items are scored.

The simulation joins Studio's session the same way a second browser tab does. It finds the session through `GET /api/runtime/sessions` and never stops that session. Use `--studio-url` if Studio does not run at `http://127.0.0.1:7860`.

**Autopilot** has a third mode, **Drive the arm**. It moves the arm directly and ignores incoming actions, which is useful for demos, but don't record in this mode.

### 5. Run a trained policy

On the **Models** page, select **Run model** for a policy trained on the simulation dataset. Studio loads the environment the dataset was recorded with, so keep the simulation running with the same owner name and camera ports. Use **Reset Scene** or **Fixed seed** in the viewer between runs.

To run the policy outside Studio, download its runtime configuration and run it with the Physical AI CLI. The plugin must be installed in that environment too; from this repository:

```bash
uv run --package physicalai-mujoco-plugin physicalai run --config runtime.yaml
```

### 6. Stop

Press `Ctrl+C` in the `start` terminal, use the viewer's **Shutdown** button, or run the [`stop` command](#stop-an-owner). Stop any teleoperation or inference session in Studio first; otherwise Studio reports the robot as disconnected.

## CLI options

```bash
uv run --no-sync physicalai-mujoco start --help
```

Common options:

- `--name <robot-name>`: transport name (must match Studio payload)
- `--profile <name>`: select a registered profile (`so101`, `trossen_wxai`, `rebot_b601`, `ur5e`, `aloha`, ...) or Menagerie model name; `physicalai-mujoco profiles` lists registered profiles
- `--bimanual`: two arms in the chosen scene (any tabletop scene), named `left_…` and `right_…`, under `mujoco-<profile>-bimanual-follow`
- `--model <path>`: scene XML to load instead of the registered scene's. Its `robot_mount` frames get the profile's robot and decide the arm count (`--bimanual` needs `left_robot_mount` and `right_robot_mount`); an XML without them is used as is
- `--scene <name>`: registered scene name. Default comes from the selected profile
- `--overview <shoulder|front>`: where a tabletop scene's `overview` camera stands, see [Overview camera styles](#overview-camera-styles) (default `shoulder`)
- `--seed <0..4294967295>`: fixed seed for scene resets, so object layouts repeat, as with **Fixed seed** in the viewer or `POST /seed` (default: random layouts)
- `--no-gui`: disable all viewers
- `--viser-port <port>`: browser viewer port (default `9090`; `0` picks a free port)
- `--viser-host <host>`: browser viewer bind host (default `127.0.0.1`; use `0.0.0.0` to expose it remotely)
- `--viewer-theme <default|studio|compact>`: `studio` hides the browser viewer's share and **Shutdown** buttons, for a viewer embedded in Studio, which stops the simulation itself; `compact` does the same and starts with the control panel minimized, for a viewer embedded next to its own controls (a notebook); the look stays viser's default
- `--no-cameras`: disable camera rendering (HTTP streams and viewer previews)
- `--http-host <host>`: host for the camera/control HTTP server (default `127.0.0.1`)
- `--http-port <port>`: port for the camera/control HTTP server (default `8080`; `0` picks a free port)
- `--no-http`: disable the camera/control HTTP server
- `--status-json`: print startup events as JSON lines on stdout, see [Start from another program](#start-from-another-program)
- `--exit-with-parent`: shut down when stdin reaches end of file, see [Start from another program](#start-from-another-program)
- `--rate-hz <float>`: owner loop frequency
- `--substeps <int>`: MuJoCo steps per control cycle (default: real time at `--rate-hz`, `10` in the SO-101 scenes)
- `--unit <normalized|degrees>`: joint units for observations and actions (default comes from the selected profile; SO-101 uses `normalized`, see [Joint units](#joint-units))
- `--idle-timeout <seconds>`: seconds with zero subscribers before self-exit
  (default `10` without HTTP, disabled when HTTP is enabled so stream viewers keep the sim alive)
- `--allow-remote`: allow non-loopback zenoh connections
- `--studio-url <url>`: Physical AI Studio backend for automatic episode recording (default `http://127.0.0.1:7860`)

### Start from another program

A program that starts and supervises the simulation, such as Physical AI Studio, can follow its startup and tie its lifetime to its own:

```bash
physicalai-mujoco start --status-json --exit-with-parent --http-port 0 --viser-port 0 --viewer-theme studio
```

`--status-json` prints one JSON object per line on stdout and flushes each line; logs stay on stderr:

```json
{"event": "phase", "phase": "fetch"}
{"event": "phase", "phase": "fetch", "bytes": 1048576, "total": 4194304}
{"event": "phase", "phase": "connect"}
{"event": "phase", "phase": "load"}
{"event": "phase", "phase": "cameras"}
{"event": "ready", "name": "mujoco-so101-follow", "pid": 4242, "profile": "so101", "scene": "single_pick_place", "arms": 1, "overview_style": "shoulder", "http_url": "http://127.0.0.1:53412", "viewer_url": "http://127.0.0.1:53413", "cameras": ["wrist", "overview"]}
```

- `fetch` comes before the robot model download from MuJoCo Menagerie; while it downloads, more `fetch` events report `bytes` so far and the `total` (`0` when unknown). A cached model sends no `bytes` events.
- `connect` comes before the owner process starts, `load` once the owner loads the scene and starts the viewer, cameras and HTTP server, and `cameras` while `start` waits up to 15 s for each camera's first frame.
- `ready` comes once the owner runs. `pid` is the owner process, `scene` is `null` for a `--model` without `--scene`, `arms` is the number of robots, and `overview_style` is the `overview` camera's style at start. `cameras` lists only the cameras that streamed a first frame, served at `<http_url>/cameras/<name>/mjpeg`; a camera whose renderer failed is left out, with a warning on stderr. Without HTTP (`--no-http`), `http_url` is `null` and `cameras` is empty. `viewer_url` is `null` without a viewer. A bind-all host (`0.0.0.0`, `::`) is reported as loopback.
- `error`, with a `message`, means `start` failed; it exits with a non-zero code. This includes a requested HTTP server that does not answer (for example its port was taken) and a requested browser viewer that did not start (viser failed to import or to build its scene; without `--no-gui`): `start` then stops the owner it started. When the viewer port is taken, viser listens on the next free port, and `viewer_url` reports that one. A shutdown before `ready` (stdin end of file with `--exit-with-parent`, or a signal) is an `error` too, and `start` stops the owner it started. Invalid arguments exit with code 2 before any event.

With `--status-json` or `--exit-with-parent`, `start` only runs a simulation it starts itself. If a simulation with the same name is already running, `start` fails with an `error` and leaves that simulation alone; it never stops an owner it did not start.

`--http-port 0` and `--viser-port 0` pick free ports; `ready` reports the ports in use. `start` picks them before the owner binds them, so another process could take one in between.

`--exit-with-parent` makes `start` read stdin and shut down at end of file, as on `SIGTERM`. A parent keeps the write end of a pipe to `start`'s stdin open and never writes to it; when the parent exits or crashes, the operating system closes the pipe (macOS and Linux). The owner process watches `start` too, from the moment it starts loading, and shuts down if `start` itself is killed. Don't use `--exit-with-parent` with stdin at `/dev/null`: that is end of file at once.

The browser viewer has no `X-Frame-Options` or `Content-Security-Policy` header, so a page can embed `viewer_url` in an iframe.

Studio gets this command line from the catalog: every simulated follower's `simulation` field (a [`SimulationLaunch`](../physicalai-studio-plugin/README.md#simulationlaunch)) lists the scenes that entry runs in, with the profile's default scene preselected, labels the entry with its arm count and tier, and builds `python -m physicalai_mujoco_plugin start --profile <profile> [--bimanual] --scene <scene> --name=<payload name> --status-json --exit-with-parent --http-port 0 --viser-port 0 --viewer-theme studio [--seed <n>]` in Studio's own Python. Building it loads no robot model and downloads nothing; the `fetch` phase does that.

## Joint units

The simulation reports and accepts joint values in the same units as the real SO-101 (`physicalai.robot.SO101`):

- Body joints go from `-100` to `100`, and the gripper from `0` to `100`.
- On a real robot, calibration sets these ranges: `-100` and `100` are the two ends you moved each joint to while calibrating, and `0` is halfway between them.
- The simulation has no calibration, so it uses the joint limits from the robot's URDF (`urdf/so101/so101_new_calib.urdf`) instead: `-100` and `100` are each joint's limits, and `0` is halfway between them. The MuJoCo models use the same limits.

This lets a real leader arm drive the simulation, and makes simulated datasets use the same values as real ones.

The match is not exact. If a joint was moved further one way than the other during calibration, its `0` is not at the joint's straight position, so the same value points the real joint and the simulated joint in slightly different directions. A real arm's calibrated range also differs from the URDF limits by a few degrees because of assembly tolerances, which adds to the difference towards the ends of a joint's range.

Use `--unit degrees` (or `unit="degrees"` on `MuJoCoRobot`) to get joint angles in degrees instead. Sliding joints, such as a parallel gripper's, stay in metres.

## Other robots

Every robot comes from MuJoCo Menagerie and is described by a _profile_: what the plugin knows about it beyond its model. `physicalai-mujoco profiles` lists the hand-written ones. Their _tier_ says how closely they are maintained:

- `twin`: matches a real robot's driver, with the same joint names and units.
- `dataset`: an arm that public robot-learning datasets use. Channels, home pose and cameras are derived from the model. Channel names change only where LeRobot datasets name the joints (ALOHA, SO-100, Koch); values stay in model units from the model's zero pose, without any dataset's calibration.
- `experimental`: floating-base robots on floor scenes; nothing balances them.

| Profile                | Robot                                                   | Tier           | Scenes              | Arms | Units        |
| ---------------------- | ------------------------------------------------------- | -------------- | ------------------- | ---- | ------------ |
| `so101`                | SO-101, matching the real `SO101` driver                | `twin`         | all tabletop scenes | 1, 2 | `normalized` |
| `trossen_wxai`         | Trossen WidowX AI, matching Runtime's `WidowXAI` driver | `twin`         | `single_pick_place` | 1, 2 | `degrees`    |
| `rebot_b601`           | Seeed reBot B601-RS, `ReBotB601RS` names (provisional)  | `twin`         | `single_pick_place` | 1, 2 | `degrees`    |
| `ur5e`                 | Universal Robots UR5e (no gripper)                      | `dataset`      | `single_pick_place` | 1, 2 | `degrees`    |
| `aloha`                | ALOHA, both arms as one robot (14 joints)               | `dataset`      | `single_pick_place` | 1    | `degrees`    |
| `so_arm100`            | SO-ARM100 (Menagerie `trs_so_arm100`)                   | `dataset`      | `single_pick_place` | 1, 2 | `degrees`    |
| `koch`                 | Koch low-cost arm (`low_cost_robot_arm`)                | `dataset`      | `single_pick_place` | 1, 2 | `degrees`    |
| `piper`                | AgileX PiPER                                            | `dataset`      | `single_pick_place` | 1, 2 | `degrees`    |
| `franka_fr3`           | Franka Research 3 (no gripper)                          | `dataset`      | `single_pick_place` | 1, 2 | `degrees`    |
| `franka_panda`         | Franka Emika Panda                                      | `dataset`      | `single_pick_place` | 1, 2 | `degrees`    |
| `xarm7`                | UFACTORY xArm7                                          | `dataset`      | `single_pick_place` | 1, 2 | `degrees`    |
| `kinova_gen3`          | Kinova Gen3 (no gripper)                                | `dataset`      | `single_pick_place` | 1, 2 | `degrees`    |
| `unitree_g1`           | Unitree G1 humanoid                                     | `experimental` | `floor_flat`        | 1    | `degrees`    |
| `unitree_go2`          | Unitree Go2 quadruped                                   | `experimental` | `floor_flat`        | 1    | `degrees`    |
| `boston_dynamics_spot` | Boston Dynamics Spot                                    | `experimental` | `floor_flat`        | 1    | `degrees`    |

**Arms** is how many copies of the robot one simulation runs: 2 with `start --bimanual` (`physicalai-mujoco profiles` lists it too).

ALOHA attaches as Menagerie's two-arm model at the scene's single mount, so both arms keep their real spacing and their wrist cameras (`wrist_cam_left`, `wrist_cam_right`). Its joints are named `left_waist` ... `right_gripper`, as in LeRobot's ALOHA datasets. The Koch and SO-ARM100 profiles turn Menagerie's arm about its base so that it faces +x like the others (Menagerie's face -x and -y, and their base joint limits keep part of the table out of reach); joint names, ranges and zero pose are unchanged. The Kinova Gen3 has four unlimited joints, so it runs in `degrees` only.

`single_pick_place` scales with the arm (`SceneConfig.layout="reach"`). Its layout is sized for the SO-101; for another arm, the block's spawn arc (centre and radii), the target disc and the `overview` camera move away from the mount by `reach / SO-101 reach`, so the camera frames the larger layout as it frames the SO-101's. Object sizes and separations stay as they are, and the SO-101 gets the scene exactly as written. ALOHA's arc is centred between its two arms (`SceneConfig.profile_spawn_centers`), and each arm covers its own side. Arms whose home pose the scaled camera would crop get their own overview rig pose (`SceneConfig.profile_overview_rigs`), pulled back along the line of sight; ALOHA's looks across the table. A test projects every arm's geometry at home, the spawn arc and the target into the 640x480 overview, and live XML edits of the rig keep the scaling and the override.

With two arms side by side (`--bimanual`), the arc moves 2 cm closer (`SceneConfig.two_arm_spawn_center`) and the separation scales like the rest of the layout. The UR5e, whose home pose reaches 54 cm sideways, stands 65 cm apart (`SceneConfig.two_arm_separations`). The framing test covers both arm counts.

A reset tries not to spawn the block (or, in `yahtzee`, drop a die) under a low part of an arm: it takes every arm where it stands at that moment, projects each robot geom whose lowest point is within 10 cm of the table (scaled with the layout) onto the table, and draws again any position closer to that footprint than the object's horizontal radius plus 1 cm. When none of 200 draws is clear, it keeps the draw farthest from the footprint (one that also keeps clear of the target and the other objects, if any does), so an arm that covers the whole spawn area still gets an object, possibly under it. Parts higher up, such as a forearm reaching over the spawn area, do not count. At home this only moves the SO-ARM100's block, whose jaws hang 7 cm above the middle of its arc.

Two profile fields drive this:

- `RobotProfile.end_effectors`: each arm's tool frame, as a model site (`EndEffector(site="attachment_site")`) or a point in a body (`EndEffector(body="hand", pos=(0, 0, 0.110))`), and its approach `axis`, the direction in that frame that points out of the gripper or flange. Menagerie's tool sites are used where they exist; the SO-ARM100, Koch, PiPER, Panda and reBot B601 use the point between their finger tips. The arms without a gripper (UR5e, FR3, Kinova Gen3) use their flange.
- `RobotProfile.reach`: the farthest horizontal distance from the arm's base, toward the spawn area, at which the end effector reaches a point 2 cm above the table within 5 mm, with its approach axis within 15° of straight down (the SO-101: 0.367 m). A `reach` scene refuses a profile without one.

The reachability test (`tests/test_reach.py`, marker `mujoco_smoke`) solves a damped least-squares IK for every arm on the centre and the extremes of its scaled spawn arc: within 1 cm, approach axis within 15° of straight down (the SO-101 needs 12° at the far edge of its own arc). With two arms, both reach the points on the arc's axis and the nearest arm reaches the others. It also measures each `reach` and checks the stored value to 1 cm. After changing an arm, print fresh measurements with `uv run python packages/physicalai-mujoco-plugin/tests/test_reach.py [profile ...]`.

Every profile in these tiers has a Studio follower entry for one robot. The twins (SO-101, WidowX AI, reBot B601), whose real robots have bimanual leaders, also have a bimanual entry; other arms run two from the CLI only (`start --bimanual`). Entries whose mapping onto the real driver is not established yet (`RobotProfile.provisional`, today the reBot B601) end in **(provisional)** and carry a `provisional` label in the start dialog.

Any other Menagerie model name loads as an unsupported profile, outside CI and Studio: its channels, home pose, sensors and cameras are derived from the model, one channel per actuator. It needs a scene that supports it (`--scene`) or a robot-complete `--model`. Floor scenes such as `floor_flat` take any floating-base robot. Every channel's action is a position, also for velocity actuators, which the simulation drives toward it. Torque actuators on a tendon or site need `MuJoCoRobot(..., torque_mode="raw")`, whose actions are `ctrl`.

```bash
uv run --package physicalai-mujoco-plugin physicalai-mujoco start --profile ur5e
```

From Python, or with `physicalai robot serve`, use the generic driver:

```python
from physicalai_mujoco_plugin import MuJoCoRobot

robot = MuJoCoRobot(profile="ur5e", scene="single_pick_place")
```

### Twins

A twin uses its real driver's joint names, units and `state`. The SO-101's and the WidowX AI's model joints map one to one onto their real drivers' joints, so their datasets and policies are meant to move between the simulation and the real robot; the reBot B601's mapping is still unverified:

- **WidowX AI** (`trossen_wxai`, Menagerie's `wxai_follower`): `shoulder_pan`, `shoulder_lift`, `elbow_flex`, `wrist_flex`, `wrist_yaw`, `wrist_roll` in degrees and `gripper` (the carriage opening) in metres; `sensor_data["velocities"]` in rad/s and m/s, unconverted like the real driver's. The real driver's `efforts` are not simulated, so the key is absent. Its `wrist` camera sits where Menagerie places the D405. Two arms (`--bimanual`) are named `left_…` and `right_…`, like `BimanualWidowXAI`. Studio's **MuJoCo WidowX AI Follower** shows Studio's own WidowX AI model.
- **reBot B601** (`rebot_b601`, Menagerie's `seeed_rebot_devarm`, **provisional**): the joint names, order and degrees of the B601 plugin's `ReBotB601RS` in the LeRobot-style joint frame of [#360](https://github.com/openvinotoolkit/physicalai/pull/360) (see [#343](https://github.com/openvinotoolkit/physicalai/issues/343)): Menagerie's joints one to one, and `gripper` in the driver's degrees too (−270 open to 0 closed). The joints stop where the driver clips its actions (±145° pan, −170…0° lift, −90…80° wrist flex, ±90° roll), except `elbow_flex`, which the model stops at −180° while the driver allows −200°.
  Until #360 merges, the B601 plugin still uses its previous frame, which differs in the signs of `shoulder_pan`, `shoulder_lift` and `wrist_roll` and in the gripper's sign and scale, so data recorded with one does not carry over to the other. The mapping follows #360's joint-frame URDF and **has not been checked on a real arm**, so moving datasets or policies between this simulation and a real B601 is **not established**.
  It needs a read-only comparison on hardware first: move each joint by hand and compare the driver's readings with the simulation's, without zeroing or actuating the arm. Studio marks its followers **(provisional)**. Its `wrist` camera is the RealSense D405 on Seeed's printed mount, placed from the mount's CAD geometry; which side of the gripper it faces is assumed until that hardware check.

### Floating-base robots

Humanoids and quadrupeds spawn at a scene's `<frame name="{prefix}robot_spawn"/>`: at the frame's position and heading, at the height and orientation of the robot's home keyframe. The frame's height, roll and pitch are ignored. A weld holds the base there until the first action, so an idle robot does not fall over; nothing balances it after that. The experimental profiles stand on their home pose when the action holds it.

- `state` is `joint_positions ‖ base_quat (4) ‖ base_angvel (3) ‖ base_linvel (3) ‖ model sensors`, in that order. With several floating robots, the joints of all robots come first, then one base-and-sensors block per robot in anchor order. Fixed-base robots keep `state = joint_positions`.
- `sensor_data` adds `base_pos` (world, m), `base_quat` (wxyz), `base_angvel` and `base_linvel` (body frame), and `sensor/<name>` for every model sensor.
- `robot.reset()`, `POST /reset` and the viewer's **Reset** button put the robot back at its start pose and hold its base again, then reset the scene. Call `robot.reset()` from the control-loop thread. Fixed-base robots keep the scene-only reset; `POST /home` homes them.
- The base hold always lasts until the first action; there is no setting to keep or disable it.
- `/health` reports `floating_base`, `fallen` (the base tilted more than 60° for 1.5 s) and per robot `bases` (height, tilt, held, fallen).

```bash
uv run --package physicalai-mujoco-plugin physicalai-mujoco start --profile unitree_g1
```

## Cameras over HTTP

The plugin renders the camera feeds on their own thread and serves them over HTTP. The control loop only hands the camera thread a pose snapshot each tick, so rendering never delays physics. By default it streams the robot's first camera, the scene's `overview`, then the other robot cameras, at 640×480 and 30 fps. For the SO-101:

- `wrist` -> `http://127.0.0.1:8080/cameras/wrist/mjpeg`
- `overview` -> `http://127.0.0.1:8080/cameras/overview/mjpeg`

A robot's cameras come from its profile, else from the model's own sensor cameras (not Menagerie's viewer cameras such as `track`), else the plugin generates one: `wrist` beside an arm's gripper or flange, `head` on a floating base's head, or `front` on its body. A generated `wrist` of a profile with an end effector (`RobotProfile.end_effectors`) sits on the end effector's body, 4 cm behind its tool point and to the side, and looks along its approach axis at a point 10 cm past the tool point, with the fingers or flange at the bottom of the image; without one, the tool axis is guessed from the model's geometry.

A wrist camera follows the gripper, as on a real arm, so its view at the start depends on the home pose: where Menagerie's home pose points the tool forward (Kinova Gen3's own camera, PiPER, Koch), the wrist camera looks toward the horizon until the arm reaches down to the table. The UR5e has no camera of its own, so it streams a generated `wrist` and `overview`. A second robot's cameras carry its prefix (`left_wrist`, `right_wrist`). `GET /health` labels each camera's `source` (`override`, `model`, `default` or `scene`). A scene camera named `chase` follows the first robot's floating base and streams after `overview`.

Each camera is also available as a single JPEG snapshot at
`http://127.0.0.1:8080/cameras/<name>/frame.jpg`.

### Overview camera styles

The tabletop scenes place their `overview` camera in one of the two common third-person styles:

- `shoulder` (default): high behind the robot base, looking over it at the work area, like the over-the-shoulder camera of [BridgeData V2](https://rail-berkeley.github.io/bridgedata/). This is the scene file's own pose, and the SO-101's is unchanged from earlier releases, so datasets recorded and policies trained on it keep working. A gripper hovering at home can hide a spawned block from it.
- `front`: across the work area from the robots, facing them, 45 degrees down at the spawn area's centre (between the arms with two), like the `agentview` camera of [robosuite](https://robosuite.ai) and [LIBERO](https://libero-project.github.io). Its distance and aim point grow with the arm's reach like the scene's layout, so every arm, the spawn area and the target stay in frame; in `single_pick_place` the arms at home hide neither a spawned block nor the target. ALOHA's front camera faces its arms from the side opposite its teleoperator.

The camera is named `overview` in both styles, so the camera set and a dataset's camera features stay the same. Choose the style when the simulation starts, with `start --overview front` or `MuJoCoRobot(overview="front")`. While it runs, switch it with the viewer's **Overview camera** dropdown (Simulation tab, **Scene** folder) or `POST /overview`. The switch moves only the camera; the scene, arms and objects stay where they are, and later scene switches keep the style. Datasets and policies expect the view they were recorded with: don't switch during a recording, and keep one view per dataset. While the conveyor's automatic Studio recording runs, the switch is refused (`409`, and the dropdown is disabled).

The style is not saved: it lasts as long as the simulation, and every start uses `--overview` or the `overview` argument (default `shoulder`). `floor_flat` and custom `--model` XMLs keep their own cameras and offer only `shoulder`; asking them for `front` fails at start, and `POST /overview` returns `409`. `GET /health` reports `overview_style`, the scene's `overview_styles`, and `overview_locked`, the reason a switch is refused now (or `null`).

### REST control API

The HTTP server exposes the same controls as the viewer's Simulation tab:

```bash
# List available scenes, the ones this robot can switch to, and the current one
curl http://127.0.0.1:8080/scenes

# Switch to another scene
curl -X POST http://127.0.0.1:8080/scenes/yahtzee

# The 3D viewer: GET / reports viewer_url, and /viewer redirects to it (404 without a web viewer)
curl -i http://127.0.0.1:8080/viewer

# Reset/randomize the current scene (floating bases also return to their start pose), or move the arm to its home pose
curl -X POST http://127.0.0.1:8080/reset
curl -X POST http://127.0.0.1:8080/home

# Move the overview camera across the table, facing the robots, and back behind them
curl -X POST http://127.0.0.1:8080/overview -H 'content-type: application/json' -d '{"style": "front"}'
curl -X POST http://127.0.0.1:8080/overview -H 'content-type: application/json' -d '{"style": "shoulder"}'

# Make resets repeatable, then random again
curl -X POST http://127.0.0.1:8080/seed -H 'content-type: application/json' -d '{"seed": 42}'
curl -X POST http://127.0.0.1:8080/seed -H 'content-type: application/json' -d '{"seed": null}'

# Pause episode auto-reset and change its success dwell
curl -X POST http://127.0.0.1:8080/episode/auto-reset -H 'content-type: application/json' \
  -d '{"enabled": false, "dwell_s": 3.0}'

# conveyor_sort only: set the belt speed in m/s (pause it with /episode/auto-reset enabled=false)
curl -X POST http://127.0.0.1:8080/conveyor/belt-speed -H 'content-type: application/json' -d '{"speed": 0.05}'

# Read and move free objects (world frame, metres; wxyz is optional)
curl http://127.0.0.1:8080/objects
curl -X POST 'http://127.0.0.1:8080/objects/block1:joint/pose' -H 'content-type: application/json' \
  -d '{"position": [0.25, 0.0, 0.05], "wxyz": [1, 0, 0, 0]}'

# Replay recorded joint positions (one row per frame, in the robot's units), then resume
curl -X POST http://127.0.0.1:8080/replay -H 'content-type: application/json' \
  -d '{"joint_positions": [[0, -50, 40, 20, 0, 10], [1, -49, 39, 20, 0, 12]], "fps": 30, "base": null}'
curl -X POST http://127.0.0.1:8080/replay/stop

# Stop the simulation owner
curl -X POST http://127.0.0.1:8080/shutdown
```

| Endpoint                    | Method | Description                                                                                     |
| --------------------------- | ------ | ----------------------------------------------------------------------------------------------- |
| `/`                         | GET    | Service info, endpoint index                                                                    |
| `/health`                   | GET    | Sim status: connected, scene, compatible scenes, overview style, seed, episode, timing, cameras |
| `/cameras`                  | GET    | Camera list with stream/snapshot URLs                                                           |
| `/cameras/{name}/mjpeg`     | GET    | MJPEG stream (`multipart/x-mixed-replace`)                                                      |
| `/cameras/{name}/frame.jpg` | GET    | Latest frame as a JPEG snapshot                                                                 |
| `/scenes`                   | GET    | Current scene, available scene IDs, and IDs compatible with this robot                          |
| `/scenes/{scene_id}`        | POST   | Switch to a compatible scene (`409` for another arm count)                                      |
| `/reset`                    | POST   | Reset/randomize the scene; floating bases return to their start pose (held)                     |
| `/home`                     | POST   | Move the arm joints and targets to the scene's home pose                                        |
| `/overview`                 | POST   | `{"style": "shoulder" \| "front"}`: move the overview camera; `409` while Studio records        |
| `/seed`                     | POST   | `{"seed": <0..4294967295> or null}`: fix or clear the reset seed                                |
| `/episode/auto-reset`       | POST   | `{"enabled": bool, "dwell_s": 0.5..120}` (either field); `409` if unsupported                   |
| `/conveyor/belt-speed`      | POST   | `{"speed": 0..0.1}` belt speed in m/s; `409` if the scene has no conveyor belt                  |
| `/autopilot`                | POST   | `{"mode": "off" \| "drive" \| "leader"}`; `409` if the scene has no autopilot                   |
| `/leader`                   | GET    | Virtual leader pose: `seq`, `mode`, `unit`, `joint_names`, `joint_positions`                    |
| `/studio/recording`         | POST   | `{"enabled": bool, "task": str (1-200), "keep": "perfect" \| "all", "max_episodes": 0..10000}`  |
| `/objects`                  | GET    | Free-object joint names and world poses                                                         |
| `/objects/{joint}/pose`     | POST   | `{"position": [x, y, z], "wxyz": [w, x, y, z]}`: teleport a free object                         |
| `/replay`                   | POST   | `{"joint_positions": [[...], ...], "fps": 0..1000, "base": null}`: replay recorded joints       |
| `/replay/stop`              | POST   | End the replay and restore the simulation state from before it                                  |
| `/shutdown`                 | POST   | Gracefully stop the simulation owner                                                            |

Control requests are queued and applied on the next control cycle. Invalid bodies return `422`, except for `/replay`. Object coordinates must be finite and within ±2 m.

#### Replay recorded joint positions

`POST /replay` plays recorded joint positions back kinematically, for example a dataset episode recorded from this robot. Each control tick sets the joints to the frame due at the replay's clock and runs `mj_forward`; physics does not step, so objects stay where they are. The replay clock advances by one control period per tick, like the simulation clock it replaces.

- `joint_positions`: one row per frame, one value per joint in `joint_names` order, in the robot's configured units (the same as its observations). Values are not clipped to the joint ranges, so a recording that pressed past a limit replays as recorded. Values the robot cannot show unchanged are refused with 400, naming the first frame and joint: `normalized` joints outside their range (−100…100, gripper 0…100), and any joint position beyond 10⁶ radians or metres after the profile's scale and offset. Every frame of every robot is checked before the replay starts, so a replay never stops half-way. At most 36000 frames and 16 MiB.
- `fps`: the recorded frame rate, above 0 and at most 1000. A recording slower than the control rate holds each frame for several ticks; a faster one skips frames.
- `base`: must be `null`; replaying base poses is not supported yet. A floating-base robot (G1, Go2, Spot) still replays its joints, and its base stays where it was, because physics does not step during a replay.

An invalid body returns `400` with the reason, a larger body `413`. A request while the simulation is disconnected returns `409`, as does one for a robot with a channel driven through a tendon or site instead of a hinge or slide joint (`unsupported_joints` in `GET /health`): such a position has no unique joint configuration to place.

While a replay runs, actions are ignored, and scene automation (the conveyor, autopilot, Studio recording and episode auto-reset) pauses. After the last frame the replay holds it until `POST /replay/stop`, which restores the full simulation state from before the replay, so a policy or teleop session continues where it was. A new `POST /replay` replaces a running one and keeps that saved state. **Reset**, **Home** and object moves end the replay the same way before they apply. A scene switch ends it with the old scene. `GET /health` reports `replay`: `active`, `unsupported_joints` and, during a replay, `finished`, the shown `frame` (from 0), `frames` and `fps`. A native-viewer reset ends a replay like `POST /reset`.

With HTTP enabled, `Ctrl+C` in the start command requests owner shutdown through HTTP. If the HTTP endpoint is unavailable, the launcher sends SIGTERM to the original owner only after confirming its registered name and PID still match. It then disconnects the CLI subscriber. With HTTP disabled, the same verified signal path stops a local owner; detached owners also exit after their configured idle timeout once all subscribers leave. Use the named `stop` command to stop an owner directly.

The `start` command also returns when its local owner exits through `stop`, HTTP, or the viewer's Shutdown control. Launcher cleanup checks the endpoint's owner name and the original owner's PID before forwarding an operator-requested shutdown. An invocation that attaches without a local owner record detaches instead of waiting indefinitely.

HTTP control has no authentication and binds to loopback by default. Use explicit non-loopback bindings only on a trusted robot-cell network. The Viser viewer exposes the same controls, including moving objects and Shutdown, to anyone who can open it; it binds to loopback by default. Use `--viser-host 0.0.0.0` only when remote access is needed on a trusted network.

## Troubleshooting

### `start` fails with `an OpenGL platform library has not been loaded into this process`

MuJoCo could not create an OpenGL context, usually because there is no desktop session (headless machine, SSH, container). `start` picks EGL by itself only when neither `DISPLAY` nor `WAYLAND_DISPLAY` is set and an EGL library is installed. Otherwise set `MUJOCO_GL=egl` before `start`, or `MUJOCO_GL=osmesa` without a GPU driver. See [Quick start](#quick-start).

### HTTP server unavailable or port already in use

- The camera/control server binds to `--http-host`/`--http-port` (default `127.0.0.1:8080`).
- If the port is taken the simulation continues without HTTP and logs a warning — pass a different `--http-port`.
- When running multiple simulations, give each a distinct `--http-port`.

### Viewer opens but has Wayland warnings (`libdecor`, window position)

These warnings are typically non-fatal on Wayland and can be ignored if simulation continues.

### Camera/control server is not started

- Confirm the sim is running and the port is free: `curl http://127.0.0.1:8080/health`
- `--no-http` disables the server (`--http-port 0` picks a free port instead); `--no-cameras` disables all camera rendering

### Studio shows the MuJoCo robot as offline

- Start the simulation first, then reload the robot in Studio. A session on a follower whose simulation is not running fails with `No MuJoCo simulation named '<name>' is running. Start it with: uv run physicalai-mujoco start --profile <profile> [--bimanual] [--name=<name>] [--http-port <port>]`, built from the follower's name and `http_url`. For an `http_url` on another host, it says to start it there with `--http-host`, `--http-port` and `--allow-remote`.
- Check that the robot name in Studio matches the owner name printed by `start` (default `mujoco-so101-follow`).
- Run Studio and the simulation on the same machine, or enable remote connections on both sides.

### Studio shows no camera image

- Open the stream URL in a browser; if it does not load, the HTTP server is not running or uses another port.
- Use an **IP Camera** with the full `/cameras/<name>/mjpeg` URL, not the server root.
- If **IP Camera** is missing from the camera types, enable its feature flag (see [Add the robot and cameras](#3-add-the-robot-and-cameras)).

## Scenes

The plugin ships with built-in scenes that provide different environments for the robot:

| Scene ID                      | Description                            | Free objects | Target      | Arms |
| ----------------------------- | -------------------------------------- | ------------ | ----------- | ---- |
| `single_pick_place` (default) | One block and a target disc            | 1 cube       | target disc | 1, 2 |
| `yahtzee`                     | Six dice and a cup                     | 6 dice       | cup         | 1, 2 |
| `conveyor_sort`               | Sort items off a moving belt into bins | 24-item pool | 4 bins      | 1, 2 |
| `floor_flat`                  | A flat floor for a floating-base robot | none         | none        | 1    |

Start with a specific scene:

```bash
uv run --no-sync physicalai-mujoco start --scene yahtzee
```

### Conveyor sort

`conveyor_sort` is a factory cell: items ride a belt past the arm and have to be sorted into bins before they reach the end.

- **Items**: cubes, upright cylinders and hexagonal prisms, 3 cm across, in red, blue, green or purple. About 30% are cracked (a bold dark crack texture). Unused items wait in the lidded supply crate behind the robot, and the feed moves them onto the belt inside the entry hood.
- **Bins**: red, blue and green bins plus a striped reject bin, each labelled on its floor (RED, BLUE, GREEN, REJECT) so the labels read upright in the overview camera.
- **Rule**: cracked items and purple items (which have no bin of their own) go to the striped reject bin; the others go to the bin of their color. An item that rides off the end of the belt, or comes to rest anywhere else, is a miss.
- **Episodes**: an episode feeds 10 items, 15 cm apart along the belt (so the spacing does not depend on the belt speed). Once every item is scored, the next episode starts. **Reset Scene** starts a fresh episode.
- **Stack light**: a three-lamp light on the entry hood shows green while the belt runs, amber about 2 s before the next item leaves the hood, and red while the belt is paused. `/health` reports the same lamps and the seconds until the next item (`episode.lights`, `episode.next_item_s`).
- **Belt speed**: 3 cm/s by default, adjustable from 0 to 10 cm/s in the viewer or with `POST /conveyor/belt-speed`. **Home Arm** holds the gripper above the pick zone.

At 224 px the overview camera shows item colors clearly, but not cracks: an item covers about 5 px. The wrist camera and the full-resolution overview show them.

The scene's textures and item pool (`conveyor_items.xml`) come from `scripts/generate_conveyor_assets.py`. The visual meshes for the conveyor frame, hood and bins come from `scripts/build_conveyor_meshes.py`, run with `blender --background --factory-startup --python scripts/build_conveyor_meshes.py`. Collisions use primitives in `scene.xml`, so keep the dimensions in both files in sync. Give every textured surface UV coordinates and a 2D texture: the browser viewer draws primitives in a flat color and reduces cube maps to one color per face, while the camera streams render either.

#### Scripted demonstrator

`physicalai_mujoco_plugin.conveyor_demo.ConveyorDemonstrator` sorts items using privileged simulation state (item poses, the belt speed, the sorting rule). It is meant for generating consistent demonstrations, not as a policy. Each cycle, it waits over the belt, tracks the most downstream item in the pick window, grasps it with the jaw closing along the belt, and drops it in the bin the rule assigns. It writes the arm's joint targets directly and runs headless:

```bash
cd packages/physicalai-mujoco-plugin
uv run python scripts/run_conveyor_demo.py --speed 0.03 --episodes 3             # score
uv run python scripts/run_conveyor_demo.py --sweep 0.01 0.03 0.05 0.07 --seeds 3  # success vs belt speed
uv run python scripts/run_conveyor_demo.py --speed 0.03 --video /tmp/demo.mp4     # overview, wrist, orbit video
```

Measured over 3 seeds of 3 episodes (90 items per speed): 99–100% of items sorted correctly at 1–5 cm/s, 81% at 7 cm/s and 48% at 10 cm/s. Above about 5 cm/s, items arrive faster than one pick-and-place cycle (about 2.3 s).

### Switching scenes at runtime

Pick a scene in the viewer's **Scene** dropdown or send `POST /scenes/{scene_id}`. The switch happens on the next control cycle:

- Loads the new scene XML and attaches the arm
- Rebuilds the viewer for the new environment
- Respawns the scene's free-object joints clear of their target, using the new spawn parameters (target bodies stay fixed)

A switch keeps the arm count, so the joint names stay those the transport advertised: a bimanual simulation switches to any tabletop scene with two arms. Only scenes that support the running robot are offered; over HTTP another request returns `409` and the current scene is kept. A scene whose model lacks the joints the robot drives is also rejected.

The native MuJoCo viewer also binds **`n`** (next scene) to cycle through the compatible scenes. That viewer is only used on Linux/Windows when the browser viewer fails to start.

`--model` selects the initial model instead of resolving a registered scene. The viewer's Scene dropdown shows it as **Custom model**, and **Home Arm** uses the model's default joint positions. Runtime scene-switch requests can still explicitly replace it with a compatible registered scene; switching back to the custom model is not available unless it is registered as a scene.

## Development

Fetch LFS assets before running the real-model tests. They load MuJoCo models without opening a display:

```bash
uv sync --all-packages --extra tests --extra transport
uv run pytest --import-mode=importlib packages/physicalai-mujoco-plugin/tests
uv build --package physicalai-mujoco-plugin
```

### Changing the simulation and viewer

- **Only the simulation thread touches MuJoCo state.** Viewer and HTTP callbacks only enqueue a `SimCommand` (`http_server.py`), which `MuJoCoRobot._handle_command` applies on the next control cycle. HTTP and viewer readouts go through snapshots (`_http_status`, `_panel_state`) taken under `_state_lock`. To add a control, add a command, handle it in `_handle_command`, then add its HTTP route and its input in `viser_controls.py`.
- **Viewer inputs ignore server-side events.** Setting a Viser input's `value` from the server fires its `on_update` callbacks with `client=None`. Panel callbacks skip those events, so syncing the panel from sim state never sends a command back.
- **mjviser is used partly through its internals.** The plugin builds the viewer tabs itself, and does not use mjviser's `create_scene_gui` or `create_visualization_gui`. mjviser's camera tracking follows an arbitrary body by shifting the world, and each call registers another client-connect hook. The plugin also sets `camera_tracking_enabled`, installs a refresh handler, and moves the fixed-body handles itself. mjviser is pinned to `<0.1`; after upgrading it, check a scene switch and camera follow in a browser, because the unit tests mock viser.
- **The driver owns services, it does not implement them.** `robot.py` steps physics and converts units through `channels.py`; the viewer (`viewer.py`), cameras (`cameras.py`), free objects (`scene_objects.py`) and live camera edits (`scene_watch.py`) are services it starts, syncs once per tick, and stops. None of them steps physics.
- **Floor scenes spawn floating bases.** They mark a robot's start with `<frame name="{prefix}robot_spawn"/>` (`SceneConfig.anchors="spawn"`). Composing moves the robot's root body to its keyframe height first, so the base-hold weld `mujoco_base_hold_{prefix}` takes the start pose as its reference; `floating.py` reads the base and detects falls.
- **Scene switches rebuild the viewer.** `ViewerService.build_gui` clears every GUI element and scene node, then builds them again for the new model. Viewer preferences that should survive a switch live on `SimControlPanel`, not on the per-build handles.
- **Scenes hold no robot.** The arm is MuJoCo Menagerie's SO-101 (`robotstudio_so101`), loaded through the pinned `mujoco-menagerie` package, which downloads it on first use. Each scene marks an arm's base with a `<frame name="{prefix}robot_mount"/>`, and `compose.compose_scene` attaches the profile's robot there with that name prefix (`left_`/`right_` for two arms). `profiles/so101.py` pins the joint ranges, force limits and wrist camera the plugin has always used; change the arm or its wrist camera there. Scenes list the profiles they support in `SceneConfig.robots`. Load scenes with `SceneConfig.load_model()`: `mujoco.MjModel.from_xml_path` on a scene file gives a model without the arm. `urdf/so101/*.urdf` and their meshes remain for Studio's 3D view.
- **Arm counts are laid out per scene.** A scene file lays out one arm count (`SceneConfig.written_arms`); the other comes from `SceneConfig.mounts` (pinned poses) or is derived from the single mount (`two_arm_style`: `side_by_side` or `across`, `two_arm_separation` scaled with reach), and `compose.apply_layout` moves the mount frames there. `home_qpos` applies to each arm with its prefix.
- **Camera images are streamed as MuJoCo renders them.** `mujoco.Renderer` already returns upright images, so set a camera's orientation in the scene XML rather than flipping frames in code. In a MuJoCo camera frame, `-z` is the viewing direction and `+y` is the top of the image. `mirror_horizontal` exists for setups that need a mirrored feed; the `start` command does not use it.

### Compatibility with Studio

Studio pins `physicalai` to a release (`v0.2.0` at the time of writing), and the plugin runs against that version when it is installed into Studio. Run the tests against it before changing how the plugin uses Runtime APIs:

```bash
git fetch origin tag v0.2.0
git worktree add /tmp/physicalai-v0.2.0 v0.2.0
uv venv -p 3.12 /tmp/physicalai-v0.2.0-venv
VIRTUAL_ENV=/tmp/physicalai-v0.2.0-venv uv pip install --no-sources "/tmp/physicalai-v0.2.0[transport]" \
  -e packages/physicalai-mujoco-plugin pytest pytest-asyncio httpx
/tmp/physicalai-v0.2.0-venv/bin/python -m pytest --import-mode=importlib packages/physicalai-mujoco-plugin/tests
git worktree remove /tmp/physicalai-v0.2.0
```

`--no-sources` keeps `uv` from replacing the pinned release with this checkout.

### Checks before a pull request

- Run the repository hooks on the files you changed (`prek run --files <files>`). `prek run --all-files` also reformats unrelated files elsewhere in the repository.
- The ruff hook runs with `--unsafe-fixes` and can rewrite code, for example by collapsing a lambda into a bound method. Run the tests again after the hooks.
- Tests that step a real simulation for seconds are marked `@pytest.mark.slow`. Run the quick set with `-m "not slow"` while iterating, and the full suite before pushing.
- The smoke matrix (`tests/test_smoke_matrix.py`) runs every supported profile in its default scene: it idles, holds the home pose and nudges the joints (each nudge larger than the tolerance, so dropped actions fail), and checks that the state stays finite, MuJoCo reports no instability and the joints track within 5° or 5 mm. Tendon grippers without a joint range (Panda, xArm7) are nudged within their `ctrlrange` and must cover half the nudge. The reachability test (`tests/test_reach.py`) checks every arm of a `layout="reach"` scene. Both are deselected by default; CI runs them in their own step. Run them after adding or changing a profile: `uv run pytest -m mujoco_smoke packages/physicalai-mujoco-plugin/tests`.
- Run the security scans used by CI:

  ```bash
  uvx bandit -c .github/bandit_config.yml -r packages/physicalai-mujoco-plugin/src
  SEMGREP_RULES="p/default p/cwe-top-25 p/trailofbits p/owasp-top-ten" uvx semgrep scan --metrics=off <changed files>
  ```

  Semgrep reports two existing `urlopen` findings in `tests/test_http_server.py`; CI only fails on new findings.

The package was imported from [physicalai-plugins PR #36](https://github.com/MarkRedeman/physicalai-plugins/pull/36) at `53bf606ef0f340a2a1821fc42a1ab6f44c33edf7`. See `NOTICE` for provenance and `CHANGELOG.md` for earlier releases. Release publishing uses this repository's shared workflows; the existing PyPI project needs a trusted publisher for `openvinotoolkit/physicalai` before its first release here.

## Current status and future improvements

Planned/desired improvements:

- More configurable camera presets (pose/FOV/fps via CLI or payload)
- Scene randomization presets (object layouts, target marker variants, textures)
- RTSP streaming sink (the frame-buffer plumbing is sink-agnostic; an RTSP
  server such as `aiortsp` or MediaMTX can consume the same buffers later)
- Optional richer sensor streams (depth/segmentation style outputs)
- Additional sample tasks and policy playback recipes in this package
