Metadata-Version: 2.5
Name: hypergym
Version: 0.1.7
Summary: The fastest drone simulator and renderer (raytracer).
Author-email: Jonas Eschmann <jonas.eschmann@gmail.com>
License: MIT
Requires-Python: >=3.9
Requires-Dist: conta
Requires-Dist: nanobind>=2.0
Requires-Dist: numpy
Provides-Extra: examples
Requires-Dist: imageio; extra == 'examples'
Requires-Dist: imageio-ffmpeg; extra == 'examples'
Requires-Dist: matplotlib; extra == 'examples'
Requires-Dist: scipy>=1.15; extra == 'examples'
Provides-Extra: gym
Requires-Dist: gymnasium>=1.0; extra == 'gym'
Description-Content-Type: text/markdown

# hyperdrone

Drone simulation stack for RLtools: raytracing renderer, L2F multirotor dynamics, and
environment setup under one Python package.

| Subpackage | What it is |
|---|---|
| `hyperdrone.render` | Raytracing renderer (OptiX / Metal / Vulkan / generic CPU). Self-contained — no dependency on drone dynamics. |
| `hyperdrone.dynamics` | Vectorized, stateful L2F multirotor simulator (`Sim`); cpu and cuda variants. |
| `hyperdrone.env` | The RL environment: the C++ `MultiEnvironment<hyperdrone::World>` driven through the exact rl_tools batch verbs. |
| `hyperdrone.gym` | Optional Gymnasium `VectorEnv` adapter over `hyperdrone.env` (`pip install "hyperdrone[gym]"`). |
| `hyperdrone.jit` | Shared compile-and-cache infrastructure both domain packages build on. |
| `hyperdrone.cuda` | CUDA staging helpers (`upload` → DLPack tensor sets). |

Native components are JIT-compiled per set of compile-time constants (resolution, camera
count, drone count, ...) into a persistent cache; later uses load the cached library
directly. `render` and `dynamics` are peers that never import each other — the coupling
surface is a plain DLPack tensor of packed camera bases, device-resident on OptiX + CUDA.
Rendering uses a shared scene module and a directly bound Python extension per renderer
configuration; scene and asset-pool objects can be reused across configurations. Environment
configurations are also directly bound extensions, sharing their C++ implementation with
the golden-rollout generator.

## Notebooks

| Notebook | Open in Colab |
|---|---|
| [HyperDrone](hyperdrone/examples/HyperDrone.ipynb) | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/rl-tools/rl-tools-internal/blob/render/interface/python/hyperdrone/hyperdrone/examples/HyperDrone.ipynb#scrollTo=g-cDv8bdN7j9) |
| [Benchmark](hyperdrone/examples/Benchmark.ipynb) | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/rl-tools/rl-tools-internal/blob/render/interface/python/hyperdrone/hyperdrone/examples/Benchmark.ipynb) |
| [Custom](hyperdrone/examples/Custom.ipynb) | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/rl-tools/rl-tools-internal/blob/render/interface/python/hyperdrone/hyperdrone/examples/Custom.ipynb) |
| [Orbiter](hyperdrone/examples/Orbiter.ipynb) | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/rl-tools/rl-tools-internal/blob/render/interface/python/hyperdrone/hyperdrone/examples/Orbiter.ipynb) |

## Install

From an rl-tools checkout (development):

```bash
.venv/bin/pip install -e interface/python/conta -e interface/python/hyperdrone
```

From an sdist (self-contained — bundles the rl-tools headers and raytracing sources, no
checkout needed):

```bash
pip install hyperdrone-<version>.tar.gz
```

Install the dependencies used by the packaged examples with:

```bash
pip install "hyperdrone[examples]"
```

Requirements: CMake >= 3.24, a C++17 compiler, and git for dependency downloads.
GLB loading uses the Assimp 6.0.4 release archive with a pinned SHA-256, shared by render and env builds. Cached archives are reused offline and verified by CMake before extraction.
CUDA + OptiX driver is required for the OptiX render backend and cuda dynamics,
Vulkan dev + glslang for the VULKAN backend. The WEBGPU backend fetches a hash-pinned
wgpu-native prebuilt at first build (no extra system packages). Nothing beyond the
build toolchain for GENERIC + cpu.

## Environment

| Variable | Meaning |
|---|---|
| `HYPERDRONE_RENDER_BACKEND` | `OPTIX` \| `METAL` \| `VULKAN` \| `WEBGPU` \| `GENERIC` \| `AUTO` (default: Metal on macOS, OptiX elsewhere) |
| `HYPERDRONE_DYNAMICS_DEVICE` | `CPU` \| `CUDA` \| `AUTO` (default: cuda when available) |
| `HYPERDRONE_CACHE_DIR` | root for CMake build trees and downloaded build dependencies (default `~/.cache/hyperdrone`) |
| `HYPERDRONE_RLTOOLS_ROOT` | rl-tools source root override (default: enclosing checkout, else the vendored tree) |
| `HYPERDRONE_PROCTHOR_PATH` | local ProcTHOR example scene override (otherwise downloaded and cached) |
| `HYPERDRONE_SKIP_BUILD` | skip the CMake staleness check when the artifacts already exist |
| `HYPERDRONE_OFFLINE` | forbid network during builds (requires seeded/vendored dependencies) |
| `HYPERDRONE_BUILD_JOBS` | parallel build jobs (default 5) |

Each component and variant gets its own build tree (`render-optix-*`, `dynamics-cuda-*`,
...); switching never invalidates another's cache. All configure/build steps run under a
per-tree file lock, so many worker processes can share one cache safely. CMake
FetchContent sources and native build state live in `<cache>/.dependencies`; the native
build does not write generated files into an editable checkout or installed package.

Whenever a renderer is successfully allocated, RLtools reports the selected backend on
stderr, for example `#rl_tools::rendering::raytracing: backend=metal`. The
`Renderer.backend` property provides the same lowercase name programmatically.

## Rendering

```python
import numpy as np
from hyperdrone import render
from hyperdrone.examples.data import procthor_scene_path

scene = render.load_scene(procthor_scene_path(), fidelity="high")

renderer = render.Renderer(width=320, height=240, num_cameras=4, output="rgbd", fidelity="high")
renderer.init(scene)

camera = renderer.camera(position=(0, 0, 1.5), look_at=(1, 0, 1.5), fov=80)
renderer.set_cameras(np.repeat(camera[None], 4, axis=0))
renderer.render()

rgb   = renderer.frame()    # (4, 240, 320, 4) uint8
depth = renderer.depth()    # (4, 240, 320) float32
```

The renderer operates in the L2F FLU frame: +X forward, +Y left, +Z up; GLB assets are
swizzled at load time. Procedural scenes (`render.Scene()` + `render.Object` /
`render.Mesh` / `render.SceneLight`), segmentation output, overlays (per-camera dynamic
content via `AssetPool` + `spawn`/`attach`), motion blur, and anti-aliasing follow the
same API as before under `hyperdrone.render.*`.

Converted ProcTHOR scenes carry ambient `(0.6, 0.6, 0.6)` and a black background in
`scenes[].extras.rl_tools.environment` (`mode: "solid"`, `ambient` and `background`
are nonnegative linear RGB triples). Scene loading imports these settings; importing
objects or assemblies leaves the parent environment alone. Ambient affects `high`
and `veryhigh`, approximating indirect illumination without light bounces. Override
imported settings with `scene.set_environment(...)` before `renderer.init(scene)`.

`render()` produces all image outputs selected at construction; `render_launch()` and `render_sync()` split submission from waiting. Collision probes run separately through `probe()` or `probe_launch()` / `probe_sync()`, after `generate_probe_directions()`, and are read with `collisions()`. `num_probes` sets their count per camera (default 1); zero compiles the probe pass out and makes probe operations raise.

### Zero-copy I/O (DLPack)

```python
renderer.frame()                  # snapshot copy (safe to keep)
renderer.frame(copy=False)        # numpy view of the staging buffer (refreshed in place)
rgba = torch.from_dlpack(renderer.frame_dlpack())
# live uint8 (num_cameras, height, width, 4), zero-copy GPU tensor on OptiX
packed = torch.from_dlpack(renderer.frame_raw_dlpack())
# same memory as packed uint32 (num_cameras, height, width)
```

Inputs accept any DLPack producer. CUDA-resident camera input (OptiX):
`set_cameras` dispatches on `__dlpack_device__`, so a CUDA tensor — a torch GPU tensor,
a set pre-uploaded with `hyperdrone.cuda.upload`, or `Sim.camera_bases()` — is handed
over device-to-device on the render stream, fully async; `stream=` takes the producer's
cudaStream_t handle for event-ordered hand-off.

## Dynamics

```python
from hyperdrone import dynamics

sim = dynamics.Sim(num_drones=4096, model="x500_sim", device="cuda")
sim.reset(seed=0)                     # deterministic; host-sampled, identical on cpu/cuda
sim.step(actions)                     # (N, 4) float32 in [-1, 1], host or CUDA tensor
sim.state["position"]                 # zero-copy DLPack views, CUDA-resident on cuda
sim.state.numpy("position")           # host copy
sim.observe()                         # (N, 18): position, rotation matrix, velocities
sim.parameters["mass"] = masses       # per-drone runtime parameters (until the next reset)
sim.model_parameters()                # the preset's nominal l2f dynamics: rotor layout, mass, J, ...

cameras = sim.camera_bases(fov=100, aspect=renderer.aspect)
renderer.set_cameras(cameras, stream=sim.stream)   # device-resident hand-off
```

`num_drones` and domain randomization are compile-time (the JIT key); the model preset
(`crazyflie`, `x500_real`, `x500_sim`, `mrs`, ...), integration `dt`, and all physical
parameters are runtime. Reward and termination live in the
environment (`hyperdrone.env`), not the simulator. One process can use one dynamics
variant (cpu or cuda).

## RL environment

```python
import numpy as np
from hyperdrone.env import EnvConfig, MultiEnvironment

config = EnvConfig(num_environments=1, instances=1024, cam_width=32, cam_height=32, task="target_frame")
env = MultiEnvironment("scenes/", config=config, seed=0)

mask = np.ones(env.total_instances, dtype=bool)
env.reset(mask)                    # resample parameters + states where mask is set
env.render(mask)                   # render the FPV cameras (mask marks fresh episodes)
observations = env.observe()       # (total, observation_dim) float32
env.step(actions)                  # (total, action_dim) float32 in [-1, 1]
rewards, terminated = env.rewards(), env.terminated()
critic_input = env.observe_privileged()
env.rotate_scene()                 # deterministic scene rotation; reset all instances after
env.observation_layout             # named blocks: which channels/values mean what
```

`reset(mask)` samples parameters and states immediately. `observe()` renders as needed,
so explicit `render(mask)` calls are optional. The training loop owns episode counters and
chooses when to reset; the Gymnasium adapter below supplies same-step autoreset and time limits.

`EnvConfig` is the compile-time part (the JIT key); the runtime part is `MultiEnvironment(..., parameters={"dynamics": {"dynamics": {"model": "x500", "mass": 2.1}}, "camera": {"fov": 90}})`, a partial dict merged over the nominal parameters as the base of every drone (the specification's randomization ranges apply on top at reset). `env.parameters` returns drone zero's resolved nominal parameters, including its entry-specific overrides: `"dynamics"` is the l2f parameter set, where a `"model"` key loads an l2f registry entry (`hyperdrone.dynamics.MODELS`) before the rest is merged, and `"camera"` holds `mount.pose` (`position` and `orientation`, a (w, x, y, z) quaternion mapping the camera's FLU axes into the body frame) and the horizontal `fov` in degrees. `env.instance_parameters()` returns what each instance is using in the current episode (after randomization), and `env.set_instance_parameters(overrides, mask)` changes it until that instance's next reset (one dict for all selected instances or one per instance); only the drone itself varies per instance (`dynamics.dynamics`, `dynamics.imu`, `camera`).

Several drones fly in one environment through the drone table, `MultiEnvironment(..., drones=[{"asset": x500, "parameters": {...}}, {"asset": crazyflie, "parameters": {"dynamics": {"dynamics": {"model": "crazyflie"}}}}, ...])`: every instance samples one entry uniformly at each reset, independently of the scene its environment shows, so every scene meets every drone. An entry pairs the body/prop_* GLB rendered into the instance's cameras with a partial parameter dict merged over the shared constructor base (the per-drone keys `dynamics.dynamics`, `dynamics.imu`, `camera`); the same GLB may back several dynamics entries and distinct GLBs load once. `EnvConfig(max_drones=...)` is the table capacity (compile-time, default 1), `env.drones` returns the resolved native table, and the `"drone"` value in `env.instance_parameters()` indexes it. `env.self_visible` and `env.max_drones` report the compiled world's capabilities, including custom `spec_header` worlds. `drone_asset=X` is the one-entry table. The rigged assemblies of `hyperdrone.examples.data.RIGGED_MODELS` (`arpl`, `crazyflie`, `crazyflie_brushless`, `soft`, `x500`) pair with the l2f registry entries of the same names; each prop spins with the dynamics rotor at its position whatever rotor order the entry uses.

All environment semantics — reset, reward, termination, scene scheduling, observation
composition — live on the C++ side (`rl_tools::rl::environments::hyperdrone::MultiEnvironment<World>`);
the binding marshals tensors and nothing else, and a seeded rollout is pinned bit-exact
against the C++ verbs by a golden test. The scenes argument is a directory of `.glb`
scenes (sorted corpus) or a list of references — `.glb` paths, `conta:HASH` strings, or
conta store manifest entries `{"description": ..., "hash": ...}` — used as the corpus in
list order and partitioned across environments; conta references (scenes, `drone_asset=`,
`gate_asset=`) resolve through the `conta` package into the shared content-addressed
cache, so a config pins the exact corpus on any machine. Configuration follows the C++
extension ladder:
`preset=` names the platform (`"crazyflie"`, `"x500_fpv"` — SELF_VISIBLE, pass a
body/prop_* GLB via `drone_asset=`; `hyperdrone.env.Rotorcraft(assembly, rotor_positions=...,
rotor_torque_directions=...)` is the same rig as a `render.Rig` for manual composition: every
`prop_*` part spins about its hub, and given the dynamics rotor layout (`sim.model_parameters()`)
coordinate i is the phase of rotor i spinning against its reaction torque, so `sim.state["rpm"]`
integrates straight into the coordinates; without rotors the props follow part order and the
Quad-X convention, front-right CCW — and `"x500_fpv_imu"`, the same platform stepping at
IMU rate), `task=` names the wrapper (`"target_frame"`, `"moving_gate"`,
`"visual_inertial_localization"`), `n_agents=` enables multi-agent, and `EnvConfig(spec_header=...)` pins
an arbitrary C++ specification (a header defining `hyperdrone_env_user::WORLD`, hashed
into the JIT key). A spec header can go beyond constants to a full user-authored task
wrapper — verb overloads registering and moving entities, compiled reward, termination:
`hyperdrone/examples/Orbiter.ipynb` walks through one, and its contract is pinned by
`tests/env/user_task_header.h`. The environment's raytracing backend follows
`HYPERDRONE_RENDER_BACKEND` like the renderer.

Manual composition of `Sim` and `Renderer` (no MDP — rendering research, data
generation) remains first-class: see `python -m hyperdrone.examples.drone_flythrough`
for the wiring. `FreeSpaceSampler` lives in `hyperdrone.render` — it rides the
renderer's collision probes and is deterministic given a seed.

### Visual-inertial localization

`task="visual_inertial_localization"` on `preset="x500_fpv_imu"` is the localization
benchmark: fixed-length synchronized episodes stepping at IMU rate (`env.dt`), a camera
frame every `env.frame_stride` steps, and the IMU sample of each step from
`env.observe_imu()` (named by `env.observation_layout_imu`: accelerometer, gyroscope,
frame age, new-frame flag). The task is autonomous: the RAPTOR autopilot built into it
flies a waypoint route (its checkpoint is pinned by content hash and fetched through
conta on first use), so `action_dim` is 0 and `step()` takes no actions. The privileged
observation carries the current `waypoint` next to the dynamics state; the drone's own
body is not rendered, matching the C++ benchmark harness. Resets must be all-or-none per
environment.

```python
config = EnvConfig(instances=2, cam_width=160, cam_height=120, preset="x500_fpv_imu", task="visual_inertial_localization")
env = MultiEnvironment(scenes, config=config, seed=0)
env.reset(); env.render(np.ones(env.total_instances, dtype=bool))
for step in range(env.episode_step_limit):
    if step > 0: env.render()
    if step % env.frame_stride == 0:
        frames = env.frames()          # the estimator's camera frame
    env.step()
    imu = env.observe_imu()            # the estimator's IMU sample for this step
    truth = env.observe_privileged()   # position / rotation matrix / velocities / waypoint
```

`python -m hyperdrone.examples.visual_inertial_localization` runs an episode and scores
IMU dead reckoning against the ground truth, the same protocol as the C++ demo.

### Gymnasium

```python
from hyperdrone.gym import VectorEnv   # pip install "hyperdrone[gym]"
env = VectorEnv("scenes/", config=EnvConfig(instances=1024), seed=0)
observations, infos = env.reset()
observations, rewards, terminations, truncations, infos = env.step(actions)
```

The adapter resets instances that terminate or reach `episode_step_limit` within the same
step. Returned observations start the new episode for those instances; no final observation
is exposed. `truncations` flags time limits that are not terminations. The core packages
never import gymnasium.

End-to-end example: `python -m hyperdrone.examples.drone_flythrough`; renderer benchmark:
`python -m hyperdrone.examples.benchmark` (flag-compatible with the C++ benchmark
counterpart).

## Publishing

From the repository root, build and upload a release with:

```bash
interface/python/hyperdrone/scripts/publish.sh
```

Set the release version in `pyproject.toml` and `hyperdrone/__init__.py` first. The
script uses the repository's `.venv` (creating it if needed), installs the build and
upload tools, builds a bundled source archive and a `py3-none-any` wheel from that
archive, checks both with Twine, and uploads both to PyPI. Authenticate through your
existing Twine configuration (`~/.pypirc`, keyring, or `TWINE_PASSWORD`), or enter a
PyPI API token when prompted.

Use `interface/python/hyperdrone/scripts/publish.sh --build-only` to build and check
without uploading. Each run keeps its two artifacts in a fresh
`interface/python/hyperdrone/dist/release.*` directory, so older builds are never
included in an upload. Native components remain JIT-compiled on first use.

## Tests

```bash
HYPERDRONE_RENDER_BACKEND=GENERIC .venv/bin/python -m pytest interface/python/hyperdrone/tests -v
```

`tests/jit` exercises the build infrastructure against a toy component in seconds;
`tests/test_architecture.py` enforces the import DAG (render and dynamics never see each
other); `scripts/test_sdist.sh` is the self-containment rot guard (build sdist → clean
venv → vendored-tree render).

For labelled ProcTHOR rendering, preserve GLB root instances and select the output meaning:

```python
scene = render.load_scene(procthor_scene_path(), rgb=False, preserve_instances=True)
renderer = render.Renderer(width=512, height=512, output="segmentation",
                           semantic_segmentation=True)  # False selects instance IDs
renderer.init(scene)
# Set cameras and render as usual.
labels = renderer.segmentation()  # uint32; renderer.segmentation_mode identifies the mode
classes = scene.segmentation_classes  # {class_id: name}, taxonomy: scene.segmentation_taxonomy
```

Semantic IDs come from AI2THOR-Hab's taxonomy: unknown is `0` (including Ceiling, which has no dataset category); background is `0xFFFFFFFF` in both modes. Instance IDs identify scene instances: use `scene.instance_object(id)` then `scene.object_name(object_index)` for static hits. Overlay instance IDs use the renderer's global slot layout. Semantic IDs must never be used as object indices. Loading remains welded by default for RGB; semantic rendering rejects a welded object containing multiple classes. Assemblies and asset-pool copies retain labels and the taxonomy. Composing conflicting taxonomies requires an explicit remapping before semantic rendering.

Assign labels before `Renderer.init()`: `object.segmentation_class = id`, `scene.set_object_segmentation_class(index, id)`, `assembly.set_segmentation_class(index, id)`, or `pool.set_object_segmentation_class(asset, object_index, id)` retains the current taxonomy and explicitly makes that object's class valid, including welded objects. Assignment rejects IDs absent from that taxonomy. `object.assign_segmentation_class(id, taxonomy="custom", classes={0: "Unknown", 7: "Furniture"})` replaces the taxonomy; omitting taxonomy/classes selects unnamed numeric IDs.

Objects, scenes, assemblies and pools expose `remap_segmentation_classes(mapping, taxonomy="custom", classes={...})`; pools also expose `remap_asset_segmentation_classes(asset, mapping, ...)`. Omitting taxonomy/classes selects unnamed numeric IDs. Mappings must cover every used class, and validation failures leave labels unchanged. Remapping preserves valid class boundaries and rejects mixed-class welded objects; explicitly assign one class first if collapsing their labels is intentional. Remap conflicting sources separately before composition. The C++ equivalents are device-first `assign_segmentation_class` and `remap_segmentation_classes` in `rendering/segmentation_cpu.h`, with a shared immutable target taxonomy (or `nullptr` for unnamed IDs).

### Articulated assets

`render.Rig` is a rigid-link tree: `world_link = world_parent @ origin @ joint(q)` and
`world_geometry = world_link @ offset`, with links declared parent before child (parent `-1`
is the base pose) and joints `fixed`, `revolute` (radians about `axis`), `prismatic` (metres
along `axis`) or `spherical` (wxyz quaternion). `rig.coordinate_slices` locates each link's
coordinates, `rig.default_coordinates` is the neutral state. `Rig.from_assembly(assembly,
joints={"slider": ("prismatic", (0, 1, 0))})` builds one link per part at its rest frame and
articulates parts by name; `load_assembly(path, part_nodes=[...])` keeps nested GLB nodes as
independent parts (an enclosing node first) and records their parent for that. Procedural
assets grow with `ObjectAssembly.add_part(object, transform)`. The drone rig the environment
renders is `hyperdrone.env.Rotorcraft`, a `Rig` built from the body/prop_* convention (see the
RL environment section).

```python
renderer.set_transform(overlay, placement, body, rig=rig, coordinates=phases)
renderer.set_transform_pair(overlay, placement, body_open, body_close, rig=rig,
                            coordinates_open=phases_open, coordinates_close=phases_close)
renderer.update()
renderer.render()
```

With a rig, the rigid verbs take the rig's base pose and joint coordinates (default: the
neutral state) and place the parts at its geometry-to-world poses; the pair form samples the
shutter (the base follows the constant-twist screw, the coordinates interpolate before the
kinematics, so a revolute joint keeps every revolution). The same poses are available as
arrays: `rig.evaluate(base, q)` is `(num_parts, 3, 4)` for `set_transforms`, `rig.sample(base_open,
base_close, q_open, q_close, renderer.motion_blur_samples)` the `(samples, parts, 3, 4)` motion plus
the shutter-close poses for `set_motion_transforms`. Batches of identical assets, one per
overlay, submit the whole tensor at once with `set_all_transforms(poses)` of shape
`(num_overlays, max_overlay_instances, 3, 4)` (and `set_all_motion_transforms`); `rig.evaluate`
accepts batched `(B, 3, 4)` bases for that. On OptiX every transform verb also takes CUDA
tensors (`stream=` orders the copy after the producer), and `renderer.transforms_dlpack()` /
`transforms_motion_dlpack()` expose the live input tensors for in-place producers ordered
against `renderer.render_stream`; publish pending host writes with `update()` before writing
them. `python -m hyperdrone.examples.articulation` renders a spinning hinge and a sliding block
with motion blur; `hyperdrone/examples/Custom.ipynb` flies the x500 with spinning props.
