Metadata-Version: 2.4
Name: asmagic
Version: 1.0.1
Summary: Receive asMagic camera, motion, controller, body, hand and face protobuf streams
Author: ASMagic Team
License: MIT
Project-URL: Homepage, https://github.com/asMagicApp/asmagic-python-library
Project-URL: Documentation, https://github.com/asMagicApp/asmagic-python-library#readme
Project-URL: Repository, https://github.com/asMagicApp/asmagic-python-library
Project-URL: Issues, https://github.com/asMagicApp/asmagic-python-library/issues
Keywords: ar,imu,sensor,zmq,protobuf,streaming,face-tracking,hand-tracking
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.20.0
Requires-Dist: pyzmq>=22.0.0
Requires-Dist: protobuf>=4.25.0
Requires-Dist: opencv-python>=4.5.0
Requires-Dist: Pillow>=8.0.0
Requires-Dist: matplotlib>=3.5
Requires-Dist: av>=12.0.0
Provides-Extra: hevc
Provides-Extra: plot
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: build>=1.0.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Dynamic: license-file

# asmagic

Receive camera images, depth, motion, controller input, body joints, hands and face data from the [asMagic app](https://apps.apple.com/cn/app/asmagic/id6661033548) using Python.

[中文使用指南](README.zh-CN.md) · [Wire protocol and ports](docs/protocols.md) · [API reference](docs/api.md) · [Hand joints and window](docs/hands.md) · [Face and hand details](docs/face-and-hands.md) · [Upgrade notes](docs/compatibility.md) · [Release instructions](docs/releasing.md)

## Install

Use Python 3.8 or newer. Install **1.0.1** from PyPI:

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade asmagic==1.0.1
```

On Windows, use `python -m venv .venv`, then `.venv\Scripts\activate` in Command Prompt or `.\.venv\Scripts\Activate.ps1` in PowerShell. The default install includes image decoding and plotting dependencies; no extras or protoc are needed.

For development, run `python -m pip install -e .` from the directory containing `pyproject.toml`. AR and IMU use `frame = sub.receive()`, followed by direct field access.

## Connect your device

1. Connect the iPhone and computer to the same reachable network. Allow local-network access when prompted.
2. Open the feature you want in asMagic and enable its matching data output. Keep that feature active while receiving.
3. Put the **iPhone's IP address** into `IPHONE_IP` in your example, then run it. Pairing with the Mac app is not required.

**Watch works in the other direction:** Python starts an HTTP server, and the Watch sends to your **computer's IP address**. Start with the [Watch guide](docs/examples/watch.md).

## Choose your data

| I want to read… | Guide | Python receiver |
|---|---|---|
| Camera images, depth and pose | [AR data](docs/examples/ar.md) | `ARDataSubscriber` |
| iPhone motion sensors | [IMU data](docs/examples/imu.md) | `IMUDataSubscriber` |
| Controller buttons and sticks | [Joystick](docs/examples/joystick.md) | `JoystickDataSubscriber` |
| People and full-body joints | [Body tracking](docs/body-tracking.md) | `BodyTrackingDataSubscriber` |
| Left/right hands and finger joints | [Illustrated hand guide](docs/hands.md) | `HandPoseDataSubscriber` |
| Face landmarks and blendshapes | [Face data](docs/examples/face.md) | `FaceDataSubscriber` |
| Two-dimensional upper-body joints | [Upper body](docs/examples/upper_body.md) | `BodyPoseDataSubscriber` |
| AirPods head orientation | [AirPods](docs/examples/airpods.md) | `AirPodsDataSubscriber` |
| Gripper commands | [Gripper](docs/examples/claw.md) | `ClawDataSubscriber` |
| Apple Watch motion batches | [Watch](docs/examples/watch.md) | `WatchIMUReceiver` |

Each guide contains complete runnable files with editable settings at the top. Start from the [example directory](examples/documentation/README.md), edit `IPHONE_IP`, then run a file:

```bash
python examples/documentation/hand/01_both_hands.py
```

`receive()` waits for the next frame. It is normal for the terminal to wait until the App starts sending. Stop with **Ctrl-C**. Receive once, then read all needed fields from that frame; another receive call advances to the next frame. For timeouts and nonblocking reads, see [common receiving methods](docs/examples/common.md).

The [API reference](docs/examples/api-reference.md) lists fields, types and units; the [joint catalog](docs/examples/joint-catalog.md) lists every body and hand joint. Tracking examples use `SHOW_SKELETON = True/False` to control plotting.

## Start receiving

Open the matching feature in asMagic, enable its data output, and use the **sender's IP address**. ZeroMQ receivers connect to the iPhone; Watch HTTP is the reverse direction, described below.

```python
from asmagic import FaceDataSubscriber

IPHONE_IP = "192.168.1.100"  # Replace with your iPhone's IP address.


def main():
    sub = FaceDataSubscriber(IPHONE_IP)
    try:
        while True:
            frame = sub.receive()
            print("Time:", frame.timestamp)
            print("Face detected:", frame.has_face)
            print("Face points:", frame.points)
            print("Expressions:", frame.blendshapes)
            print("Position (m):", frame.camera_position_meters)
            print("Orientation:", frame.camera_orientation)
    except KeyboardInterrupt:
        pass
    finally:
        sub.close()


if __name__ == "__main__":
    main()
```

TrueDepth face tracking supplies 52 ARKit blendshape channels and metric face mesh geometry. Vision-only devices provide 2D landmarks without blendshapes. Missing hands are `None`, and a lost face has `has_face == False`. These are facial movement scores, not emotion labels.

## Read body data by person and joint name

```python
from asmagic import BodyTrackingDataSubscriber

IPHONE_IP = "192.168.1.100"  # Replace with your iPhone's IP address.
MODEL = "mediapipe"  # Match the App setting: "mediapipe" or "arkit".


def print_person_joints(person):
    print(f"\nPerson {person.label} (ID: {person.id})")
    # Read both sides of the upper body and legs by their anatomical names.
    named_joints = [
        ("Left shoulder", person.joints.left_shoulder),
        ("Right shoulder", person.joints.right_shoulder),
        ("Left elbow", person.joints.left_elbow),
        ("Right elbow", person.joints.right_elbow),
        ("Left wrist", person.joints.left_wrist),
        ("Right wrist", person.joints.right_wrist),
        ("Left hip", person.joints.left_hip),
        ("Right hip", person.joints.right_hip),
        ("Left knee", person.joints.left_knee),
        ("Right knee", person.joints.right_knee),
        ("Left ankle", person.joints.left_ankle),
        ("Right ankle", person.joints.right_ankle),
    ]
    for location, body_joint in named_joints:
        if body_joint is None:
            print(f"  {location}: joint unavailable")
            continue
        print(f"  {location}: body [x, y, z] (m)={body_joint.position}")
        print(f"  {location}: world [x, y, z] (m)={body_joint.world_position}")
        print(f"  {location}: rotation [qx, qy, qz, qw]={body_joint.rotation}")
        print(f"  {location}: usable={body_joint.usable}")


def main():
    sub = BodyTrackingDataSubscriber(IPHONE_IP, model=MODEL)
    try:
        while True:
            frame = sub.receive()
            print("\nPeople:", frame.person_count)
            if not frame.people:
                print("No person detected in this frame")
            for person in frame.people:
                print_person_joints(person)
    except (KeyboardInterrupt, StopIteration):
        pass
    finally:
        sub.close()


if __name__ == "__main__":
    main()
```

Use the same `frame.people`, `person.joints.left_wrist` and `person.skeleton()` API for ARKit and MediaPipe. MediaPipe preserves all 33 landmarks per person; ARKit preserves all 91 native nodes. The common layout gives twelve fixed anatomical rows. Missing positions and rotations remain explicit. See the [illustrated body guide](docs/body-tracking.md) for complete definitions, model selection and person identity.

## Subscribers

All receivers use `.receive()` to wait for one valid typed frame, or `.receive(timeout=1000)` to wait up to 1000 **milliseconds** and return `None` on timeout. `timeout=0` polls; `-1` or `None` waits indefinitely. Closure raises `StopIteration` in both modes. Watch uses the same method and returns `WatchBatch`. Iteration, `.stream()`, `.close()` and context managers are supported; context managers close receivers automatically.

Basic AR examples use `ARDataSubscriber(ip)`. Its full signature is `ARDataSubscriber(ip, port=8000, hwm=32, conflate=False, verbose=False)`; these defaults buffer up to 32 decoded frames. Other ZeroMQ subscribers use `Subscriber(ip, port=..., hwm=1, conflate=True, verbose=False)`.

| Import from `asmagic` | Default port | Data |
|---|---:|---|
| `ARDataSubscriber` | 8000 | RGB, depth, confidence, camera pose/intrinsics, frame and session metadata |
| `AirPodsDataSubscriber` | 8001 | Head orientation quaternion |
| `IMUDataSubscriber` | 8002 | iPhone motion sensors |
| `BodyPoseDataSubscriber` | 8010 | Vision 2D upper-body positions and angles |
| `JoystickDataSubscriber` | 8020 | Legacy axes/buttons plus full controller state |
| `BodyTrackingDataSubscriber(model="arkit")` | 8100 | Full native ARKit body through the people/joint API |
| `BodyTrackingDataSubscriber(model="mediapipe")` | 8104 | Complete 33-landmark body data for every detected person |
| `HandPoseDataSubscriber` | 8101 | Shared left/right hand channel from Stream or Face and Hand Tracking |
| `FaceDataSubscriber` | 8102 | Face-only pose, 52 scores and mesh |
| `ClawDataSubscriber` | 5301 | Robot gripper output; pass `port=5401` for the second gripper |

**Face and hands are independent:** enable Hand tracking and tap Send hands, then use `HandPoseDataSubscriber(ip)` on 8101 from either Stream or Face and Hand Tracking. Enable Face tracking and tap Send face to use `FaceDataSubscriber(ip)` on 8102. Stopping one output leaves the other running. Disabling a tracking toggle stops its recognition and output. The current face message has no hands. See [hand examples](docs/examples/hand.md), [face examples](docs/examples/face.md) and [reading both independently](docs/examples/face.md#example-5-receive-hands-from-face-and-hand-tracking). Older combined App packets remain supported by the legacy `FaceAndHandTrackingDataSubscriber`; current code should use the two separate receivers.

Legacy `MagiClawDataSubscriber` and `LegacyWatchDataSubscriber` require an explicit port. `ProtobufSubscriber` accepts a custom generated message class. Every typed frame exposes `.protobuf` for exact fields and unknown-field preservation.

### Existing camera API

```python
from asmagic import ARDataSubscriber

IPHONE_IP = "192.168.1.100"  # Replace with your iPhone's IP address.


def main():
    sub = ARDataSubscriber(IPHONE_IP)
    try:
        while True:
            frame = sub.receive()
            print("Time:", frame.timestamp)
            print("Frame ID:", frame.frame_id)
            print("Capture time:", frame.source_timestamp)

            print("RGB:", frame.color_array)
            print("RGB size:", frame.rgb_width, frame.rgb_height)
            print("Depth (raw):", frame.depth)
            print("Depth size:", frame.depth_width, frame.depth_height)
            print("Depth confidence:", frame.depth_confidence_array)
            print("Depth scale (m/unit):", frame.depth_meters_per_unit)

            print("Velocity (m/s):", frame.velocity)
            print("Local pose:", frame.local_pose)
            print("Global pose:", frame.global_pose)
            print("Local transform:", frame.local_pose_matrix)
            print("Global transform:", frame.global_pose_matrix)
            print("Intrinsics:", frame.camera_intrinsics)
            print("Intrinsics matrix:", frame.camera_intrinsics_matrix)

            print("Session:", frame.session_identifier)
            print("Shared world:", frame.shared_world_identifier)
            print("Global pose is shared:", frame.global_pose_is_shared)
            print("Has RGB / depth / pose:", frame.has_color_image, frame.has_depth_image, frame.has_pose)
    except KeyboardInterrupt:
        pass
    finally:
        sub.close()


if __name__ == "__main__":
    main()
```

`ARDataSubscriber` receives and decodes HEVC on a dedicated thread per connection, independently of display or user processing. RGB array conversion happens when `color_array` is accessed. `conflate=True` keeps only the newest decoded frame for preview; it never drops encoded reference packets. With `conflate=False`, up to `hwm` decoded frames are retained, replacing the oldest when full. Color arrays can be `None` until a key frame arrives after joining, packet loss or a session change. JPEG arrays are decoded on demand. Offline `ARDataSubscriber.decode()` can decode an isolated HEVC key frame, but a P-frame needs stream history.

`color_img` / `color_bytes` remain raw bytes. They are not always JPEG: inspect `rgb_codec_name`. `frame.depth` stays UInt16 in sender units.

### Full controller state

```python
from asmagic import JoystickDataSubscriber

IPHONE_IP = "192.168.1.100"  # Replace with your iPhone's IP address.


def main():
    sub = JoystickDataSubscriber(IPHONE_IP)
    try:
        while True:
            data = sub.receive()
            print("Time:", data.timestamp)
            print("Left joystick:", data.left_x, data.left_y)
            print("Right joystick:", data.right_x, data.right_y)
            print("Triangle:", data.button.triangle)
            print("Square:", data.button.square)
            print("Cross:", data.button.cross)
            print("Circle:", data.button.circle)
            print("L1/R1:", data.button.l1, data.button.r1)
            print("L2/R2:", data.button.l2, data.button.r2)
            print("L3/R3:", data.button.l3, data.button.r3)
            print("D-pad up/down:", data.button.up, data.button.down)
            print("D-pad left/right:", data.button.left, data.button.right)
            print("Share/Options:", data.button.share, data.button.options)
            print("Home/Touchpad:", data.button.home, data.button.touchpad)
            print("Controller:", data.controller_name)
            print("Connected:", data.is_connected)
            print("Battery:", data.battery_level)
            print("Charging:", data.is_charging)
            if data.has_motion:
                print("Gravity (G):", data.gravity)
                print("Acceleration (G):", data.user_acceleration)
                print("Rotation rate (rad/s):", data.rotation_rate)
                print("Attitude:", data.attitude)
    except KeyboardInterrupt:
        pass
    finally:
        sub.close()


if __name__ == "__main__":
    main()
```

Axes are returned unchanged. Current asMagic uses **−1…1, neutral 0**. Older normalized arrays are not rescaled.

### Watch HTTP batches

```python
from asmagic import WatchIMUReceiver



def main():
    sub = WatchIMUReceiver(host="0.0.0.0")
    try:
        while True:
            frame = sub.receive()
            for sample in frame.frames:
                print("Time:", sample.timestamp)
                print("Attitude:", sample.attitude)
                print("Rotation rate (rad/s):", sample.rotation_rate)
                print("Gravity (G):", sample.gravity)
                print("Acceleration (G):", sample.acceleration)
    except KeyboardInterrupt:
        pass
    finally:
        sub.close()


if __name__ == "__main__":
    main()
```

The receiver accepts `HEAD` warm-up requests and `POST /imu` batches. Default `conflate=True` keeps the newest **batch**. `conflate=False` keeps up to `max_batches=32`; it is still a bounded, best-effort queue. Use `payload_format="single"` or `"legacy"` only for explicitly selected older Watch senders.

## Reliability and diagnosis

- `is_connected` means the local socket/server is open. It does not prove that a device is reachable. `has_received_data`, `received_frames`, `last_received_at` and `last_error` describe actual receive activity.
- The asMagic feature must be open and its matching output enabled. Camera, body and face features may compete for the camera; each port is a separate stream.
- PUB/SUB delivery is best effort. A new subscription needs time to establish and can miss the first packet. Default conflation intentionally drops old messages for live use.
- Use a single thread per ZeroMQ subscriber; create independent subscribers for separate threads. Watch's HTTP queue supports receiving on its own server thread.
- A timeout means no frame arrived during that interval. Check feature output, sender IP, matching port, local-network access and firewall rules. Pairing with the Mac app is not required for a Python receiver.

For a guided check with a saved JSON report:

```bash
asmagic-test --list
asmagic-test hand --ip 192.168.1.100 --plot
asmagic-test --all --ip 192.168.1.100 --plot --images
```

The tool prompts you to enable each selected feature, lets you skip unavailable hardware, and distinguishes missing packets from empty tracking results. The [device testing guide](docs/device-testing.md) explains the report. For MediaPipe body tracking, use the [body guide](docs/body-tracking.md); the diagnostic `ar_body` entry receives the ARKit stream.

## Development

```bash
python -m pip install -e '.[dev]'
python -m pytest
python tools/sync_documentation_examples.py --check
python tools/generate_protos.py --check  # requires system protoc or grpcio-tools
python -m build
python -m twine check dist/*
```

Tests use synthetic protobuf messages and local TCP/HTTP publishers. Physical iPhone/Watch detection and device-to-computer delivery still require a device check. See the [protocol audit](docs/protocols.md) for sender locations and timestamp/coordinate semantics.
