Metadata-Version: 2.5
Name: zrb-face
Version: 0.4.0
Summary: A 3D character for zrb in a small, see-through, always-on-top window
Project-URL: Homepage, https://github.com/state-alchemists/zrb-face
Requires-Python: >=3.10
Provides-Extra: zrb
Requires-Dist: zrb; extra == 'zrb'
Description-Content-Type: text/markdown

# zrb-face

A companion for [zrb](https://github.com/state-alchemists/zrb): a 3D character in a small, see-through, always-on-top window. She idles naturally (breathing, shifting her weight, glancing around, stretching), shows what zrb is doing and how it feels, moves her mouth with zrb's voice, dances while you play music, gestures with your own animations, and can start a conversation herself.

## Quick start

```sh
pipx install zrb-face          # or: pip install zrb-face, next to zrb; from a checkout: pip install -e .

# once, e.g. in ~/.zshrc: load the zrb side into every zrb chat
export ZRB_INIT_SCRIPTS="$ZRB_INIT_SCRIPTS:$(zrb-face init-path)"

mkdir -p ~/.config/zrb-face/models && cp my.vrm ~/.config/zrb-face/models/   # a model: see Changing the model
zrb-face my                    # (re)start the server, open a window with my.vrm
zrb chat
```

The window is a small desktop app; download it from the [releases](https://github.com/state-alchemists/zrb-face/releases) or build it (see [The desktop app](#the-desktop-app)). Without it, `zrb-face` opens the page in your browser instead.

`zrb-face` opens the Windows build from WSL, the installed app on macOS, or Electron from `app/` in a source checkout (`ZRB_FACE_APP` overrides the program). It restarts the server each time, so the server always runs the current code with your current settings; open windows reload by themselves. The server keeps running across chats; its output goes to `zrb-face.log` in the temp folder, and `pkill -f "[z]rb_face/server.py"` stops it. The order doesn't matter: the window waits for the server, reloads itself when the server restarts, and zrb skips sending while either is missing. If a model can't be loaded, the window says so in red and tries again every few seconds.

## How it fits together

```
 zrb chat  ── POST /event ──▶  zrb-face server  ── page, models, animations, live events ──▶  window (face.html)
 + zrb_init.py                 (port 7777)                                                    in the app or a browser
```

| File | Role |
| --- | --- |
| `zrb_face/zrb_init.py` | The zrb side, loaded through `ZRB_INIT_SCRIPTS`: zrb's speech (moving the mouth, silent for "nothing to say"), the face hooks, her tools (`FaceActions`, `FaceAction`, `FaceLook`, `FaceModels`, `FaceModel`), and the optional triggers that let her speak first. |
| `zrb_face/client.py` | Sends events to the server: state and emotion from zrb's hooks, mouth openness from the loudness of the speech audio. Skips sending when the server is down. |
| `zrb_face/server.py` | The server (Python standard library only): serves the page, the models and the animations from their folders, and relays every event to the open windows over Server-Sent Events. |
| `zrb_face/face.html` | Everything visible, with three.js and three-vrm: loads the model, plays the animations, blinks, lip sync, eye contact, smiles, emotions, the camera. |
| `zrb_face/__main__.py` | The `zrb-face` command. |
| `zrb_face/anim/` | The animations that ship with the package. |
| `app/` | The desktop window: an Electron shell (`main.js`) that shows the page see-through and always on top. |
| `scripts/make_idle.py` | Generates the keyframed animations in `zrb_face/anim/`: eased poses, arms placed by inverse kinematics, standing feet planted. |
| `scripts/make_mocap.py` | Generates the motion-captured animations in `zrb_face/anim/` from the CMU motion capture database; also converts a BVH of your own. |
| `test_face.py` | The self-check. |

`scripts/` holds the tools that make the animations; they aren't part of the installed package (the wheel holds only `zrb_face/`), and the app never runs them.
| `zrb_init.py`, `.github/workflows/` | CI: the tests and lint on every push and pull request, and an LLM code review (the repository's own `review` and `submit-comment` zrb tasks, in the root `zrb_init.py`) posted on pull requests. |

zrb runs in WSL on Windows, but a window that stays on top has to be a Windows program. Windows can reach a server in WSL (not the other way around in WSL's default NAT mode), so the server runs next to zrb and the window connects to it.

## Running by hand

What `zrb-face` does, per platform:

```sh
zrb-face serve &                                                                          # the server
/mnt/c/Users/$USER/AppData/Local/zrb-face/zrb-face.exe rem </dev/null &>/dev/null &!      # Windows, from WSL
open -n -a zrb-face --args rem                                                            # macOS, installed app
~/zrb-face/app/node_modules/.bin/electron ~/zrb-face/app rem                              # Linux, or any OS from source
```

The window takes a model name (`rem` or `rem.vrm`; without one, the first model alphabetically) and optionally another server URL (`http://localhost:7779`). Start the Windows build detached as above, or zsh stops it ("suspended (tty input)"). Instead of the app you can open `http://localhost:7777/?model=rem` in a browser: the same page, but not see-through or always on top.

## Using the window

- **Move**: drag it. **Close**: Alt+F4 on Windows, Cmd+Q on macOS.
- **Camera angle**: click the window, then use the keys in [Keyboard](#keyboard-click-the-window-first). The angle is remembered.

## What she shows

| Glow | State | Sent by |
| --- | --- | --- |
| cyan | idle | `Stop`, `StopFailure`, `SessionStart` |
| violet (head tilts) | thinking | `UserPromptSubmit`, `PostToolUse`, `PostToolUseFailure` |
| green | running a tool | `PreToolUse` |
| red | needs attention | `Notification`, `PermissionRequest` |
| bright cyan | speaking | mouth events |

- **Voice**: the mouth follows the loudness of the speech, in 40 ms steps — one value per step of audio, windowed across the chunk boundaries of the stream, so it ends when the voice does — and she nods slightly while speaking. It needs a voice zrb plays itself (piper, openai, gemini); `say` and `termux` play the audio themselves, so zrb never sees it. The mouth still starts slightly ahead of the voice, because zrb reads audio ahead of playing it; fixing that exactly needs a frame observer in zrb's `PcmUtterance._fill`.
- **Face**: she blinks every few seconds, and her eyes are on you (the camera) the way real eyes are: small jumps around your face, a glance away now and then (often with a blink), more of them while thinking, fewer while speaking; she smiles softly every 25–50 seconds and after finishing a task.
- **Body**: each state loops its own animations, falling back to the idles; see [Animations](#animations). On `SessionStart` she plays the `greeting` action if there is one.
- **Emotion**: with zrb's emotion on (`ZRB_LLM_EMOTION_ENABLED=true`), zrb opens each reply with an emoji, and she shows it for 6 s with the matching VRM expression (`happy`, `sad`, `angry`, `surprised`, `relaxed`), and plays the action of the same name if there is one (`actions/happy.vrma`). The user's own emotion isn't shown: her reply answers it.
- **Gestures**: zrb gets her tools, all auto-approved: `FaceActions` lists her animations by pool with what each looks like and how long it takes, `FaceAction` plays one once alongside its words (an action by name: wave, nod, bow, …; any other clip by `pool/name`: `thinking/idea`), and `FaceLook` shows zrb her window as you see it (her pose and expression from your side, plus what she is doing), to check how a gesture comes across or when you ask how she looks. `FaceModels` lists the avatars (the `.vrm` models) and which one is showing, and `FaceModel` switches every open window to another one, when you ask her to look different. The actions and models are read on every call, so new ones work without restarting zrb. `FaceLook` sees only her own window, never your screen, and needs a model that reads images (or zrb's `ZRB_LLM_MULTIMODAL_MODEL`).
- **Music**: when something plays music (a browser, Spotify, …) she dances with the `dance/` animations while idle; anything zrb does takes over, and she goes back to dancing after. The server watches the system's media status: Windows' media controls (also from WSL), `playerctl` on Linux, or on macOS `media-control` (`brew install media-control`; it also sees a browser tab playing) and else Spotify and Music by AppleScript (which asks permission to control them the first time). Any playing media counts, a talk video too. Use `zrb-face music on` / `off` to override it; `ZRB_FACE_MUSIC=off` stops the watching.
- **Talking first** (off by default): with `ZRB_FACE_TALK_FIRST=idle` she speaks up after you've been quiet for a while during waking hours (8:00–22:00); with `camera`, when you sit back down after being away. She never interrupts a running turn or her own speech, waits a cooldown between such turns, and replies `…` (shown, not spoken) when there's nothing worth saying.

The mouth, the blink and the smile need the model's `aa`, `blink` and `relaxed` expressions. A model without them still nods while speaking.

## Wiring it into zrb

`zrb_face/zrb_init.py` is all zrb needs; load it through `ZRB_INIT_SCRIPTS` as in [Quick start](#quick-start) (`zrb-face init-path` prints where it is). It owns zrb's speech (`enable_speech`), so other init scripts should only choose the voice (`CFG.DEFAULT_LLM_SPEECH_BACKEND`, `CFG.DEFAULT_LLM_SPEECH_VOICE`) and not call `enable_speech` themselves; whichever script calls it last would win.

## Configuration

### Environment variables

The zrb-side names follow zrb's `CFG.ENV_PREFIX` (`ZRB` by default). The server and the `zrb-face` command run without zrb, so their names are fixed.

| Variable | Default | Read by | What it does |
| --- | --- | --- | --- |
| `ZRB_FACE_TALK_FIRST` | empty (never) | `zrb_init.py` | What may start a conversation: `idle`, `camera`, or `idle,camera`. |
| `ZRB_FACE_IDLE_MINUTES` | `45` | `zrb_init.py` | Quiet minutes before `idle` speaks. |
| `ZRB_FACE_COOLDOWN` | `1200` | `zrb_init.py` | Seconds between conversations she starts herself. |
| `ZRB_FACE_CAMERA_INTERVAL` | `20` | `zrb_init.py` | Seconds between camera looks; each one lights the camera. |
| `ZRB_FACE_CAMERA_PRESENT_BYTES` | `25000` | `zrb_init.py` | Photo size above which someone counts as present; tune it for your camera and room. |
| `ZRB_FACE_MOUTH_GAIN` | `4` | `client.py` | How wide the mouth opens for a given loudness: raise it if the mouth barely opens, lower it if it stays wide open. |
| `ZRB_FACE_URL` | `http://127.0.0.1:7777/event` | `client.py` | Where zrb sends events. |
| `ZRB_FACE_HOST` | `127.0.0.1` | server | The address the server listens on. |
| `ZRB_FACE_PORT` | `7777` | server, `zrb-face` | The server's port (change `ZRB_FACE_URL` to match). |
| `ZRB_FACE_HOME` | `~/.config/zrb-face` | server, `zrb_init.py` | Your folder: models in `models/`, animations in `anim/`. |
| `ZRB_FACE_MODELS` | empty | server | More model folders, searched before `<home>/models`; separated by `:` (`;` on Windows). `~` is expanded. |
| `ZRB_FACE_ANIMS` | empty | server, `zrb_init.py` | More animation folders, searched before `<home>/anim` and the shipped ones. |
| `ZRB_FACE_MUSIC` | `auto` | server | `auto` dances while media plays — Windows' media controls (also from WSL), `playerctl` on Linux, or `media-control` (else Spotify/Music by AppleScript, which asks permission the first time) on macOS; `off` only dances on `zrb-face music on`. |
| `ZRB_FACE_APP` | per platform | `zrb-face` | The program to open as the window, instead of the Windows build, the macOS app or Electron from source. |

### zrb settings it depends on

| Setting | Why it matters |
| --- | --- |
| `ZRB_INIT_SCRIPTS` | Must include `$(zrb-face init-path)`; that's what turns the face on. |
| `ZRB_LLM_EMOTION_ENABLED` | `true` makes zrb tag replies with an emotion, which she shows. Off by default. Needs a zrb with the `Emotion` hook. |
| `ZRB_LLM_SPEECH_BACKEND` / `ZRB_LLM_SPEECH_VOICE` (or `CFG.DEFAULT_LLM_SPEECH_BACKEND` / `_VOICE` in an init script) | The voice. For the mouth to move it must be one zrb plays itself (piper, openai, gemini), not `say` or `termux`. |
| `ZRB_LLM_SPEECH_PLAYER` | Leave it at `auto` (the default), with zrb's voice extra installed (`zrb[voice]`, for sounddevice): then zrb plays the audio itself and the mouth can follow it. `command` hands the audio to a player program, hidden from zrb. |
| `ZRB_HOOKS_ENABLED` | Must stay on (the default): the states and speech ride on zrb's hooks. |
| zrb's camera settings (`ZRB_LLM_CAMERA_*`) | Used by `ZRB_FACE_TALK_FIRST=camera` to take its photos. |

### Command line and URL

| Where | Option | Example | What it does |
| --- | --- | --- | --- |
| `zrb-face` | model name | `zrb-face rem` | (Re)starts the server and opens a window with that model; without a name, the first model alphabetically. |
| `zrb-face serve` | | | Runs the server in the foreground. |
| `zrb-face action <name>` | | `zrb-face action wave`, `zrb-face action thinking/idea` | Plays an action once, or any clip by `<pool>/<name>`; handy for trying a new clip. |
| `zrb-face actions [pool]` | | `zrb-face actions idle` | Lists the animations by pool (or one pool, nested ones included), with how long each takes and what it looks like. |
| `zrb-face snapshot [file]` | | `zrb-face snapshot me.png` | Saves the window as you see it (default `zrb-face-snapshot.png`), and prints what she is doing. |
| `zrb-face music on\|off` | | `zrb-face music on` | Dance while idle, or stop. |
| `zrb-face models` | | | Lists the models, marking the one showing with `*`. |
| `zrb-face model <name>` | | `zrb-face model rem` | Switches every open window to that model (`{"type": "model", "value": "rem"}`), and windows opened later too, until the server restarts. |
| `zrb-face dirs` | | | Lists the model and animation folders in search order, and what they hold. |
| `zrb-face init-path` | | | Prints the path of `zrb_init.py`, for `ZRB_INIT_SCRIPTS`. |
| `zrb-face.exe`, the macOS app, `electron app` | model name | `zrb-face.exe rem` or `rem.vrm` | The model to show (letters, digits, `-`, `_`; `.vrm` optional). |
| same | server URL | `zrb-face.exe http://localhost:7779` | Another server than `http://localhost:7777`. |
| the page | `?model=<name>` | `http://localhost:7777/?model=rem` | The model, in a browser; one chosen by `zrb-face model` or `FaceModel` wins while the server runs. |
| the page | `?app` | (set by the app) | See-through background, dragging moves the window, no status text. |

### Keyboard (click the window first)

| Key | What it does |
| --- | --- |
| ← → | Orbit around her. |
| ↑ ↓ | Look from higher or lower. |
| `+` / `-` | Zoom in or out (0.5× to 4×). |
| `0` | Reset the camera. |

The camera angle is saved in the window's storage and comes back next time.

### Files and folders

Models and animations are looked up in folders, in this order; the first file with a given name wins, so your own files override the shipped ones. `zrb-face dirs` shows the list.

| Kind | Folders |
| --- | --- |
| models | `$ZRB_FACE_MODELS`, `~/.config/zrb-face/models` |
| animations | `$ZRB_FACE_ANIMS`, `~/.config/zrb-face/anim`, the package's `anim/` |

| Path | Convention |
| --- | --- |
| `<models folder>/<name>.vrm` | A model, opened by `<name>`. Only files directly in the folder are served. |
| `<anim folder>/<pool>/<name>.vrma` | An animation; the pool (a subfolder, which may nest: `actions/greetings/hi.vrma`) says when it plays. See [Animations](#animations). |

### In the code

Constants with no setting of their own; edit the file to change them.

| File | Constant | Value | What it does |
| --- | --- | --- | --- |
| `face.html` | `FRAMING` | `'body'` | `'body'` frames her head to toe (with room for every clip's reach, actions and swinging hair included, from any camera angle), `'face'` head and shoulders. |
| `face.html` | `LOOK` | cyan, violet, green, red, bright cyan | The glow colour per state: idle, thinking, tool, attention, speaking. |
| `face.html` | blinking | every 2–6 s, 0.12 s long | How often and how long she blinks. |
| `face.html` | smile | every 25–50 s for 3 s at 0.5; 2 s after a task | The soft `relaxed` smile. |
| `face.html` | thinking tilt / speaking nod | 0.12 / 0.04 rad | How far the head tilts while thinking and nods with the voice. |
| `face.html` | crossfade | 0.6 s (actions 0.3 s) | Blend time between animations. |
| `client.py` | `MOUTH_STEP` | 0.04 s | Audio per mouth value; the values are windowed across chunk boundaries, so one value is one step whatever the chunking. |
| `zrb_init.py` | `AWAKE_HOURS` | 8:00–21:59 | When she may speak first. |
| `zrb_init.py` | idle check / away time | every 60 s / 5 min | How often `idle` checks; how long you must be away before `camera` greets you back. |
| `zrb_init.py` | `NOTHING_TO_SAY` | `…` | The reply she gives, unspoken, when nothing is worth saying. |
| `app/main.js` | window | 440×640, bottom right | Size and place of the window (320×420 suits `'face'` framing). |
| `app/main.js` | on top | `screen-saver` level | Stays above full-screen windows too. |
| `face.html` | emotion | 0.6 for 6 s | How strong and how long she shows an emotion. |
| `scripts/make_idle.py` | `POOLS` | 7 idles, 5 dances, 3 thinking, 2 tool, 2 attention, 11 actions | The keyframed animations, as eased poses; run the script to regenerate them. |
| `scripts/make_idle.py` | `FPS` | 15 | Frames per second in the keyframed clips. |
| `scripts/make_idle.py` | `LAG` | 0.02–0.13 s | How far each bone trails the keyframes (follow-through: the hand after the forearm after the upper arm). |
| `scripts/make_mocap.py` | `CLIPS` | 5 idles, 1 dance, 2 speaking, 10 actions | The motion-captured animations: which CMU take, which seconds of it, which pool. |
| `scripts/make_mocap.py` | `FPS` | 30 | Frames per second in the motion-captured clips. |
| `face.html` | gaze | jumps every 0.6–3 s; glances away 18% (thinking 55%, speaking 8%) | How her eyes move around you. |
| `face.html` | loop speed | 0.9–1.1× (dances 1×) | Each loop plays a little faster or slower, so repeats never match exactly. |

## Changing the model

Put any VRM file (VRM 0.x or 1.0) in `~/.config/zrb-face/models/` (or a folder in `ZRB_FACE_MODELS`) as `<name>.vrm` and open it by name: `zrb-face <name>`. Without a name, the window shows the first model alphabetically. To switch while she runs, `zrb-face model <name>` (or ask zrb: its `FaceModel` tool) reloads every open window with the new model; the server remembers it for windows opened later, until it restarts. `GET /model` says which one is chosen and which ones the open windows show. No model ships with the package. Model files are kept out of git (`*.vrm` is ignored). Nothing in the code depends on the model: the camera fits it, and the pose and idle come from the animations, which three-vrm retargets onto any VRM, as VRoid Hub's playground does.

Where to get models:

- **[VRoid Studio](https://vroid.com/en/studio)**: free, for Windows and macOS. Make your own and export it as VRM. You own it, and it has every expression.
- **[VRoid Hub](https://hub.vroid.com/en)**: thousands of models. Only the ones marked downloadable can be downloaded, and each has its own conditions (avatar use, commercial use, redistribution, changes). Read them before you change or share the model.
- **[BOOTH](https://booth.pm/en/search/VRM)**: search for `VRM` and filter for free items. The terms are on each item's page, often only in Japanese.
- **[Open Source Avatars](https://github.com/toxsam/open-source-avatars)**: free VRM avatars with direct download links. The license is set per collection (CC0, CC-BY, …), so check it.
- **[OpenGameArt: VRoid Studio CC0 models](https://opengameart.org/content/vroid-studio-cc0-models)**: public domain sample models from early VRoid Studio.

Fan models of existing characters (Cortana, Kusanagi, and others) are fine on your own desktop, but don't commit or share them.

## Animations

Every `<pool>/<name>.vrma` (VRM Animation) in the animation folders loads when the page starts. The pool, the subfolder, says when it plays:

| Pool | Plays |
| --- | --- |
| `idle/` | Loops while idle, and for every state with no pool of its own. |
| `thinking/`, `tool/`, `attention/`, `speaking/` | Loops in that state. |
| `dance/` | Loops instead of `idle/` while music plays. |
| `actions/` | Once, on `{"type": "action", "value": "<name>"}`, then back to looping. zrb plays them through its `FaceAction` tool; `greeting` plays when a chat starts, and `happy`, `sad`, … with that emotion. |

Any other pool loads too, and plays once something asks for it (`zrb-face action <pool>/<name>`, or zrb's `FaceAction`).

Pools nest: a subfolder's clips belong to the pool above it, as one group in the random pick. `idle/calm/*.vrma` loops with the idles, as often as any one other idle; `actions/greetings/hi.vrma` is the action `greetings/hi`. Overrides go by the whole path: your `actions/greetings/hi.vrma` replaces only that one. A clip is always played by its full path (`thinking/idea`, `actions/greetings/hi`); an action's short name (`greetings/hi`) works too, unless it is some other clip's full path (an `actions/thinking/idea.vrma` is `actions/thinking/idea`, as `thinking/idea` is the loop).

To add your own, drop the file in, e.g. `~/.config/zrb-face/anim/actions/wave.vrma`, and restart with `zrb-face` so the window loads it; zrb's tools see it at once. The folders cascade per file: yours replaces a shipped clip of the same `<pool>/<name>`, adds any new one, and every shipped clip you don't have still plays. A description, which `FaceActions` shows the agent, is read from the file's animation `extras.description` (the generators write one; see `NOTES` in the scripts). A file named like a shipped one replaces it: `~/.config/zrb-face/anim/idle/rest.vrma`. Try an action with `zrb-face action wave`.

Within a pool, clips play in random order with crossfades, picked by group: the first part of the name (`stretch-side` belongs to `stretch`). A random group, then a random clip in it, so the stretches together come up as often as any single other idle. On top of whatever plays: the head tilts while thinking, nods while speaking, the eyes follow you, and the face shows zrb's emotion.

The shipped clips come from two generators, and every one passes the same check before it is written (see below).

- **Motion capture** (`python3 scripts/make_mocap.py`): real people, from the [CMU Graphics Lab Motion Capture Database](http://mocap.cs.cmu.edu/), retargeted onto the VRM skeleton. That's where the natural weight, timing and follow-through come from: waiting, stretching, talking with the hands, waving, laughing, crying, dancing. The script downloads the takes it needs (as BVH, from [una-dinosauria/cmu-mocap](https://github.com/una-dinosauria/cmu-mocap)) into `~/.cache/zrb-face/cmu`, maps each joint's world rotation onto the matching VRM bone (correcting for how differently the two skeletons stand at rest), calibrates the torso and legs on the actor's own neutral stance (the skeleton's zero pose isn't how a person stands, and mapped across as-is it leaves her leaning and her feet trailing), keeps the hips' sway but not where the actor walked to, stands her on the floor, faces her forward and turns her head to you on average (a conversation take's actor looks at the other person), keeps only part of the actors' side-to-side sway and head tilt (`CALM`: half in the loops, a third while speaking; acted big, it reads as tipsy), smooths the noisy hands, and picks each loop's start and end where the pose and the hips match best, blending the seam. CMU doesn't capture fingers, so the hands keep a relaxed curl.
- **Keyframes** (`python3 scripts/make_idle.py`): eased poses, for what no take has (a V sign, a thumbs-up, fidgeting, the beat-exact dances). The motion passes through the poses on a smooth curve rather than stopping at each, and each bone trails a little behind the one before it (the hand after the forearm after the upper arm), so a gesture travels out along the arm. Arms that must reach somewhere (the chin, a V sign by the face, hands clapping) are placed by where the wrist goes, not by angles: `reach()` solves the arm (two-bone IK), with the elbow and the palm pointing where asked.

Both plant the feet: wherever a foot stands (near the floor and not swinging, with some hysteresis so a hovering foot doesn't flicker), it is pinned where it stands and the legs are solved (IK) to keep it there, so when the hips shift, sway or dip, the knees bend instead of the feet sliding; a foot lets go only when it lifts, and no foot goes through the floor. The knee is a hinge that faces where the thigh's front does, so it never turns sideways or backward, and it eases into straight instead of snapping. That's also what gives the model's spring bones (hair, coat, chest) real motion to swing with: VRM physics runs on the model's own spring bones, every frame.

The idles:

| Clip | What she does |
| --- | --- |
| `still` | Mocap: standing, small shifts and glances. |
| `wait` | Mocap: hands on her hips, looking around, waiting. |
| `poise` | Mocap: arms folded, a gesture, the weight going from foot to foot. |
| `rest` | Relaxed, a slow weight shift. |
| `hands` | Hands clasped in front, swaying; the free foot twists on its toes. |
| `weight` | Shifts from hip to hip, the free knee bending, the feet staying put. |
| `look` | Glances left and right. |
| `stretch` | Mocap: the yawn-stretch, arms out wide. |
| `stretch-up` | Mocap: reaching up into a Y with a yawn, then out to the sides. |
| `stretch-side` | One arm arcs over her head, the other hand slides down the thigh, the hips shift the other way; each side. |
| `stretch-neck` | Ear to each shoulder, then chin down; shoulders dropped, eyes closed. |
| `stretch-twist` | Turns from the hips up, the head leading, the arms swinging loosely, the back heel lifting. |

The dances: `macarena` (mocap) and, at 120 BPM in 8 beats, `bounce` (down on every beat, arms pumping), `sway` (side to side every two beats), `step-touch` (stepping side to side, hands up on the touch), `hands-up` (arms high, waving) and `twist` (the hips and shoulders swinging one way and back). They don't follow the song's tempo yet.

The state loops, so something is already happening before you drop your own clip in:

| Clip | Plays | What she does |
| --- | --- | --- |
| `thinking/ponder` | `thinking` | Chin in hand, the other arm folded under the elbow, looking up and away. |
| `thinking/fidget` | `thinking` | Restless: weight from hip to hip, fingers fidgeting in front of her. |
| `thinking/idea` | `thinking` | The lightbulb: a finger up by her face, leaning back, eyes up. |
| `tool/busy` | `tool` | Hands up at her chest, working; one hand reaches in and out. |
| `tool/hum` | `tool` | A small, contented shuffle while a tool runs. |
| `attention/alert` | `attention` | Leans toward you, hands clasped, waiting for an answer. |
| `attention/tap` | `attention` | The free foot taps on its toes, impatiently. |
| `speaking/explain` | `speaking` | Mocap: talking with her hands, walking you through it; the page adds the mouth and small nods. |
| `speaking/teach` | `speaking` | Mocap: teaching, pointing things out. |

The actions, each of which starts and ends standing so it can be cut into any loop. `happy`, `sad`, `angry`, `surprised` and `relaxed` are named after the VRM emotion presets, so zrb's own emotions drive the body as well as the face (the page looks the clip up by the emotion's name):

| Clip | What she does |
| --- | --- |
| `greeting` | Mocap: hello, a wave of the right hand. |
| `wave` | Mocap: a big wave, then both arms. |
| `curtsey` | Mocap: a curtsey. |
| `shrug` | Mocap: shoulders up, hands out. |
| `happy` | Mocap: arms thrown up, delighted. |
| `laugh` | Mocap: laughing. |
| `sad` | Mocap: head down, crying into her hands. |
| `surprised` | Mocap: startled, drawing back. |
| `angry` | Mocap: upset, hands to her head. |
| `flex` | Mocap: showing off her muscles. |
| `relaxed` | A slow sigh, the shoulders dropping. |
| `nod` | Yes, twice. |
| `shake` | No, twice. |
| `peace` | A V sign by her face, palm out, head tilted toward it. |
| `point` | The arm straight out toward you, index leading. |
| `thumb` | A fist up in front of her chest, thumb up: nice! |
| `clap` | Three claps in front of her chest. |
| `pose` | A model pose: hand on her hip, elbow out, weight on one leg, chin up. |
| `squat` | A squat down and back up, arms forward for balance. |
| `bow` | A bow from the waist, arms at her sides, eyes down. |
| `show` | Ta-da: the arms sweep open and up, chest proud, chin up. |

Every keyframed clip also has breathing (the chest and shoulders rising), uneven head drift, knees that flex a little, and fingers that relax and tighten.

To change or add keyframed clips, edit the pools in `scripts/make_idle.py` and run it again. Its docstring lists which way each joint turns; `reach()` places an arm by its wrist, and `arm()` builds one from angles. To add a motion-captured one, add a line to `CLIPS` in `scripts/make_mocap.py` (the CMU take, the seconds to use, the pool; [the index](https://github.com/una-dinosauria/cmu-mocap/blob/master/cmu-mocap-index-text.txt) lists every take) and run `python3 scripts/make_mocap.py <name>`. Your own BVH with CMU joint names works too: `python3 scripts/make_mocap.py --bvh take.bvh --start 2 --end 9 --pool idle --name mine --out ~/.config/zrb-face/anim`.

Before either writes a file, `audit()`/`check()` checks every frame: nothing sinks through the floor or through her body, no hand reaches further than an arm, no elbow folds past its stop (about 155°), no hand goes inside her head (touching her face is fine), her head never drops below her hips, and every clip ends where it starts. Poses can also move the hips (`hipsMove`) and set expressions (`blink`, `aa`, `happy`, `angry`, `sad`, `relaxed`, `surprised`). The actions wear a face (`FACES` in `scripts/make_idle.py`): a laugh closes her eyes in a smile and opens her mouth in bursts, a sad clip frowns, the friendly gestures smile. The page takes the larger of the clip's expression and its own blinking, lip sync, smile and emotion. A model without one of the expressions (many have no `surprised`) just skips it.

More free VRMA:

- **VRoid's official 7-motion pack** (show full body, greeting, peace sign, shoot, spin, model pose, squat), free on [BOOTH](https://vroid.com/en/news/6HozzBIV0KkcKf9dc1fZGW). Free to use, commercial use with the credit "Animation credits to pixiv Inc.'s VRoid Project", but its terms forbid redistributing the files, so keep them out of git. E.g. `cp VRMA_02.vrma ~/.config/zrb-face/anim/actions/greeting.vrma`.
- **Dances**: motion capture through bvh2vrma, e.g. the [AIST++](https://google.github.io/aistplusplus_dataset/) dance dataset (CC BY 4.0, credit needed) or the [CMU](http://mocap.cs.cmu.edu/) library (free). Mixamo's dances can be converted too, but its terms forbid sharing the files, so keep them in your own folder.
- **Your own**: [bvh2vrma](https://vrm-c.github.io/bvh2vrma/) (VRM Consortium, MIT) converts BVH motion capture to VRMA, and VRoid Studio exports poses and animations as VRMA.

## The desktop app

Each [release](https://github.com/state-alchemists/zrb-face/releases) has a zip per platform:

- **Windows** (`zrb-face-win32-x64.zip`): unzip into `%LOCALAPPDATA%\zrb-face\`, so `zrb-face` finds `zrb-face.exe` there from WSL.
- **macOS** (`zrb-face-darwin-arm64.zip` for Apple silicon, `-x64` for Intel): move `zrb-face.app` to `/Applications`. The app isn't signed, so clear macOS's block once with `xattr -dr com.apple.quarantine /Applications/zrb-face.app`.
- **Linux** (`zrb-face-linux-x64.zip`): unzip anywhere and set `ZRB_FACE_APP` to its `zrb-face` program. On Wayland (GNOME), a window can't keep itself on top, so pin it by hand.

To build it yourself:

```sh
cd app
npm install
npm start                                   # run from source
npm run pack:win                            # or pack:mac / pack:linux -> dist/
cp -r dist/zrb-face-win32-x64 /mnt/c/Users/$USER/AppData/Local/zrb-face
```

If GitHub downloads are slow, prefix the build with `ELECTRON_MIRROR=https://npmmirror.com/mirrors/electron/`, then check the zip in `~/.cache/electron` against GitHub's `SHASUMS256.txt`.

## Test

```sh
python3 test_face.py                        # with a Python that has zrb installed
```

It checks the search paths, the event relay, the remembered model choice, the music state and watcher (`media-control` and AppleScript), and every shipped `.vrma`: real glTF, the humanoid bones and channel references, and ends that meet, so the loops close. It also runs the same pose `audit()` the generator does before writing a file, regenerates every keyframed clip and fails if a shipped one is stale or a shipped file comes from neither generator, checks that standing feet don't slide, that `reach()` lands the wrist, and that a BVH take converts into a clip that loops, and fails if an emotion zrb can send has no action to play.

CI (`.github/workflows/test.yml`) runs it with a pyflakes lint on every push to `main` and every pull request: Python 3.11–3.14 on Linux, and 3.13 on Windows and macOS. On pull requests from a branch of this repository (not a fork), `code-review.yml` also has zrb review the diff and post the report as a comment (`zrb submit-comment`, defined in the root `zrb_init.py`); it needs the `OPENAI_API_KEY` secret (and optionally `ZRB_LLM_MODEL`), and skips itself without it.

## Releasing

Bump the version in `pyproject.toml` and `app/package.json`, then push a tag: `git tag 0.1.0 && git push --tags`. `.github/workflows/release.yml` publishes the package to PyPI and attaches the app zips to the GitHub release. PyPI publishing is trusted publishing, so first add this repository, the workflow `release.yml` and the environment `pypi` as a trusted publisher of `zrb-face` on pypi.org.

## Credits

`scripts/make_idle.py`, `scripts/make_mocap.py` and the keyframed clips in `zrb_face/anim/`: ours, CC0.

The motion-captured clips in `zrb_face/anim/` (listed in `scripts/make_mocap.py`'s `CLIPS`): the data used in this project was obtained from [mocap.cs.cmu.edu](http://mocap.cs.cmu.edu/). The database was created with funding from NSF EIA-0196217. CMU allows any use, including in products, except reselling the data itself.

The models used during development, not included: *ram* and *rem made in mexico* (Re:Zero fan models) by pon3mx on [VRoid Hub](https://hub.vroid.com/en); search for them there. Ram: use, changes, commercial use and redistribution allowed, no credit needed. Rem: the same but no commercial use. Both have mouth, blink and emotion expressions. They are fan models of the Re:Zero characters, so keep them on your own machine.
