APEX-style Research

SPX Benchmarks

The Silent Productivity Index family. Measures whether silent speech interfaces can perform economically valuable tasks — from intent reconstruction to streaming stability — offline and in real time.

50 Tasks · 5 Domains · 11 Commands 4 Benchmarks · 1 Harness 74.0% @ 0.36 ms heuristic

View leaderboard → Code — benchmarks/

In the spirit of Mercor APEX — one harness, every bench imports subvocal. MIT, HF subvocal/spx-* coming soon.

Our benchmarks

Each benchmark tests a different dimension — intent, signal, streaming, hardware. All tasks are built from src/subvocal/shorthand/eval_set.py and real HAL drivers.

Read spec →

SPX-Intent · 50 tasks5 domains

Shorthand → Intent reconstruction

Tests whether articulatory shorthand + context can resolve noisy tokens (e.g., srch snt spch → SEARCH silent speech) across GOTO, CLICK/SCROLL/CLOSE, SEARCH, TYPE, System.

View Leaderboard·Paper·Data (HF)·Code

Leaderboard — SPX-Intent50 tasks

1Hybrid LLM (target) · simulated96.0%

2Heuristic — articulatory distance74.0%

3GOTO / CLICK / SCROLL · 100%100%

4SEARCH · 10 tasks30%

5TYPE · 10 tasks40%

0.36 ms avg heuristic latency13 failures · plural & vocab collisions

SPX-Signal · classifiers2 models

Raw sEMG → tokens

Profiles real-time inference: latency, footprint, FLOPs, and cross-subject generalization. Will extend to LOSO harness across 9 datasets (Ninapro DB2/DB3/DB5, CapgMyo DB-b, Myo, UCI, MCS, Hyser, FlexWear-HD).

View Leaderboard·Paper·Data

Leaderboard — SPX-Signallatency @ params

11D CNN · 102,053 params1.66 ms

2GRU · 105,669 params8.27 ms

3SpeechNet · ~15k · GAP9 63.9 µJ~0.9 ms*

4SVM baseline0.00 ms

* SpeechNet Tiny CNN depthwise-separable (Spacone SilentWear 2026). Next: cross-subject LOSO + few-shot n_shot adaptation.

SPX-Streaming · runtimebounded

Bounded streaming & stability

Measures pipeline health: pipeline.step(window_ms) ≤5s timeout, bounded OpsQueue(maxsize) + backpressure (is_full/qsize), thread-safe inject_token, JSONL 10 MB rotation.

View Leaderboard·Paper·Code

Leaderboard — SPX-Streamingcorrectness

1step() ≤5s → HardwareError (no hang)pass

2OpsQueue bounded · backpressurepass

3inject_token thread-safe (deque+lock)pass

4JSONL trace 10 MB rotation · opt-outpass

Session state machine STARTING→ACTIVE→DEGRADED→CLOSED (no rollback) + watchdog liveness + Prometheus low-cardinality.

SPX-Hardware · HAL4 drivers

Hardware abstraction

Hardware-agnostic HAL: SyntheticSignalGenerator, FileReplayDriver, OpenBCICytonDriver, DelsysTrignoDriver + BrainFlow BoardShim pure-Python fallback, bounded buffers.

View Leaderboard·Paper·BOM

Leaderboard — SPX-Hardwaredrivers

1Synthetic · 8ch @ 1000 Hzready

2FileReplay · CSV/HDF5ready

3OpenBCI Cyton · Delsys Trignovia HAL

4BoardShim · pure-Python fallbackready

Dataset loaders: Ninapro DB2/DB5, PutEMG, CSL-HDEMG (192ch), Gaddy (8ch facial), MetaEMG (16ch wrist + pose).

SPX — Full leaderboard

Generated from benchmarks/eval_runner.py · full report →

SPX-Intent — Shorthand to Intent (50 tasks) Heuristic (offline articulatory distance) · deterministic

Rank	Decoder	Accuracy	Latency avg
1	Hybrid LLM (target) simulated · contextual prompt + Claude/GPT/Gemini adapters	96.0% ±2.4% 48/50	0.68 ms
2	Heuristic (offline) asymmetric Levenshtein · Glottal/Labial/Alveolar/Velar/Rhotic	74.0% 37/50	0.36 ms
3	GOTO · CLICK · SCROLL · CLOSE · System 26 tasks · deterministic vocab	100% 26/26	0.36 ms
4	TYPE 10 tasks · contact-aware matching	40.0% 4/10	0.36 ms
5	SEARCH 10 tasks · free-text + calendar context	30.0% 3/10	0.36 ms

Category breakdown: GOTO 10 @100% · CLICK 9 @100% · SCROLL 4 @100% · CLOSE 2 @100% · SEARCH 10 @30% · TYPE 10 @40% · System 5 @100%

SPX-Signal — sEMG Classifier Inference FLOPs & energy estimated · real wall-clock

Model	Mean	Median	Params	Disk	FLOPs	Energy
1D CNN	1.66 ms	1.64 ms	102,053	409.7 KB	3,678,336	367.834 µJ
GRU	8.27 ms	8.25 ms	105,669	420.5 KB	30,190,912	3019.091 µJ
SVM Baseline	0.00 ms	0.00 ms	0	0.0 KB	0	0.000 µJ

Source: docs/content/benchmark-report.md · v2.0.1 hardened wall-clock. Next: EMGBench LOSO across 9 datasets + SpeechNet 15k GAP9 profile.

Subvocal-extended

OSS datasets + SPX tasks — like Mercor-extended (OSS + 50–99 Mercor tasks), but for sEMG.

Ninapro DB2 / DB5

Hand gesture + SPX

Public sparse sEMG hand gestures. Add SPX shorthand labels, evaluate cross-subject adaptor (112→768→3072) to frozen Llama.

Requires subvocal[hardware]

PutEMG · CSL-HDEMG

Dense HD + contractions

PutEMG sequential, CSL-HDEMG 192ch high-density. SAL/LBN 7-param spatial adaptation (<2 min calibration) benchmark.

192ch grid 8×16 · affine warp

Gaddy · MetaEMG

Silent speech + pose

Gaddy 8ch facial EMG + audio (Zenodo 4064409); Meta sEMG-RD 16ch wrist + 63-D pose. EMGBench LOSO + few-shot n_shot.

Drivers: GaddyDriver · MetaEMGDriver

Install: pip install "subvocal[hardware]" · pip install "subvocal[ml]" for MONA/LISA/SpeechNet/TinyMyo etc.

Failure profile — why hybrid?

13 heuristic failures highlight where LLM reasoning wins.

Pluralization

srch nrl ntwk

Expected SEARCH neural networks → got Neural network (singular bias — shorter Levenshtein distance).

Fix: LLM picks plural via language semantics.

Phonetic tie

srch fgt calb

Expected Figma calibration → got logout confirm (fgt→logout tie, hallucinated bigram).

Fix: LLM knows “Figma calibration” is a calendar event.

Vocab collision

srch snt spch

Expected silent speech → got send Speech (snt→send present in dictionary).

Fix: bigram cohesion “silent speech” vs “send speech”.

Read benchmark spec → Full report →

How to run

One harness, every bench imports subvocal. Heuristic runs offline; hybrid needs API keys.

Start building → GitHub

terminal

# heuristic — no keys, deterministic
PYTHONPATH=src python3 benchmarks/eval_runner.py

# live LLM — picks best provider with key present
export ANTHROPIC_API_KEY=... # or OPENAI_API_KEY / GEMINI_API_KEY
PYTHONPATH=src python3 benchmarks/eval_runner.py

# install extras
pip install "subvocal[all,dev]"  # then pytest / ruff / pyright

Off-the-shelf data

50 tasks MIT today (src/subvocal/shorthand/eval_set.py), 250+ next with sEMG waveform pairs. License-ready, expert-written and graded — like Mercor’s 50k+ tasks across 30+ domains, but for silent speech.

50+ tasks MIT250+ next + waveformsHF: subvocal/spx-*

Roadmap for adoption (not revenue)

1SPX-Intent v1.1 — 250 text, hidden 50, publish HF + leaderboard

2SPX-Signal — 100 sEMG traces, cross-subject LOSO

3SPX-Live — real-time streaming + hardware partner