Metadata-Version: 2.5
Name: arbi-serve
Version: 0.1.0
Summary: Standalone OpenAI-compatible inference server with native Turbo Attention KV codec backend.
Project-URL: Homepage, https://github.com/arbicity/arbi-serve
Project-URL: Repository, https://github.com/arbicity/arbi-serve
Project-URL: Issues, https://github.com/arbicity/arbi-serve/issues
Author-email: Dmitri Evseev <dmitri.evseev@arbi.city>
License: Apache-2.0
License-File: LICENSE
Keywords: inference,kv-cache,llm,quantization,tkv,turbo-attention
Classifier: Environment :: GPU :: NVIDIA CUDA
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: attrs>=22.2.0
Requires-Dist: httpx<0.29.0,>=0.23.1
Requires-Dist: python-dateutil<3,>=2.8.0
Provides-Extra: audio
Requires-Dist: onnxruntime; extra == 'audio'
Requires-Dist: s3tokenizer; extra == 'audio'
Requires-Dist: silero-vad>=5.1; extra == 'audio'
Requires-Dist: torchaudio; extra == 'audio'
Provides-Extra: dev
Requires-Dist: commitizen>=4.5.0; extra == 'dev'
Requires-Dist: cryptography>=41; extra == 'dev'
Requires-Dist: docker; extra == 'dev'
Requires-Dist: fastapi; extra == 'dev'
Requires-Dist: fla-core<0.6,>=0.5.2; extra == 'dev'
Requires-Dist: httpx; extra == 'dev'
Requires-Dist: huggingface-hub; extra == 'dev'
Requires-Dist: jinja2>=3.1; extra == 'dev'
Requires-Dist: loguru<0.8,>=0.7.2; extra == 'dev'
Requires-Dist: msgspec>=0.19; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: ninja; extra == 'dev'
Requires-Dist: numpy; extra == 'dev'
Requires-Dist: nvidia-cutlass-dsl[cu13]==4.6.2; (sys_platform == 'linux' and platform_machine == 'x86_64') and extra == 'dev'
Requires-Dist: nvidia-ml-py>=12.0; extra == 'dev'
Requires-Dist: onnxscript; extra == 'dev'
Requires-Dist: openai>=1.0; extra == 'dev'
Requires-Dist: openapi-python-client>=0.21.0; extra == 'dev'
Requires-Dist: opentelemetry-api>=1.36.0; extra == 'dev'
Requires-Dist: opentelemetry-exporter-prometheus>=0.58b0; extra == 'dev'
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.58b0; extra == 'dev'
Requires-Dist: opentelemetry-instrumentation-logging>=0.58b0; extra == 'dev'
Requires-Dist: opentelemetry-sdk>=1.36.0; extra == 'dev'
Requires-Dist: orjson>=3.9; extra == 'dev'
Requires-Dist: peft; extra == 'dev'
Requires-Dist: pillow>=10.0; extra == 'dev'
Requires-Dist: pre-commit; extra == 'dev'
Requires-Dist: prometheus-client; extra == 'dev'
Requires-Dist: psutil>=5.9; extra == 'dev'
Requires-Dist: pydantic>=2; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-asyncio; extra == 'dev'
Requires-Dist: pytest-httpx; extra == 'dev'
Requires-Dist: pytest-timeout; extra == 'dev'
Requires-Dist: pytest-xdist; extra == 'dev'
Requires-Dist: pyyaml; extra == 'dev'
Requires-Dist: pyzmq>=25; extra == 'dev'
Requires-Dist: quack-kernels>=0.6; (sys_platform == 'linux' and platform_machine == 'x86_64') and extra == 'dev'
Requires-Dist: ruff==0.15.21; extra == 'dev'
Requires-Dist: safetensors; extra == 'dev'
Requires-Dist: tokenizers; extra == 'dev'
Requires-Dist: torch<2.13,>=2.12; extra == 'dev'
Requires-Dist: torchvision>=0.21; extra == 'dev'
Requires-Dist: transformers<5.16,>=5.15; extra == 'dev'
Requires-Dist: turbo-attn>=0.55.0; extra == 'dev'
Requires-Dist: uvicorn[standard]; extra == 'dev'
Requires-Dist: websockets>=13; extra == 'dev'
Requires-Dist: xgrammar<0.3,>=0.2; extra == 'dev'
Provides-Extra: evals
Requires-Dist: lm-eval>=0.4.9; extra == 'evals'
Requires-Dist: pyyaml; extra == 'evals'
Provides-Extra: exl3
Provides-Extra: fa
Provides-Extra: otlp
Requires-Dist: grpcio>=1.78.1; extra == 'otlp'
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.36.0; extra == 'otlp'
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.36.0; extra == 'otlp'
Provides-Extra: server
Requires-Dist: cryptography>=41; extra == 'server'
Requires-Dist: fastapi; extra == 'server'
Requires-Dist: fla-core<0.6,>=0.5.2; extra == 'server'
Requires-Dist: huggingface-hub; extra == 'server'
Requires-Dist: jinja2>=3.1; extra == 'server'
Requires-Dist: loguru<0.8,>=0.7.2; extra == 'server'
Requires-Dist: msgspec>=0.19; extra == 'server'
Requires-Dist: numpy; extra == 'server'
Requires-Dist: nvidia-cutlass-dsl[cu13]==4.6.2; (sys_platform == 'linux' and platform_machine == 'x86_64') and extra == 'server'
Requires-Dist: nvidia-ml-py>=12.0; extra == 'server'
Requires-Dist: onnxscript; extra == 'server'
Requires-Dist: opentelemetry-api>=1.36.0; extra == 'server'
Requires-Dist: opentelemetry-exporter-prometheus>=0.58b0; extra == 'server'
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.58b0; extra == 'server'
Requires-Dist: opentelemetry-instrumentation-logging>=0.58b0; extra == 'server'
Requires-Dist: opentelemetry-sdk>=1.36.0; extra == 'server'
Requires-Dist: orjson>=3.9; extra == 'server'
Requires-Dist: pillow>=10.0; extra == 'server'
Requires-Dist: psutil>=5.9; extra == 'server'
Requires-Dist: pydantic>=2; extra == 'server'
Requires-Dist: pyyaml; extra == 'server'
Requires-Dist: pyzmq>=25; extra == 'server'
Requires-Dist: quack-kernels>=0.6; (sys_platform == 'linux' and platform_machine == 'x86_64') and extra == 'server'
Requires-Dist: safetensors; extra == 'server'
Requires-Dist: tokenizers; extra == 'server'
Requires-Dist: torch<2.13,>=2.12; extra == 'server'
Requires-Dist: torchvision>=0.21; extra == 'server'
Requires-Dist: turbo-attn>=0.55.0; extra == 'server'
Requires-Dist: uvicorn[standard]; extra == 'server'
Requires-Dist: xgrammar<0.3,>=0.2; extra == 'server'
Provides-Extra: test
Requires-Dist: commitizen>=4.5.0; extra == 'test'
Requires-Dist: cryptography>=41; extra == 'test'
Requires-Dist: docker; extra == 'test'
Requires-Dist: fastapi; extra == 'test'
Requires-Dist: fla-core<0.6,>=0.5.2; extra == 'test'
Requires-Dist: httpx; extra == 'test'
Requires-Dist: huggingface-hub; extra == 'test'
Requires-Dist: jinja2>=3.1; extra == 'test'
Requires-Dist: loguru<0.8,>=0.7.2; extra == 'test'
Requires-Dist: msgspec>=0.19; extra == 'test'
Requires-Dist: mypy; extra == 'test'
Requires-Dist: ninja; extra == 'test'
Requires-Dist: numpy; extra == 'test'
Requires-Dist: nvidia-cutlass-dsl[cu13]==4.6.2; (sys_platform == 'linux' and platform_machine == 'x86_64') and extra == 'test'
Requires-Dist: nvidia-ml-py>=12.0; extra == 'test'
Requires-Dist: onnxscript; extra == 'test'
Requires-Dist: openai>=1.0; extra == 'test'
Requires-Dist: openapi-python-client>=0.21.0; extra == 'test'
Requires-Dist: opentelemetry-api>=1.36.0; extra == 'test'
Requires-Dist: opentelemetry-exporter-prometheus>=0.58b0; extra == 'test'
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.58b0; extra == 'test'
Requires-Dist: opentelemetry-instrumentation-logging>=0.58b0; extra == 'test'
Requires-Dist: opentelemetry-sdk>=1.36.0; extra == 'test'
Requires-Dist: orjson>=3.9; extra == 'test'
Requires-Dist: peft; extra == 'test'
Requires-Dist: pillow>=10.0; extra == 'test'
Requires-Dist: pre-commit; extra == 'test'
Requires-Dist: prometheus-client; extra == 'test'
Requires-Dist: psutil>=5.9; extra == 'test'
Requires-Dist: pydantic>=2; extra == 'test'
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-asyncio; extra == 'test'
Requires-Dist: pytest-httpx; extra == 'test'
Requires-Dist: pytest-timeout; extra == 'test'
Requires-Dist: pytest-xdist; extra == 'test'
Requires-Dist: pyyaml; extra == 'test'
Requires-Dist: pyzmq>=25; extra == 'test'
Requires-Dist: quack-kernels>=0.6; (sys_platform == 'linux' and platform_machine == 'x86_64') and extra == 'test'
Requires-Dist: ruff==0.15.21; extra == 'test'
Requires-Dist: safetensors; extra == 'test'
Requires-Dist: tokenizers; extra == 'test'
Requires-Dist: torch<2.13,>=2.12; extra == 'test'
Requires-Dist: torchvision>=0.21; extra == 'test'
Requires-Dist: transformers<5.16,>=5.15; extra == 'test'
Requires-Dist: turbo-attn>=0.55.0; extra == 'test'
Requires-Dist: uvicorn[standard]; extra == 'test'
Requires-Dist: websockets>=13; extra == 'test'
Requires-Dist: xgrammar<0.3,>=0.2; extra == 'test'
Provides-Extra: ui
Requires-Dist: fastapi; extra == 'ui'
Requires-Dist: openai>=1.0; extra == 'ui'
Requires-Dist: opentelemetry-api>=1.36.0; extra == 'ui'
Requires-Dist: opentelemetry-sdk>=1.36.0; extra == 'ui'
Requires-Dist: prometheus-client; extra == 'ui'
Requires-Dist: pydantic>=2; extra == 'ui'
Requires-Dist: pyyaml; extra == 'ui'
Requires-Dist: uvicorn[standard]; extra == 'ui'
Requires-Dist: websockets>=13; extra == 'ui'
Description-Content-Type: text/markdown

# arbi-serve

[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.12%2B-blue.svg)](pyproject.toml)

## Why arbi-serve

**Long-context inference that's actually viable on consumer GPUs: full
model context, virtually lossless quality, concurrency built for real
traffic — not a demo.** Charts backing each of these land in
[Benchmark and value](#benchmark-and-value-receipts-pending) below as
they're published.

The wedge between datacenter inference (vLLM, SGLang) and edge tools
(Ollama): self-hosted LLM serving with datacenter-grade concurrency and
VRAM efficiency, one-command-easy to run, monitor, and configure. Built
for businesses that want real production characteristics on their own
GPUs — RTX included — without a datacenter budget, and without
depending on someone else's cloud to run their own models. Distributed
inference, in the sense that matters: the power to run this yourselves
stays with you, not a central provider.

It's a serving engine for a documented, explicit set of model
architectures ([Feature status](#feature-status) below), not a generic
fallback for every checkpoint on the Hub.

## Quickstart

Needs [uv](https://docs.astral.sh/uv/), one NVIDIA GPU, Python 3.12+:

```bash
uvx arbi-serve run Qwen/Qwen3.5-0.8B
```

Installs itself, boots, drops you into a chat prompt. Everything below is
detail you don't need yet.

## Details

`arbi-serve` is the **native serving engine for
[Turbo Attention](https://github.com/arbicity/turbo-attn) (`tkv`)** —
the 2–8-bit asymmetric KV-cache codec (WHT rotation + calibrated
per-channel scaling, dequant fused inline into both prefill and decode).
Calibration, per-layer bit allocation, hot-swap, profiling, and memory
accounting are first-class rather than bolted on. Where SOTA kernels fit
(`flash-attn`, CuTeDSL, FlashInfer, Triton, vendor kernels) the
engine adopts them; where the serving path needs tighter control — fp8
paged-KV attention on consumer RTX cards the standard kernels refuse to
run, hybrid GDN/Mamba/ShortConv state, full-duplex audio — it carries
its own.

- **Calibrated, near-lossless quantization.** The TKV codec (2–8 bit,
  WHT rotation + per-channel calibration bundles) is built on our own
  attention backend (`turbo-attn`) specifically to hold quality at a
  fraction of the KV-cache memory — not a generic INT8 cast.
- **VRAM efficiency and management.** A deterministic budget across every
  pool (weights, activations, graphs, KV, LoRA, recurrent state, codec
  scratch — [docs/memory-accounting.md](docs/memory-accounting.md)),
  per-component attribution across ~35 gauges held to a checkable
  invariant, and a boot that fails loud with a component-level reason
  instead of OOMing under load.
- **Concurrency and iteration, live.** Continuous batching and CUDA graph
  capture for throughput; swap the attention backend, reload the model,
  or reload calibration at runtime with zero restart; compare numerical
  paths head-to-head; switch between resident models by naming a
  different one in the standard API. Flag correctness is CI-governed.
- **Easy to monitor and configure.** Prometheus at `/metrics`, an admin
  console, and structured boot/flag logging — the compile/cache paths and
  memory budget stay inspectable instead of opaque.
- **Real-time audio.** Full-duplex speech-to-speech primitives include VAD,
  barge-in, turn-taking, and TTS output for the supported audio models.
- **Fail loud, never subtly wrong.** Unsupported model/backend/quant
  combinations raise with a precise boundary instead of falling through
  to a slow or silently incorrect path.

The trade is deliberate: this is an explicit model and hardware surface, not a
generic Transformers fallback or a promise of feature parity with other
servers. See the [feature analysis](docs/feature-analysis.md) for boundaries.

`arbi-serve` consumes `tkv` through its public API only:
`tkv.runtime.*`, `tkv.codec`, and `tkv.core`. It never imports
`tkv.integrations.*` or private/internal turbo-attn modules.

## Install

`uvx arbi-serve run <model>` (above) needs nothing installed ahead of time.
Into your own project or venv instead:

```bash
uv pip install arbi-serve                 # lightweight generated Python client
uv pip install "arbi-serve[server]"       # standard NVIDIA inference server
uv pip install "arbi-serve[ui]"           # remote admin console + client
uv pip install "arbi-serve[server,ui]"    # local server + admin console
uv pip install "arbi-serve[fa]"           # no-op extra (kept for resolve compat)
uv pip install "arbi-serve[exl3]"         # compatibility no-op; see EXL3 note below
uv pip install "arbi-serve[dev]"          # + tests + lint tooling
```

(plain `pip install` works identically if you don't have uv)

### One-command bare-metal run

`arbi-serve run <model>` — see Quickstart — installs `arbi-serve[server]`
with uv (falling back to `pip`) if it isn't there yet, boots, waits for
readiness, and drops into a chat REPL. Run it again with a different model
while the first is still up and it hot-swaps the new model into the *same*
resident server (`POST /v1/admin/model`, the same fast park/wake path used
for A/B iteration) instead of booting a second process — pass `--new` to
force a separate instance on another `--port`. Extra flags for a fresh
boot go after `--` (e.g. `arbi-serve run demo -- --tp-size 2`). Add
`--console` to also boot the admin console (installs `[ui,otlp]` too, and
leaves OTLP push at the engine's own default instead of forcing it off).

Companion commands:

```bash
arbi-serve pull Qwen/Qwen3.5-0.8B   # fetch weights without booting a server
arbi-serve list                    # locally cached models
arbi-serve ps                      # running instances + their active model
arbi-serve stop                    # stop a tracked instance
```

`run`/`pull`/`ps`/`stop` are Python-only orchestration around the existing
bare-metal entrypoint (`python -m arbi_serve`) and the generated admin
client — they never fall back to Docker Compose on their own. If the
native CUDA/kernel stack can't be made to work bare-metal, they say so and
stop; use the Docker Compose quickstart below instead.

`arbi-serve doctor` diagnoses a broken/incomplete install (works even
without torch/tkv — that's exactly when it's most useful) —
driver/CUDA/compute-capability, disk and cache writability, Hugging Face
reachability, and, once the server deps are present, turbo-attn/tkv
compatibility and environment sanity. `arbi-serve plan --model <path>`
estimates VRAM/context/concurrency for a config *before* booting, without
loading any weights — an estimate, not the persisted boot-confirmed
sizing cache described in [docs/memory-accounting.md](docs/memory-accounting.md).
`arbi-serve config explain <flag>` / `config flags` / `config resolve`
inspect the same `RuntimeFlags`/`ServerConfig` surface the admin console
does. `arbi-serve --version`, `arbi-serve --help` (lists every
subcommand, including `openapi`/`convert`/`calibrate`/`warmup`/`bench`),
and each subcommand's own `--help` are the full reference.

### Talking to a server directly

The base install (no CUDA/PyTorch needed) is also a useful command-line
client for any already-running endpoint — one started with `arbi-serve run`
above, the Docker Compose path below, or a remote deployment:

```bash
arbi-serve chat --url http://localhost:8000 --model Qwen3.5-0.8B \
  "What is 2 + 3? Answer with only the number."
```

The model can be omitted when the endpoint advertises exactly one model, and
the prompt can come from stdin. Use `--token` (or `ARBI_SERVE_TOKEN`) for an
authenticated endpoint, `--temperature` and `--max-tokens` for sampling
controls, and `--json` when a complete OpenAI-compatible response is needed:

```bash
printf '%s\n' 'Name three primary colors.' | arbi-serve chat --url http://localhost:8000
```

GDN and structured generation are part of the standard server runtime, not
model-specific extras. arbi-serve does not use Transformers for serving; the
official container omits it. For bare-metal installs, upstream `xgrammar`
currently declares Transformers transitively even though arbi-serve supplies
its own tokenizer metadata. Transformers is otherwise test/reference-only.

EXL3 serving is complete in the official server image. For a bare-metal
installation, install the pinned weights-only fork separately without its
unused model-stack dependencies (the published `[exl3]` extra remains an empty
compatibility marker because Python package metadata cannot express
`--no-deps`):

```bash
pip install --no-deps \
  "exllamav3 @ git+https://github.com/arbi-dev/exllamav3.git@cb76869b47eecb30a8ed788d9218c4bf2cb7f09e"
```

See [`pyproject.toml`](pyproject.toml) for the full extras list and version
floors. For source installs and a development setup, see
[`CONTRIBUTING.md`](CONTRIBUTING.md).

## Docker Compose quickstart: one NVIDIA GPU

The Quickstart above (`uvx arbi-serve run`) is the fastest path for one
model on one machine. This is the Docker Compose path instead — same GPU
requirement, more explicit control over backend/codec, and what's actually
running behind the scenes if you want to see it. It explicitly selects `auto`,
which uses bf16 paged KV through the `tkv-bypass` kernels and needs no
calibration bundle. The general Compose file defaults to `tkv`, so keep this
override in the command. `Qwen/Qwen3.5-0.8B` is the small GDN smoke model used
for quick bring-up and high-concurrency testing. It is not the
project's performance or quality reference: that is the Qwen3.8-27B GDN path
below. See the [supported model matrix](docs/supported_models.md) before
choosing another checkpoint.

Prerequisites: Docker Compose v2, the NVIDIA Container Toolkit, one CUDA GPU,
and enough VRAM for the selected model, context, and CUDA graphs.

```bash
git clone https://github.com/arbicity/arbi-serve
cd arbi-serve

ARBI_MODEL=Qwen/Qwen3.5-0.8B \
ARBI_KV_CACHE_DTYPE=auto \
ARBI_MAX_CONTEXT=2048 \
docker compose up -d --build
```

The first boot builds the image and downloads the hub checkpoint into the
persistent Compose cache. Follow detail with `docker compose logs -f arbi-serve`.
Wait for the serving gate, not just the web server binding:
`/health` is liveness and can return 200 while the model is building;
`/health/ready` is 503 until a request can be admitted. Probe both (as the
Compose healthcheck does):

```bash
until curl -fsS http://localhost:8000/health/ready >/dev/null; do sleep 2; done
curl -fsS http://localhost:8000/health
curl -fsS http://localhost:8000/health/ready
```

Discover the exact model id before making a request:

```bash
curl -fsS http://localhost:8000/v1/models
```

For a hub id, the default served id is the repository name (`Qwen3.5-0.8B` in
this example; an `@revision` suffix is retained). For a local checkpoint it is
the final path component. Send that served id, not necessarily the original
host path. The lightweight CLI uses the generated arbi client and
auto-discovers the model when exactly one is advertised:

```bash
arbi-serve chat --url http://localhost:8000 --model Qwen3.5-0.8B \
  --max-tokens 24 "What is the capital of France?"
```

Applications can use the official OpenAI Python SDK directly:

```bash
pip install openai
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
answer = client.chat.completions.create(
    model="Qwen3.5-0.8B",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=24,
)
print(answer.choices[0].message.content)
```

`ARBI_MODEL` accepts a Hugging Face id (`org/name` or `org/name@revision`) or a
container-side local path. Set `HF_TOKEN` for gated/private repos;
`ARBI_HF_OFFLINE=1` forbids hub traffic. Local checkpoints require
`ARBI_MODELS_ROOT=/host/path/to/models` and an `ARBI_MODEL` path under `/models`.

### Canonical reference: Qwen3.8-27B GDN

Qwen3.8-27B is the primary development, performance, and quality reference.
Two checked-in recipes serve it on one 24-GB card, both EXL3 weights with
calibrated TKV and explicit memory/capture budgets:
[`compose.serve.yaml`](compose.serve.yaml) with the checkpoint's own native MTP
head at depth 5, and [`compose.dflash.yaml`](compose.dflash.yaml) with a
separate DFlash2 draft checkpoint at depth 7 — the latter is the faster
single-card path. `compose.serve.yaml` also documents a separate TP=2 variant. The 0.8B quickstart above is for smoke and high-concurrency coverage,
not representative benchmark numbers.

For the single-card reference, place the matching EXL3 checkpoint and curated
calibration bundle under the mounted directories, then run (replace the bundle
filename with the one matched to that exact checkpoint):

```bash
ARBI_IMAGE=arbi-serve:latest \
ARBI_MODELS_ROOT=/path/to/models \
ARBI_MODEL=/models/Qwen3.8-27B-exl3-4.0bpw \
ARBI_CALIBRATIONS_ROOT=/path/to/calibrations \
ARBI_CALIBRATION_FILE=/cal/YOUR_MATCHING_QWEN38_27B_BUNDLE.json \
docker compose -f compose.serve.yaml up -d
```

`ARBI_MAX_BATCH` is left at the recipe's own default (8) on purpose: pinning it
to 1 gives a single-stream server whose KV pool, capture ladder and acceptance
figures are not the reference ones.

`ARBI_IMAGE` is required here rather than defaulted: compose does not fail on
an image it cannot find — with a `build:` stanza present it silently builds
one, which reads as a several-minute hang. Name the image you mean (the tag
the quickstart's `docker compose up --build` produced, or a published one), and
add `--build` when building is what you want. `compose.yaml`, the quickstart
above, is the one recipe that still builds by default.

Read the recipe's TP1/TP2 notes before changing batch, context, GPU-memory, or
MTP settings; those knobs jointly determine whether the reference fits and
whether a comparison remains apples-to-apples.

### Calibrated TKV notes

Use this arm when you have a calibration bundle matched to the checkpoint and
want the Turbo Attention KV codec. The general Compose file currently does the
following by design:

- `ARBI_KV_CACHE_DTYPE` defaults to `tkv` (the quickstart overrides it to
  `auto` for evaluation).
- `TKV_BITS` is a literal Compose value of `"4"`, meaning uniform K4V4. It is
  not an operator override; exporting `TKV_BITS` on the host does not change
  this file. Smart-mix/per-layer allocation uses a float such as `4.0` and a
  matching bundle, and requires an intentional recipe edit.
- `ARBI_CALIBRATION_FILE`, when set, is forwarded as `TKV_CALIBRATION_FILE`
  inside the container. It must be a container path (for example
  `/cal/qwen3.5-0.8b_k4v4.json`) under the read-only
  `ARBI_CALIBRATIONS_ROOT` mount. A matching bundle discoverable under `/cal`
  or the persistent calibration cache can also be selected by the runtime.

On a single 24-GB GPU, do not rely on in-process first-boot calibration: the
temporary calibration swap can need roughly 12 GiB in addition to loaded
weights and may OOM. Supply a pre-built matching bundle, or use the bf16
quickstart. An explicit TKV boot does not silently change to bf16; with no
bundle it stages calibration (and may fail from VRAM pressure), or refuses when
that path is not available.

Example that exercises TKV plumbing with the 0.8B smoke/high-concurrency model
(not the canonical performance reference):

```bash
ARBI_MODEL=Qwen/Qwen3.5-0.8B \
ARBI_KV_CACHE_DTYPE=tkv \
ARBI_CALIBRATIONS_ROOT="$PWD/calibrations" \
ARBI_CALIBRATION_FILE=/cal/qwen3.5-0.8b_k4v4.json \
docker compose up -d --build
```

### Admin console and API

The optional UI is a separate process and does not sit in the request path:

```bash
ARBI_MODEL=Qwen/Qwen3.5-0.8B ARBI_KV_CACHE_DTYPE=auto \
docker compose --profile ui up -d --build
```

Open <http://localhost:8899>; the API remains on `http://localhost:8000`.
OpenAI-compatible routes include `/v1/completions`, `/v1/chat/completions`
(including SSE), `/v1/models`, `/v1/embeddings`, `/v1/rerank`, `/v1/files`, and
`/v1/batches`. Metrics are available at `/metrics`.

### Observability

The optional local Prometheus/Grafana/Tempo/Loki stack is a separate Compose
project:

```bash
docker compose -f docker-compose.observability.yaml up -d
```

Open <http://localhost:3000> for the provisioned dashboards. The server also
serves Prometheus text at `/metrics`; disable OTLP export with
`ARBI_SERVE_ENABLE_OTEL=0` when the collector is not wanted.

### Bare metal

Install the server dependency set first:

```bash
pip install "arbi-serve[server]"
arbi-serve --model /path/to/checkpoint --kv-cache-dtype auto --max-context 2048
```

For calibrated TKV, use `--kv-cache-dtype tkv --calibration /path/to/bundle.json`.
`--max-context` states the deployment cap explicitly; choose it to fit the
model, backend, and GPU budget. The same `auto`/`tkv` semantics apply outside
Compose.

The console runs in its own container and dials the engine by address. The
recipes state two things for it and nothing else: `--engine`, the engine's
service name on the compose network, and `--admin-token-file`, a path to the
operator credential the engine mints on first boot and both containers read
off the shared cache volume. The credential is what lets the console reach
`/v1/admin/*` from another container — or another machine — without the
loopback fallback and without ever setting
`ARBI_ALLOW_UNAUTHENTICATED_ADMIN`. See [docs/admin-auth.md](docs/admin-auth.md).
Add further servers — including remote ones, each with its own credential —
from the console's server picker; they are kept server-side and persist across
restarts, and the browser only ever names one by id.

To run only the admin console from a lightweight machine:

```bash
pip install "arbi-serve[ui]"
python -m arbi_serve.admin_console --host 0.0.0.0
```

Open <http://localhost:8899> and add a server in the picker. Pass `--engine`
and `--admin-token-file` when it runs beside one; the console then refuses to
serve until it has verified that engine answers, that the port it bound is one
its container publishes, and that it is running the same `arbi_serve` source as
that engine — each of which used to fail silently.

## Benchmark and value (receipts pending)

The repository does not publish universal speed, context, or cost numbers here.
Record results on the target GPU, model, context, and workload before making a
deployment decision. These are intentionally explicit placeholders:

| Metric | Value | Conditions / receipt |
|---|---|---|
| Time to first token (TTFT) | **TBD** | GPU, model, prompt/output lengths, warm/cold state: **TBD** |
| Decode throughput (tokens/s) | **TBD** | concurrency and sampling settings: **TBD** |
| Steady-state VRAM | **TBD** | weights, KV mode/bits, context, batch: **TBD** |
| Maximum validated context | **TBD** | model + GPU + backend: **TBD** |
| Quality/parity result | **TBD** | dataset, seed, and reference path: **TBD** |

For measurement pitfalls and a reproducible-measurement checklist, see
[`docs/bench-measurement-footguns.md`](docs/bench-measurement-footguns.md).
Methodology and raw benchmark receipts for these public values: **TBD (to be
published; no stable link yet)**.

## Feature status

This is a forward-looking feature map, not a backwards-compatibility
matrix. The server intentionally owns the seams that matter for fast
self-hosted iteration: load, profile, capture, swap, schedule, offload,
and observe. If a doc disagrees with the implementation, the
implementation wins.

### Shipped

| Area | Status |
|---|---|
| OpenAI API | `/v1/completions`, `/v1/chat/completions`, `/v1/models`, `/v1/embeddings`, `/v1/rerank`, `/v1/files`, `/v1/batches`; streaming for generation. |
| Native turbo-attn | TKV is the default center of gravity: KV-cache quantization, calibration bundles, per-layer bit allocation, hot-swap, profiling, and compare tooling. |
| Other attention kernels | tkv-bypass (the Turbo prefill CuTeDSL kernel + the split-K decode kernel, the default bf16 path), FlashInfer prefill backend, and vendored kernels where the engine needs tighter control. |
| Scheduling | Continuous batching, chunked prefill, prefix-aware admission, interactive-vs-batch priority, batch prefill caps, MTP K-bucketing, and pool-aware admission. |
| Prefix cache | Radix prefix cache with per-request opt-out, tenant namespace isolation, admin flush, and recurrent savepoints for hybrid models. |
| Fast load / reload | Safetensors loading, flat-dump cache, model reload, backend reload, calibration reload, post-load compaction, and memory-attribution checks. |
| Backend hot-swap | Per-`StateKind` active backend map; admin hot-swap drains, swaps, and rebuilds per-rank state coherently. |
| Tensor parallel | TP=2 serving under `torchrun`, including rank-0 scheduling, worker-rank forward replay, TP collectives, hot-swap, CUDA graph capture, and bundled MTP. |
| Speculative decoding | Bundled MTP heads, block-diffusion draft models (DFlash2 — the drafter the reference single-card recipe serves, see [`compose.dflash.yaml`](compose.dflash.yaml)), mixed-K admission bucketing, greedy and stochastic rejection sampling, graph-safe seed buffers, captured drafter/verify paths, and external draft-model mode for single-GPU paged-KV models. |
| Capture / compile | CUDA graph decode, piecewise capture, MTP drafter/verify capture, sleep/resume graph survival, and optional `torch.compile` integration. |
| Hybrid architectures | GDN, Mamba/Mamba2, ShortConv, MLA, SWA/GQA dense paths; recurrent rollback/savepoint plumbing where needed. |
| Suspended-job offload | Batch-job KV snapshots to host RAM and disk, fault-in on resume, persisted disk index, scheduler integration, and hybrid recurrent-state opt-in. |
| Weight quantization | AWQ, EXL3, FP8, NVFP4 detection and load paths behind a backend-agnostic quant registry. |
| LoRA | Runtime LoRA load/list/unload, mixed-adapter batches, BGMV apply, capture-aware state. |
| Structured output | xgrammar-backed JSON schema, regex, grammar, and JSON-object constraints with batched bitmask application and in-memory/disk grammar cache. |
| Multimodal | OpenAI image input parsing, Qwen-VL preprocessing, M-RoPE positions, vision/text merge, and Gemma/Qwen vision seams. |
| Observability / DX | OTel, Prometheus metrics, NVML/process gauges, Grafana dashboards, request timelines, memory diagnostics, allocator-history dump endpoints, and backend comparison harnesses. |
| Admin operations | Critical-section drain/mutate/resume for backend swap, model reload, calibration reload, cache flush, dynamic config, sleep/release/resume, and flat-weight dump/resume. |

### Roadmap / partial

| Area | Current boundary |
|---|---|
| Arbitrary TP | TP=2 is shipped. Current roadmap priority is general TP across 2/4/8/16 and non-power-of-two layouts where the model geometry allows it. |
| LoRA at TP>1 | Refused at driver construction until per-row adapter state has a worker broadcast path. |
| Multi-group at TP>1 | Single-rank multi-group hot-swap exists; TP composition is not wired. |
| Expert parallel / MoE all-to-all | Dense and some MoE model code exists, but EP serving and missing all-to-all paths are not a production surface yet. |
| Draft models | Bundled MTP is the production multi-rank speculative path. On a single card the reference recipe serves a separate DFlash2 draft checkpoint ([`compose.dflash.yaml`](compose.dflash.yaml)); external draft models more broadly are an active expansion area. |
| Prefix-cache warm/cold tiers | Suspended batch-job KV offload is shipped. Reusable prefix-cache page migration across GPU/host/NVMe tiers is a separate follow-up. |
| Broader architecture matrix | Qwen3 and Qwen3.5/3.6/3.8 GDN, DeepSeek/Kimi MLA, LFM2, Gemma4, Nemotron-H, Granite, and Llama paths are wired. More architectures are added through the documented model seam rather than a broad Transformers fallback. |
| Hardware validation | Ada / sm_89 / RTX 4090 is the reference architecture. Hopper and Blackwell paths are present, with feature-specific live gates and ongoing validation. |

### Out of scope

| Area | Reason |
|---|---|
| Generic Transformers fallback | The server owns model code and kernel/capture contracts directly; unsupported architectures fail loudly instead of running slow or subtly wrong fallback paths. |
| Cross-repo private TKV imports | `arbi-serve` intentionally consumes only public `tkv.runtime.*`, `tkv.codec`, and `tkv.core` surfaces. |
| Silent compatibility aliases | First-party consumers are updated in lockstep; stale shims are avoided. |
| Returning plausible data from stubs | Unwired model/backend/quant paths raise `NotImplementedError` with a precise boundary. |
| Distributed prefix cache across server instances | Current cache isolation is per-process/per-tenant. Cross-instance cache sharing is a separate product architecture. |


- [Public release scope](docs/public-release-scope.md) — files retained in or
  excluded from the public distribution.
- [Supported models](docs/supported_models.md) — architecture coverage
  and tiering.
- Runtime flags — `python -m arbi_serve.runtime_flags --doc` lists every `ARBI_*` / `TKV_*`
  runtime config knob (name, type, default, allowed values, impact).
  Auto-generated from `arbi_serve.runtime_flags`.
- [API schemas](docs/SCHEMAS.md) — request / response shapes.
- [Slim image](docs/slim-image.md) — production-tuned Docker image.
- [Container privilege](docs/container-privilege.md) — the engine serves as a
  non-root uid with every capability dropped; what was measured to establish
  that nothing here needs privilege, and the one stated exception.
- [Benchmark measurement guidance](docs/bench-measurement-footguns.md) —
  controls and pitfalls for collecting comparable receipts.

## Repo layout

- `arbi_serve/` — the importable package (backends, scheduler, server,
  models, …). See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the per-
  module breakdown.
- `client/` — generated OpenAI-style client.
- `Dockerfile` + `compose.yaml` — container build and general-purpose stack.
- [`docker/`](docker/) — model-specific Docker Compose recipes.
- `docs/`, `examples/` — user-facing reference and runnable snippets.
- `tests/` — pytest suite (CPU, GPU, and live tiers).

## Contributing

Issues and PRs welcome. See [`CONTRIBUTING.md`](CONTRIBUTING.md) for
development setup, test tiers, and style.

Security reports: please email **security@arbi.city** rather than
filing a public issue. See [`SECURITY.md`](SECURITY.md).

## License

Apache-2.0. See [`LICENSE`](LICENSE).
