Metadata-Version: 2.4
Name: flyweight-llm
Version: 0.25.0
Summary: Native GGUF and safetensors inference runtime for Qwen, Laguna, K2-Horizon, Muse, DeepSeek-V4, Gemma and BailingMoE3 models
License-Expression: Apache-2.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: Jinja2>=3.1
Provides-Extra: client
Requires-Dist: openai>=2.0; extra == "client"
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Requires-Dist: pytest-xdist>=3; extra == "test"
Provides-Extra: vision
Requires-Dist: Pillow>=10; extra == "vision"
Dynamic: license-file

# Flyweight

Flyweight serves GGUF and safetensors language models from a native C++/CUDA
runtime, behind OpenAI- and Anthropic-compatible HTTP APIs and a bundled chat
UI. It is built for one NVIDIA card (AMD RDNA cards run too, experimentally,
through ROCm) and a lot of host RAM: mixture-of-experts
checkpoints larger than VRAM run with their routed experts on the CPU, or
split between the two, and the runtime measures what fits rather than asking.

| Family | Formats | Highlights |
| --- | --- | --- |
| Qwen 3 / 3.5 / 3.6 / 3.8, dense and MoE | GGUF; safetensors (3.5 family) | Full feature set: multi-token prediction, expert offload, image input via `mmproj` |
| Qwen3.8-Flash-Next (`qwen4exp`) | GGUF | Hybrid DeltaNet + gated attention; image input; MTP with a draft block or `--mtp-model` |
| DeepSeek-V4 / V4-Flash | GGUF | Its own CPU/hybrid runtime; DSpark speculative drafts via `--mtp-model`; DSML tool calls |
| BailingMoE3 (Ling 3.0) | GGUF; safetensors | Its own runtime with per-slot snapshot prefix reuse |
| K2-Horizon (dense, MoVA) | GGUF | Grouped norms, softplus attention gate; the MoVA value experts page through their own cache |
| Gemma 4 | GGUF | Routed experts must be Q4_0 (the QAT release); no MTP |
| Laguna 2.1 | GGUF | Per-head attention gate; no MTP |
| Muse Glimmer | GGUF | Channel-tagged reasoning; native ATEM tool calls; no speculative decoding |
| LFM2 / LFM2-MoE (LFM2.5) | GGUF | Gated short-conv + GQA hybrid; native pythonic tool calls; no speculative decoding |
| Nemotron 3.5 Bolt (`nemotron_h_moe`) | GGUF | Mamba-2 + GQA + latent-MoE hybrid; routed experts on the CPU; no speculative decoding |

Image generation with [Z-Image-Turbo](https://huggingface.co/Tongyi-MAI/Z-Image-Turbo)
or [Qwen-Image-2.1](https://huggingface.co/Qwen/Qwen-Image-2.1) (RGBA out)
runs on the same engine beside a chat model (`--image-model`).

## Install

Wheels are on PyPI as `flyweight-llm` for Linux x86-64 (manylinux 2.28) and
Windows x64. The import package and the command are `flyweight`:

~~~bash
python -m venv .venv && source .venv/bin/activate   # .venv\Scripts\Activate.ps1 on Windows
pip install flyweight-llm
flyweight doctor
~~~

`doctor` reports whether this machine can serve and names the fix for
anything missing. Only `FAIL` lines block; a missing GPU is a warning, since
`--backend cpu` serves without one.

### GPU requirements

Nothing CUDA is linked into the wheel. At startup the runtime loads the
NVIDIA driver and NVRTC, compiles its kernels for your card, and reads the
CUDA headers (`cuda_fp16.h`, CCCL) from `CUDA_PATH`, `CUDA_HOME`,
`/opt/cuda` or `/usr/local/cuda`. So you need:

- the proprietary NVIDIA driver (`nvidia-smi` shows a device),
- the CUDA toolkit for NVRTC and the headers: the `cuda` package on Arch,
  `cuda-toolkit` on Debian, Ubuntu and Fedora, or NVIDIA's installer. `nvcc`
  is never used.

cuBLAS is optional and used when present. Any compute capability runs; int8
tensor-core kernels need 7.5 or newer, and the NVFP4 prefill path needs
Blackwell.

### AMD GPUs (experimental)

`--backend rocm` runs the same kernels on AMD RDNA GPUs (Radeon RX 6000, 7000
and 9000, and RDNA APUs) through HIP, Linux only. `auto` tries it after CUDA.
Nothing ROCm is linked into the wheel either: the runtime loads `libamdhip64`
and `libhiprtc` from the loader path or `ROCM_PATH` (default `/opt/rocm`),
and hipRTC compiles the kernels at startup (about 30 s, cached on disk by
ROCm afterwards). You need the HIP runtime -- `hip-runtime-amd` on Arch, or
AMD's ROCm packages elsewhere -- and your user in the `render` group.
`flyweight doctor` reports what it finds.

What to expect: output agrees with the CUDA path (greedy tokens can differ at
near-ties, as between any two GPUs). On RDNA3 (Radeon RX 7000) the prompt's
quantized projections run on the WMMA matrix cores, about twice the prompt
throughput of the dot-product kernels on a Qwen3.5-4B Q4_K_M (RX 7600);
`FLYWEIGHT_WMMA=0` turns them off. RDNA2 has no matrix cores and RDNA4's are
not used yet, and prompt attention runs on scalar kernels everywhere, so
prompt processing is still below what the card can do. CDNA (Instinct) cards are refused: the kernels
assume a 32-lane warp, and CDNA only runs 64-lane wavefronts. An APU ROCm does
not list may need `HSA_OVERRIDE_GFX_VERSION` (10.3.0 for RDNA2, 11.0.0 for
RDNA3).

Unsupported APUs can be unreliable. On a Radeon 610M (gfx1036) with ROCm 7.2,
the driver occasionally drops a kernel launch or runs a kernel ahead of the
copy queued before it. A plain HIP loop shows this with Flyweight not
involved. Mixture-of-experts models, whose CPU expert path churns host memory,
then decode garbage in a fraction of runs. Treat output from a GPU ROCm does
not list as untrusted. A discrete RDNA2/3/4 card, or an APU on ROCm's support
list, is what this backend targets.

### From source

Needed on macOS, on ARM, and for developing Flyweight. Python 3.11+, CMake
3.24+, and a C++20 compiler: GCC 13+ or Clang 16+ on Linux, MSVC v143 (any
Visual Studio 2022 edition or the Build Tools) on Windows. Ninja is optional
but makes the Windows build parallel.

~~~bash
git clone https://github.com/yairpatch/flyweight
cd flyweight
pip install .            # compiles the runtime; a few minutes
flyweight doctor
~~~

If `pip install` fails on the compiler, `PYTHONPATH=src python -m flyweight
doctor` from the checkout says which of CMake, the compiler and the build
tool it cannot see. If the `flyweight` command is not on PATH, `python -m
flyweight` takes the same arguments.

## Quick start

~~~bash
flyweight serve model.gguf
~~~

Open `http://127.0.0.1:8000/` for the chat UI, or call the API:

~~~bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "local", "messages": [{"role": "user", "content": "Say hi."}], "max_tokens": 64}'
~~~

`MODEL` is a `.gguf` file (the first shard of a split archive) or a
safetensors checkpoint directory. The defaults choose the backend and memory
layout from what the machine has; the flags below are the ones worth
knowing. `flyweight serve --help` lists everything, grouped.

## Serving

### Context and memory

| Flag | Default | Meaning |
| --- | --- | --- |
| `--context N` | 32768 | prompt plus output tokens per sequence |
| `--max-tokens N` | 4096 | output cap for requests that do not set one |
| `--parallel N` | 1 | independent sequence slots, each with its own KV cache |
| `--scratch-context N` | 0 | smaller context for the slots past the first |
| `--cache auto\|off\|MIB` | auto | host-RAM cache for displaced conversations, restored by longest prefix. `auto` is one eighth of free RAM, capped at 8 GiB |
| `--cache-dir DIR` | off | keep that cache on disk across restarts: saved at shutdown (Ctrl-C or SIGTERM), restored at startup within the `--cache` budget. Each model and configuration gets its own subdirectory, and the three most recent are kept |
| `--cache-type-k`, `--cache-type-v` | f16 | KV precision: `f32`, `f16`, `bf16`, `q8_0`, `turbo3`, `turbo4` |
| `--gpu-cache-mib N` | 0 | VRAM for the expert cache; 0 sizes it from what is free at startup |
| `--expert-mode` | auto | where routed experts run, below |
| `-ngl N\|auto\|all` | auto | layers kept on the GPU, as llama.cpp's `-ngl`: a dense model's FFN blocks, or a MoE model's whole expert layers (the rest run on the CPU). Attention and KV stay on the GPU |
| `--cpu-threads N` | 0 | CPU expert workers; 0 picks the physical cores |
| `--dense-requant auto\|q8\|off` | off | optionally repack BF16 dense weights to Q8_0 on the GPU; `auto` uses VRAM pressure and `q8` forces conversion |
| `--mtp-drafts N` | 0 | speculative decode with the checkpoint's draft block, up to 8 |
| `--mtp-model PATH` | | a standalone draft GGUF (Qwen MTP files, DSpark for DeepSeek-V4) |
| `--backend auto\|cuda\|rocm\|cpu` | auto | `rocm` runs on an AMD RDNA GPU; `cpu` runs every kernel on the host |

Expert modes:

| Mode | Prompt experts | Decode experts | Use |
| --- | --- | --- | --- |
| `auto` | CPU | hot set on the GPU, misses on the CPU | default |
| `cpu` | CPU | CPU | least VRAM |
| `resident` | GPU | GPU | refuses to start unless every expert fits |

The older `hybrid`, `gpu`, `legacy-hybrid` and `legacy-paging` values are
accepted as aliases. On `--backend cpu` the mode is forced to `cpu`.

Prompt caching is automatic and needs no flag: a conversation that leaves
the GPU is packed into the host cache and restored on its next request, and
mid-prompt checkpoints (`--prefill-checkpoint-interval`, default 256 tokens)
let a conversation that diverged part-way reuse the part it shares.

### Choosing KV precision

`q8_0` halves the KV cache and `turbo4` quarters it. That matters most on a
dense model with a wide `head_dim`, where KV competes with the weights for
VRAM and every dense block that loses is re-read over PCIe on each token.
Qwen3.8-27B (UD-IQ2_XXS) on a 12 GB card:

| context | KV | dense blocks spilled | decode |
| --- | --- | --- | --- |
| 16K | `f16` | 5 of 64 | 16.6 tok/s |
| 16K | `q8_0` | none | 23.2 tok/s |
| 32K | `f16` | 16 of 64 | 10.8 tok/s |
| 32K | `q8_0` | 5 of 64 | 15.3 tok/s |
| 32K | `turbo4` | none | 22.9 tok/s |

If the startup banner reports dense blocks on the CPU, spend KV precision to
buy them back before anything else. `auto` only reaches for `turbo4` on a
MoE checkpoint above 32K context.

### Public deployment

~~~bash
flyweight serve model.gguf --host 0.0.0.0 --api-key "$KEY" \
  --concurrency 8 --max-connections 64 --cors-origin https://app.example
~~~

- `--api-key` (or `FLYWEIGHT_API_KEY`) requires `Authorization: Bearer` or
  `x-api-key` on every route except the UI's static files. No key means no
  authentication.
- `--concurrency` (default 64) bounds requests admitted to inference; the
  rest get HTTP 429 with `Retry-After`. `--max-connections` (default 128)
  bounds HTTP threads, `--request-timeout-seconds` (default 30) the time a
  client may take to send its body.
- `--strict-model` rejects requests naming a model other than `--model-name`.
- Request bodies are capped at 16 MiB.

### The server log

One row per request:

~~~
          endpoint   prompt  cached   ttft     out   tok/s  finish        total
10:49:45  chat        26.5k     98%   1.7s     112    35.8  tool call      4.9s
11:30:01  chat           --      --     --      --      --  400         0.1s  prompt is too long: 70212 tokens > 65536 maximum
~~~

`prompt` is the rendered prompt length, `cached` how much of it the prefix
cache served, `ttft` the wait before the first token, `finish` why generation
stopped. A low `cached` on a conversation that only appended is what a cache
problem looks like. `--quiet` prints failures only; `--verbose` adds the
access log and the prefix-cache diagnostics. Colour is off when redirected
or under `NO_COLOR`.

### Chat UI

The single-page app at `/` reaches everything the server exposes: chat over
any of the three protocols with a collapsible thinking panel and an **Answer
now** button, tool definitions with cards to paste results into, image
attachments when a vision tower is loaded, PDF, Word, Excel and text
attachments always (extracted in the browser, never uploaded as files), a
settings panel for every sampling and reasoning knob, a runtime dashboard
over `/health`, a tokenizer view, a raw-completions playground, an image
studio when `--image-model` is set, and an inspector that keeps each
request's body, SSE frames and an equivalent `curl` line. Conversations live
in the browser's IndexedDB; the API key lives in session storage.

### Codex CLI

Codex CLI can use Flyweight as a custom provider through its Responses API.
Start Flyweight with an API key if you want the local endpoint authenticated:

~~~bash
FLYWEIGHT_API_KEY=local-dev-key flyweight serve model.gguf
~~~

Add a provider to `~/.codex/config.toml`:

~~~toml
model_provider = "flyweight"
model = "local"
model_reasoning_effort = "medium"

[model_providers.flyweight]
name = "Flyweight"
base_url = "http://127.0.0.1:8000/v1"
wire_api = "responses"
env_key = "OPENAI_API_KEY"
~~~

Set `OPENAI_API_KEY=local-dev-key` in the shell that launches Codex when the
server uses `FLYWEIGHT_API_KEY`. If the server has no API key configured,
`env_key` can be omitted. Use the model name reported by `GET /v1/models` if
you enabled `--strict-model`.

Codex may include built-in Responses tools such as hosted web search in its
request. Flyweight ignores non-function tool entries; function tools are
passed through to the model and returned for Codex to execute.

## API

| Route | Protocol |
| --- | --- |
| `POST /v1/chat/completions`, `POST /v1/completions` | OpenAI |
| `POST /v1/responses`, `GET`/`DELETE /v1/responses/{id}`, `POST /v1/responses/input_tokens` | OpenAI Responses; the 128 most recent are kept, `store: false` skips one |
| `GET /v1/models`, `GET /v1/models/{id}`, `GET /v1/me` | OpenAI |
| `POST /v1/messages`, `POST /v1/messages/count_tokens` | Anthropic |
| `POST /v1/images/generations`, `/v1/images/edits` | OpenAI Images, with `--image-model` (edits: Qwen-Image-2.1) |
| `POST /v1/chat/completions/{id}/stop_thinking`, `POST /v1/messages/{id}/stop_thinking` | Flyweight |
| `GET /health`, `GET /props`, `GET /slots`, `POST /tokenize`, `POST /detokenize` | llama.cpp-style |

Every generation route streams over SSE with `"stream": true`; chat streams
honour `stream_options.include_usage`. `usage` reports `cached_tokens` (the
prompt prefix reused) and `reasoning_tokens`. `logprobs`, `logit_bias` and
`n` other than 1 are rejected with 400 rather than ignored.

Chat requests render through the checkpoint's own `chat_template`. A
`generation_config.json` beside the model supplies sampling defaults;
otherwise the defaults are llama.cpp's (temperature 0.8, `top_k` 40, `top_p`
0.95, `min_p` 0.05, penalties off). Precedence is request, then server flag,
then `generation_config.json`, then built-in. `GET /props` reports the
resolved defaults, the reasoning-effort levels the template accepts, and a
`capabilities` list.

### Thinking

- `reasoning_effort` (`low`, `medium`, `high`, `xhigh`, or `none` for off)
  is read from the flat field, `reasoningEffort`, Responses-style
  `reasoning.effort`, `chat_template_kwargs.reasoning_effort`, and Anthropic
  `output_config.effort`, so Claude Code, opencode and pi all work unchanged.
  A level the template does not name maps to its nearest neighbour.
  `--reasoning-effort` sets the server default; `none` there means the server
  does not reason unless a request asks.
- `reasoning_budget_tokens` is a hard ceiling: at the limit the sampler
  closes the thinking block and the answer resumes. Anthropic's `thinking:
  {"type": "enabled", "budget_tokens": N}` maps onto it, and a `/v1/messages`
  request that thinks without a budget gets half its `max_tokens` or 2048,
  whichever is smaller. `--thinking-budget N` applies one cap everywhere; 0
  removes it.
- The `stop_thinking` routes interrupt a live stream's thinking on the next
  token. `{id}` is the id from the stream's first event.
- `prefill_progress: true` on a streaming request adds `ping` frames with
  `flyweight.prefill` progress while a long prompt is evaluated.

Chain-of-thought arrives in `reasoning_content` on OpenAI routes and as
`thinking` blocks on `/v1/messages`, never in `content`.

### Tools and structured output

Declared tools are enforced by a sampler grammar: the name must be declared,
required parameters present, array and object values well-formed JSON.
`response_format` (`json_object`, `json_schema`; `text.format` on
`/v1/responses`) is enforced the same way. Tool-call arguments stream
incrementally where the format supports it. DeepSeek-V4, BailingMoE3 and
K2-Horizon render tools through their own templates and are parsed but not
grammar-enforced; Muse Glimmer uses its native ATEM format and supports one
tool call per turn. LFM2 writes its calls as a Python-style list between
`<|tool_call_start|>` and `<|tool_call_end|>`; they are parsed, not
grammar-enforced. Other families get the Hermes-style tool prompt and
grammar.
`parallel_tool_calls: false` caps a turn at one call, and
`--max-tool-call-tokens` bounds a runaway one.

Inside a tool call, repetition penalties pause and temperature is capped at
0.2, because an edit tool's arguments must reproduce file text exactly.

### Sampling

`temperature`, `top_k`, `top_p`, `min_p`, `repetition_penalty` (1 = off,
looks over the last `penalty_window` = 64 generated tokens),
`presence_penalty`, `frequency_penalty`, `seed`, `stop`. Each has a server
flag of the same name. A heavily quantized checkpoint that locks onto a line
usually wants `repetition_penalty: 1.1`; otherwise leave it off, since it also
penalizes quoted file content.

### Images in

Qwen 3.5-family and Qwen3.8-Flash-Next GGUFs accept images when their vision
tower is attached. The tower is the `mmproj-*.gguf` published beside the
model; decoding needs Pillow (`pip install 'flyweight-llm[vision]'`):

~~~bash
flyweight serve Qwen3.5-35B-A3B-Q6_K.gguf --mmproj mmproj-Qwen3.5-35B-A3B-BF16.gguf
~~~

OpenAI `image_url`, Responses `input_image` and Anthropic `image` parts are
accepted, as `data:` or `http(s)` URLs (`--image-urls deny` refuses the
latter). Each image is resized to at most `--image-max-tokens` (default 1024,
one token per 32x32 block), counted as prompt tokens, and never re-encoded
when it sits inside a reused prefix. Without a tower an image part degrades
to a visible `[image omitted]` note rather than failing the request.

### Images out

`--image-model DIR` loads a diffusers snapshot of
[Z-Image-Turbo](https://huggingface.co/Tongyi-MAI/Z-Image-Turbo) or
[Qwen-Image-2.1](https://huggingface.co/Qwen/Qwen-Image-2.1)
(`text_encoder/`, `transformer/`, `vae/`, as `hf download` leaves it) and
serves it at `/v1/images/generations` and in the UI's image studio:

~~~bash
flyweight serve Qwen3.6-35B-A3B-Q6_K.gguf --image-model /path/to/Z-Image-Turbo
curl http://127.0.0.1:8000/v1/images/generations -H 'Content-Type: application/json' \
  -d '{"prompt": "a red bicycle leaning on a brick wall", "size": "1024x1024", "seed": 7}'
~~~

The request takes `prompt`, `size` (on the model's grid -- multiples of 16
for Z-Image, 32 for Qwen-Image-2.1 -- up to `--image-max-size`, which
defaults to the model's own maximum: 1024 and 2048), `n` (1 to
4), `seed`, `steps` (the model's own default: 8 for Z-Image, 40 for
Qwen-Image-2.1) and `shift` (0 lets Qwen-Image-2.1's shift follow the
image's size, as its pipeline does); the response is base64 PNG. Qwen-Image-2.1
renders RGBA: the PNG carries an alpha channel, and a prompt starting "This is
an RGBA image with transparency" is what makes the model use it. It also
edits: up to ten reference images go in as `images` (data URLs or base64) on
the same request, or as `image` parts of an OpenAI-shaped multipart
`POST /v1/images/edits`; the picture takes the last reference's shape unless
`size` is set, and the studio has an attach button for them. One render at a
time; a concurrent request gets
429. The encoder and DiT are quantized to Q8_0 on first open and cached
beside the checkpoint. The workspace starts sized for 1024x1024 and grows when a request needs
more; a render the card cannot fit fails on its own rather than at startup,
and `--image-reserve` holds the whole `--image-max-size` from the start
instead. `--image-weights` keeps them on the GPU (`device`) or
streams them from pinned RAM (`host`, about 1.7 GB of VRAM at 1024x1024);
`auto` picks `host` when they would take more than half the card, which is
what lets a 12 GB card serve a 35B chat model and Z-Image together.
`--image-precision balanced` (default) runs bf16 tensor-core GEMMs; `fast`
uses int8; `exact` keeps f32 activations at about eight times the time. A
1024x1024 render takes about 14 s on an RTX 5070 Ti laptop. Render at 1024:
the model is trained there.

Qwen-Image-2.1 also supports GGUF DiT checkpoints (e.g. from
[Abiray/Qwen-Image-2.1-GGUF](https://huggingface.co/Abiray/Qwen-Image-2.1-GGUF),
such as `qwen_image_2.1_Q6_K.gguf` or `Q4_K_M.gguf`). Use `--image-transformer PATH`
to point directly to the GGUF file alongside the snapshot:

~~~bash
flyweight serve Qwen3.8-27B-UD-IQ2_XXS.gguf \
  --image-model /path/to/Qwen-Image-2.1 \
  --image-transformer /path/to/qwen_image_2.1_Q6_K.gguf
~~~

## Models and formats

### GGUF

Split archives (`-00001-of-0000N`) load from their first shard. Every
quantization type llama.cpp writes is readable, plus NVFP4 and the Q2_0
type. Which kernel serves a tensor depends on where it runs:

- **Dense projections on the GPU**: all of F32, F16, BF16, the K-quants,
  Q4_0, Q8_0, the IQ family including IQ4_NL, IQ1_S and IQ1_M. Q2_0 dense
  tensors are requantized to Q8_0 on upload.
- **Routed experts on the GPU** (grouped kernels): Q2_K, Q4_K, Q5_K, Q6_K,
  Q4_0, Q8_0, Q2_0, IQ1_S, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS,
  IQ4_NL and NVFP4. Q3_K experts run on the CPU.
- **Routed experts on the CPU**: every type above except IQ1_M, with
  AVX-512 VNNI kernels where the host has them. IQ1_M routed experts are
  unsupported, and an IQ1_M LM head is requantized to Q8_0.
- **BailingMoE3** has grouped GPU expert kernels for Q4_K and Q6_K only;
  other types run one expert at a time, correctly but slowly.

### Safetensors

A safetensors checkpoint (Qwen 3.5 family and BailingMoE3) is packed to a
quantization on first open and cached beside it. On a terminal the CLI asks
which, listing sizes and marking cached ones; `--quant` or
`FLYWEIGHT_HF_QUANT` answers ahead of time, and a non-interactive run takes
`Q6_K`. Choices: `IQ2_XS`, `Q2_K`, `IQ3_XXS`, `Q3_K`, `IQ4_XS`, `Q4_K`,
`Q5_K`, `Q6_K`, `Q8_0`, `F32`.

- Pick the largest target that fits VRAM, not the largest you can pack. A
  dense block that does not fit runs on the CPU at about 3 ms per token.
  `--gpu-cache-mib` pins which blocks spill on a card shared with a desktop.
- `Q2_K` and `Q3_K` are refused on MoE checkpoints. `IQ4_XS` is the smallest
  MoE target below `Q4_K` whose experts stay on the GPU.
- `IQ2_XS` needs an importance matrix. An `imatrix.dat` beside the checkpoint
  is used automatically; `--imatrix PATH` names one elsewhere and `off`
  disables it. `flyweight imatrix MODEL --text calibration.txt` gathers one
  over any Qwen-family model, in llama.cpp's `.dat` layout.
- `IQ3_XXS` packing takes minutes rather than seconds, once.
- A spilled dense block in a codebook format is re-encoded to Q3_K for the
  host kernels, which is lossy. `FLYWEIGHT_HOST_FFN_FORMAT` picks `q2_k`,
  `q3_k`, `q8_0` or `off`.

### Speculative decoding

| Family | Mechanism |
| --- | --- |
| Qwen (all) | `--mtp-drafts N` with the checkpoint's draft block or a `--mtp-model` overlay. Drafting is trialled against plain decode and kept only when it wins |
| DeepSeek-V4-Flash | `--mtp-model` pointing at the DSpark draft GGUF |
| BailingMoE3 | draft block loads; the server does not drive it yet |
| Qwen without a draft block | `FLYWEIGHT_LOOKUP_DRAFTS=N` (up to 7): prompt-lookup self-speculation, worth 10-15% on code, neutral on prose |
| Gemma 4, Laguna, Muse Glimmer, K2-Horizon, LFM2, Nemotron H | none |

Every verified token passes through the request's own sampler, so a
drafting request answers as a non-drafting one would.

## Other commands

| Command | What it does |
| --- | --- |
| `flyweight generate MODEL --prompt TEXT` | print one response and exit (`--max-tokens`, `--temperature`, `--seed`, `--enable-thinking`) |
| `flyweight benchmark MODEL` | preparation, prefill and steady decode timings as JSON (`--chat`, `--iterations`, `--cold-cache`) |
| `flyweight inspect MODEL` | metadata, resolved config and tensor list as JSON |
| `flyweight plan MODEL` | load with serve's placement flags and print where weights, KV and experts went in VRAM and RAM (`--json`); serve prints the same at startup |
| `flyweight imatrix MODEL --text FILE` | gather an importance matrix |
| `flyweight probe MODEL` | run a few tokens and dump runtime counters |
| `flyweight doctor` | check the install |
| `flyweight transcript-audit DIR` | explain a coding harness's failed edits from a request dump |

For repeatable comparisons, `python -m flyweight.runtime_benchmark run` writes
JSONL across prompt lengths and `compare` diffs two files. Run GPU benchmarks
alone on the card: another process changes free VRAM and therefore the
automatic expert-cache size.

`transcript-audit` answers one question: when a harness's edit replaced text
that was not in the file, did the model edit blind, or did the file content
get lost between the client and the model? With
`FLYWEIGHT_TRANSCRIPT_DUMP=DIR` set, `serve` writes one JSON file per request
holding both what the client sent and what the model saw.

## Environment variables

| Variable | Meaning |
| --- | --- |
| `FLYWEIGHT_API_KEY` | default for `--api-key` |
| `FLYWEIGHT_HF_QUANT`, `FLYWEIGHT_HF_IMATRIX` | defaults for `--quant` and `--imatrix` |
| `FLYWEIGHT_HF_CACHE` | where packed safetensors go; `off` disables the cache |
| `FLYWEIGHT_TRANSCRIPT_DUMP`, `FLYWEIGHT_TRANSCRIPT_PROMPT=0` | request recording for `transcript-audit`; `=0` stores a digest instead of the prompt |
| `FLYWEIGHT_LOOKUP_DRAFTS` | prompt-lookup speculation, above |
| `FLYWEIGHT_DS4_EXPERT_CACHE_MIB` | a GPU expert cache for DeepSeek-V4 |
| `FLYWEIGHT_QSA=1` | Qwen3.8-Flash-Next's learned sparse-attention indexer (experimental; dense by default) |
| `FLYWEIGHT_V2_MLOCK=1` | lock the mapped model in RAM |
| `FLYWEIGHT_TOOL_GRAMMAR=0`, `FLYWEIGHT_RESPONSE_GRAMMAR=0` | disable the tool and JSON sampler constraints |
| `CUDA_PATH`, `CUDA_HOME` | where NVRTC and the CUDA headers are looked for |
| `ROCM_PATH` | where the HIP runtime and hipRTC are looked for after the loader path |

The native runtime reads many more `FLYWEIGHT_*` switches (`_PROFILE`,
`_TRACE`, kernel A/B toggles such as `FLYWEIGHT_CUDA_GRAPHS=0` and
`FLYWEIGHT_PREFILL_PIPELINE=0`). They exist for measurement; leave them unset
when serving.

## Limitations

- CUDA is the main accelerator. ROCm (`--backend rocm`) is experimental:
  RDNA only, Linux only, no tensor-core kernels, cuBLAS and NVFP4 paths off.
  `--backend cpu` runs everything on the host.
- Two platforms ship wheels; everything else builds from source.
- No embeddings, audio, fine-tuning or hosted-tool APIs. Response records
  are process-local, and so are prompt caches unless `--cache-dir` is set.
- Qwen3.8-Flash-Next runs its sparse-attention layers as dense attention by
  default, exact within the trained 2048-token selection budget and an
  approximation beyond it. MTP needs a release with a draft block (the
  Q4_K_XL builds) or the standalone MTP file.
- Vision covers still images through a GGUF `mmproj` on the Qwen 3.5 family
  and Flash-Next. The safetensors loader drops the tower. Towers with
  deepstack layers are refused.
- Gemma 4 has no MTP and its routed experts must be Q4_0 (the QAT release).
- BailingMoE3 interleaves its slots rather than batching them, so
  `--parallel` removes waiting but does not multiply throughput, and it has
  no expert paging: a model that does not fit runs on the host entirely.
- Laguna supports only the per-head attention gate; larger Laguna
  checkpoints with the per-element gate are rejected at load.
- LFM2 decodes its `--parallel` slots one after another rather than in one
  batched step, and LFM2-VL's vision tower is not wired.
- Nemotron H runs its routed experts on the CPU (no GPU kernel for its gateless
  relu² experts yet), so `--expert-mode auto` keeps them there and `resident`
  is refused; its `--parallel` slots decode one after another.
- Text is not NFC-normalized before tokenizing, so a decomposed accent can
  tokenize differently from the reference tokenizer.
- Special-token spellings inside message content (`<|im_start|>`,
  `<tool_call>`, `<think>`) are tokenized as the control tokens, as the HF
  and llama.cpp tokenizers do. A client relaying untrusted text should strip
  them.

## Development

~~~bash
pip install -e '.[test]'
python -m flyweight.native_build     # same build tree, plus contract tests and benchmarks
ctest --test-dir build/native --output-on-failure
pytest -q                            # ~3 minutes; --run-slow adds the parity tests
ruff check src tests bench tools native/tools setup.py && mypy src/flyweight
~~~

Never keep an editable and a regular install in one environment; the regular
one wins every import, and `doctor` reports the shadowing. Never copy a new
library over a loaded one; `native_build` installs it by rename.

The chat UI is a Vite + React app in `web/`, and its build is committed under
`src/flyweight/ui/` so the wheel ships it without Node. After changing
`web/`, run `pnpm build` there and commit the output; CI fails on a stale
bundle.

`FLYWEIGHT_TEST_MODEL=/path/to/model.gguf` opts the suite into the real-model
tests. `tools/check_*.py` are parity checks that need real weights.

Where things live:

- `native/src/v2_runtime.cpp`: GGUF parsing, memory planning, scheduling,
  prefix reuse, sampling and the C ABI; `v2_mtp_verifier.inc` (prefill
  driver and MTP), `v2_vision.inc` (the tower), `v2_diffusion.inc` (the image tower, Z-Image), `v2_qwenimage.inc` (Qwen-Image-2.1)
- `native/src/gpu_driver.cpp`: CUDA driver, NVRTC, cuBLAS, graphs, transfers;
  the same table filled from HIP and hipRTC for ROCm, whose translation of the
  kernel corpus lives in `flyweight_hip_prelude.hpp`
- `native/include/flyweight_v2_qwen_kernels.hpp` and siblings: the CUDA
  kernel source, JIT-compiled by NVRTC and also compiled as host C++ for
  `--backend cpu`; `flyweight_v2_format_dispatch.hpp` maps tensor formats to
  kernels
- `native/include/flyweight_v2_bailing.hpp`, `flyweight_v2_deepseek4*.hpp`,
  `flyweight_v2_hf*.hpp`, `flyweight_v2_tool_grammar.hpp`: the BailingMoE3
  and DeepSeek-V4 runtimes, the safetensors loader and packer, the sampler
  grammar
- `src/flyweight/cli.py`, `v2.py`, `v2_server.py`, `server.py`: the command
  line, the ctypes bindings, the engine thread, the HTTP protocols;
  `deepseek4_server.py` and `dspark.py` for DeepSeek-V4; `images.py` and
  `vision.py` for image generation and input
- `plans/`: dated design records, each with a status line, for what was
  built and what was measured and dropped

See [CONTRIBUTING.md](CONTRIBUTING.md) for how changes land, including the
version-bump rule, and [SECURITY.md](SECURITY.md) for reporting a
vulnerability.

## License

Apache-2.0.
