Metadata-Version: 2.4
Name: glrmask
Version: 0.3.0
Requires-Dist: numpy>=1.21
License-File: licenses/LICENSE-MIT
License-File: licenses/LICENSE-APACHE
License-File: licenses/NOTICE-LARK
Summary: Extremely fast grammar-constrained decoding for LLMs
License: MIT OR Apache-2.0
Requires-Python: >=3.9, <3.14
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/IsaacBreen/glrmask/blob/v0.3.0/CHANGELOG.md
Project-URL: Documentation, https://glrmask.com/reference/python/
Project-URL: Repository, https://github.com/IsaacBreen/glrmask

# GLRMask

<!-- Generated by scripts/sync-guide.py from docs/content. Do not edit this output. -->

GLRMask is a grammar-constrained decoding library for high-throughput LLM serving. It is optimized for fast mask computation with extremely low tail latency, even for complex grammars.

GLRMask supports general context-free grammars, including ambiguous ones. Unfortunately, there is no universally accepted EBNF dialect. In keeping with this tradition, GLRMask includes its own: GLRM. It also supports JSON schemas and Lark.

[Website](https://glrmask.com) · [GitHub](https://github.com/IsaacBreen/glrmask) · [Paper](https://doi.org/10.5281/zenodo.23273556)

## Benchmarks

### JSON Schema

Time between masks. Lower is better.

<p><img src="https://raw.githubusercontent.com/IsaacBreen/glrmask/main/docs/assets/benchmark-tbm-json-2026-10-14.png" alt="JSON Schema time between masks: measured GLRMask Static and llguidance 1.8.0 curves; logarithmic exceedance probability and latency axes. Lower is better." width="480"></p>

**TBM (µs)**

| Statistic | Static / FastRuntime | Dynamic / FastBuild | Balanced / O2 | llguidance |
|---|---:|---:|---:|---:|
| Mean | 3.0 | 31.5 | 12.7 | 25.9 |
| p50 | 2.6 | 9.1 | 6.9 | 10.6 |
| p90 | 5.1 | 86.4 | 30.3 | 40.7 |
| p99 | 12.2 | 239.5 | 74.8 | 289.9 |
| p99.9 | 24.9 | 754.2 | 206.1 | 1,092.6 |
| Max | 283.1 | 3,647.8 | 2,140.9 | 6,931.5 |

**Time to first mask (ms)**

| Statistic | Static / FastRuntime | Dynamic / FastBuild | Balanced / O2 | llguidance |
|---|---:|---:|---:|---:|
| Mean | 33.7 | 3.9 | 12.3 | 1.2 |
| p50 | 12.6 | 2.3 | 6.2 | 0.8 |
| p90 | 78.9 | 6.5 | 19.0 | 1.8 |
| p99 | 302.3 | 35.9 | 125.5 | 11.1 |
| p99.9 | 854.3 | 88.8 | 445.0 | 17.9 |
| Max | 1,293.4 | 274.6 | 631.1 | 231.0 |

14 October 2026 · 11,277 complete valid trajectories accepted by all four engines · 1,734,930 TBM positions per engine · 8,326 common-built cases for TTFM. One unpinned native wall-clock timing run on AWS c8i.8xlarge (Intel Xeon 6975P-C, 32 available CPUs). Vocabulary preparation is excluded.

### JavaScript

Time between masks. Lower is better.

<p><img src="https://raw.githubusercontent.com/IsaacBreen/glrmask/main/docs/assets/benchmark-tbm-javascript-2026-10-14.png?v=30b3888e1af3" alt="JavaScript time between masks: measured GLRMask Static and llguidance 1.8.0 curves; logarithmic exceedance probability and latency axes. Lower is better." width="480"></p>

**TBM (µs)**

| Statistic | Static / FastRuntime | Dynamic / FastBuild | Balanced / O2 | llguidance |
|---|---:|---:|---:|---:|
| Mean | 16.9 | 1,392.4 | 389.0 | 1,052.2 |
| p50 | 18.2 | 145.5 | 69.9 | 856.6 |
| p90 | 30.4 | 3,626.1 | 1,088.8 | 2,521.7 |
| p99 | 47.9 | 4,091.7 | 1,509.6 | 3,097.0 |
| p99.9 | 71.1 | 4,656.9 | 2,016.8 | 3,597.1 |
| Max | 77.4 | 5,517.4 | 2,139.0 | 3,867.9 |

**Time to first mask (ms)**

| Statistic | Static / FastRuntime | Dynamic / FastBuild | Balanced / O2 | llguidance |
|---|---:|---:|---:|---:|
| Mean | 5,089.4 | 113.3 | 458.4 | 8.0 |
| p50 | 5,099.6 | 113.0 | 457.8 | 8.0 |
| p90 | 5,152.7 | 116.9 | 464.3 | 8.1 |
| p99 | 5,223.0 | 120.7 | 470.4 | 8.2 |
| p99.9 | 5,225.9 | 121.7 | 472.3 | 8.2 |
| Max | 5,226.2 | 121.8 | 472.5 | 8.2 |

14 October 2026 · 28 complete valid trajectories accepted by all four engines · 3,805 TBM positions per engine · 55 common-built cases for TTFM. One unpinned native wall-clock timing run on AWS c8i.8xlarge (Intel Xeon 6975P-C, 32 available CPUs). Vocabulary preparation is excluded.

#### Dynamic mode

GLRMask’s dynamic mode trades some mask-generation speed for lower cold-start latency. It is heavily based on [llguidance](https://github.com/guidance-ai/llguidance).

<p><img src="https://raw.githubusercontent.com/IsaacBreen/glrmask/main/docs/assets/benchmark-tbm-json-dynamic-2026-10-14.png" alt="JSON Schema time between masks: GLRMask faded, GLRMask dynamic prominent, and llguidance; logarithmic exceedance probability and latency axes. Lower is better." width="480"></p>

[Explore Dynamic JSON results](https://glrmask.com/?dataset=json&mode=dynamic#benchmarks).

There was no CPU pinning or SMT sibling reservation, and checkpoint publication overlapped the benchmark. Saved telemetry cannot isolate scheduling effects; latency stabilization is not certified. Successful builds do not establish semantic correctness. All 9,558 JSON schemas were attempted: build errors are Dynamic 232, Balanced 232, Static 233 and llguidance 1,226. Incorrect examples are 460 for each GLRMask mode and 100 for llguidance; runtime-error examples are 0 and 6 respectively. JavaScript attempts all 55 fixtures: all builds succeed, with 1 incorrect example per GLRMask mode and 4 for llguidance. Headline TBM excludes incomplete or nonaccepted valid trajectories; all outcomes remain in the raw records. This finite corpus does not establish full-language equivalence or whole-corpus dominance.

[Native AWS benchmark details](https://github.com/IsaacBreen/glrmask/blob/main/docs/benchmark-aws-2026-10-14.md). Historical [October 8 Mac JSON](https://github.com/IsaacBreen/glrmask/blob/main/docs/benchmark-cfa-2026-10-08.md) and [October 9 Mac JavaScript](https://github.com/IsaacBreen/glrmask/blob/main/docs/benchmark-javascript-2026-10-09.md) reports are retained with their original measurement protocols. These are source builds, not measurements of the published 0.2 packages.

## Installation

```bash
python -m pip install glrmask
```

## Usage

GLRMask has four core types:

- `Grammar`: an immutable source or constructed grammar, with its bindings.
- `PreparedGrammar`: vocabulary-specific reusable machinery, which can still have unresolved bindings.
- `Constraint`: a complete, runnable grammar compiled against that vocabulary.
- `ConstraintState`: the mutable state for a single generated sequence.

For direct use, compile a `Grammar` into a `Constraint`. Use `prepare()` and `link()` when you need reusable components or late binding. Then call `constraint.start()` to initiate a `ConstraintState`. Then in the decoding loop:

1. Compute the next-token mask.
2. Sample a token the mask allows.
3. Commit that token to the state.

```python pseudocode
state = constraint.start()
end_token_ids = configured_end_token_ids

while generating:
    in parallel:
        logits = llm.forward(...)
        mask = state.mask()

    logits = apply_mask(logits, mask)
    token_id = sample(logits)
    state.commit_token(token_id)
    if token_id in end_token_ids:
        break
```

Optional `@mark(id)` annotations report numeric marker IDs and byte positions
during generation. Read `state.marker_events()` immediately after `start()` and
each commit. These are possible boundary observations, not captures or a selected
parse. See the [marker reference](https://github.com/IsaacBreen/glrmask/blob/main/docs/markers.md).

## Quickstart

```bash
python -m pip install glrmask llama-cpp-python torch
```

```python
import numpy as np
from llama_cpp import Llama
from torch import from_numpy
from torch.distributions import Categorical

import glrmask

llm = Llama(model_path="model.gguf", logits_all=True)
vocab = glrmask.Vocab.from_llama_cpp(llm)
end_token_ids = vocab.llama_cpp_end_token_ids

schema = {"type": "string", "enum": ["positive", "negative", "neutral"]}
constraint = glrmask.Grammar.from_json_schema(schema).compile(
    vocab,
    end_tokens=end_token_ids,
    optimization=glrmask.Optimization.AUTO,
)

prompt = "Classify this review: The story dragged badly. Sentiment: "
input_tokens = llm.tokenize(prompt.encode())
llm.reset()
llm.eval(input_tokens)

state = constraint.start()
generated = []

for _ in range(64):
    logits = llm.scores[llm.n_tokens - 1]
    mask = state.mask(llm.n_vocab())
    logits[~mask] = -np.inf
    token_id = Categorical(logits=from_numpy(logits)).sample().item()

    llm.eval([token_id])
    generated.append(token_id)
    state.commit_token(token_id)
    if token_id in end_token_ids:
        break

print(llm.detokenize(generated).decode())
```

`state.mask()` returns a NumPy Boolean array indexed by model token ID. Pass a size when the model's logits vector is wider than the constraint's natural token coordinate.

## How it works

GLRMask maintains a GLR parser state for the generated prefix. A precomputed deterministic weighted automaton computes the next-token mask by reading each parser stack, intersecting Boolean vocabulary masks along each traversal and unioning alternative paths.

See [How it works](https://glrmask.com/implementation/) for the explanation and diagrams.

## License

Licensed under either the MIT License or the Apache License, Version 2.0, at your option.

