Metadata-Version: 2.5
Name: workingset
Version: 0.2.0
Summary: How many agentic-coding users a local LLM deployment serves: capacity model, CLI and live-endpoint hypothesis tests
Project-URL: Homepage, https://workingset.tomvaucourt.com
Project-URL: Repository, https://github.com/T0mSIlver/working-set
Author: Tom Vaucourt
License-Expression: MIT
License-File: LICENSE
Keywords: benchmark,capacity-planning,kv-cache,llm,serving,vllm
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: numpy>=2.0
Provides-Extra: study
Requires-Dist: matplotlib>=3.9; extra == 'study'
Requires-Dist: pandas>=2.0; extra == 'study'
Requires-Dist: scipy>=1.13; extra == 'study'
Description-Content-Type: text/markdown

# Working Set

Tools for scaling **local LLM deployments**: how many concurrent users or
agents a given GPU configuration can keep warm, where KV cache, decode
bandwidth, and prefill compute each become the binding constraint, and which
knob (topology, dtypes, `max_num_seqs`, prompt caching) buys the most headroom
for agentic coding workloads.

**Start with the [interactive explorer](https://workingset.tomvaucourt.com/)** —
live sliders for the workload, model (Qwen3.8-27B / 35B-A3B /
Mistral-Medium-3.5 / GLM-5.3 / DeepSeek-V4-Flash / DeepSeek-V4.1-Flash /
Qwen3.8-Flash-Next /
GLM-5.3-Flash), GPU (H200 / B300), weight & KV dtypes, and DP × TP
topology. It answers as a decision tool: a binding-constraint verdict, a
deploy recipe (vLLM flags), the **bill** (€/GPU-hour and €/kWh sliders:
hardware plus a duty-cycle power model), a **sensitivity panel** showing which assumption
would flip the decision, the **steady-state decode point** (how many
sessions are actually decoding at your load, and how fast each one runs —
Little's law, not the all-warm stress test), **shareable links** that
encode the whole configuration, and a **download button** in the Test step
that hands out the configuration on screen as a `workingset.toml` — feed it to
`ws test` below and measure the real limits on a live vLLM endpoint.

## The `workingset` package

The model behind the explorer is a Python package (`src/workingset/`, the
source of truth; the explorer's JS mirrors it), published on PyPI as
[`workingset`](https://pypi.org/project/workingset/). It ships one console
script, `ws`, so the package name travels in `--from`:

```bash
uvx --from workingset ws init --model Q38FN --gpu B300 --tp 8 --weight-dtype nvfp4   # writes workingset.toml
uvx --from workingset ws predict workingset.toml        # the four ceilings, which one binds, the operating point
uvx --from workingset ws test workingset.toml --dry-run # the plan, the sampler self-check, no requests
uvx --from workingset ws test workingset.toml --all --exclusive --out run.json   # measure it
```

or, as a dependency, `pip install workingset` / `uv add workingset`. From a
checkout the same commands run under `uv run`:

```bash
uv run ws init --model Q38FN --gpu B300 --tp 8 --weight-dtype nvfp4   # writes workingset.toml
uv run ws predict workingset.toml        # the four ceilings, which one binds, the operating point
uv run ws predict workingset.toml --json # the same as a run record
uv run ws link workingset.toml           # the explorer URL showing this config
uv run ws hypotheses                     # the H-* and what each one needs
uv run ws test workingset.toml --dry-run # the plan, the sampler self-check, no requests
uv run ws test workingset.toml --exclusive --out run.json   # measure it
uv run ws report run.json                # re-print the verdicts
uv run ws models                         # model / GPU keys
uv run pytest                            # self-checks + config round-trips
```

The explorer's `workingset.toml` needs no checkout at all: the three `uvx`
lines above are exactly what its Test step prints. To run
the model at a commit PyPI does not have yet, point `--from` at git:
`uvx --from git+https://github.com/T0mSIlver/working-set ws …`.

Predictions live in no file: `ws predict` recomputes them from the config every
time, so a config can never carry a number the code did not produce. A harness
`.py` downloaded from the explorer before the package existed still loads (its
CONFIG block is extracted).

`ws link` goes the other way: it prints the explorer share URL that opens the
page on a config (`--base http://127.0.0.1:PORT/` for a local copy of
`interactive/`). The page has no control for `subagent_prefix_tokens` or
`[endpoint]`, and
it clamps each slider to its range and resets combinations it does not price,
so when a config sets one of those, or a value the page would move, `ws link`
names the field on stderr with what the page shows instead.

`ws test` puts the predictions to a live endpoint, one falsifiable hypothesis
at a time. Without `--exclusive` it runs only the hypotheses that need a
handful of requests (miss TTFT, the inter-token gap distribution, the steady
decode point) and lists the rest as skipped — a hypothesis that has to
generate its own population is never measured against someone else's load.
With `--exclusive` it drives the geometric load ladder once, and every ceiling
reads from it. `--burst N` adds the correlated-flush probe (B*).
`--tokenizer Qwen/Qwen3.8-27B` sizes the synthetic prompts with the model's
real tokenizer (via [toklen](https://pypi.org/project/toklen/); add
`--with toklen` to the `uvx` line) instead of the chars-per-token guess.
For a vision model, set `image_share` in `[workload]` (with
`images_per_request`, `image_width` and `image_height`) and pass `--api chat`:
that share of warm turns then attaches generated PNGs, which stay in the
session history, and the report splits warm-turn TTFT by whether the turn
sent a new image. The model does not price images yet.

## Contents

- [docs/writeup.md](docs/writeup.md) — baseline study: KV-cache capacity and
  the prompt-caching / offload / `max_num_seqs` trade-offs.
- [docs/scenarios.md](docs/scenarios.md) — extended scenario model: multi-GPU
  topologies, MoE vs dense, subagent workloads, the cost of a cache miss, and
  cold-spike tolerance.
- [scripts/](scripts/) — everything is reproducible:

  ```bash
  uv run ws selfcheck                # the shared model's self-checks (src/workingset/model.py)
  uv run scripts/scenarios.py        # renders the scenario figures
  uv run scripts/tables.py           # regenerates every number in docs/scenarios.md
  ```

- [research/](research/) — sourced constants for each model and GPU.
- [interactive/](interactive/) — the explorer, a dependency-free page
  mirroring the Python model: `index.html` holds the markup and styles,
  `src/*.js` the model and the charts as ES modules (`src/main.js` is the
  entry and lists the layering). Browsers refuse module scripts from
  `file://`, so serve the folder to open it locally:

  ```sh
  python3 -m http.server 8000 --directory interactive   # then http://localhost:8000
  ```

Method, calibration, and caveats are laid out in the docs above.

MIT licensed; see [LICENSE](LICENSE).
