Metadata-Version: 2.4
Name: narwhal-inference
Version: 0.2.0
Summary: Adaptive hot-swap disaggregation for LLM inference
Author: Athrael Soju
License-Expression: Apache-2.0
Project-URL: Repository, https://github.com/athrael-soju/Narwhal
Project-URL: Issues, https://github.com/athrael-soju/Narwhal/issues
Keywords: llm,inference,disaggregation,prefill,decode,scheduling,vllm,kv-cache
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Distributed Computing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: fastapi>=0.110
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.7
Requires-Dist: uvicorn>=0.29
Provides-Extra: dev
Requires-Dist: pytest>=8.3; extra == "dev"
Requires-Dist: pytest-asyncio>=1.0; extra == "dev"
Requires-Dist: pytest-cov>=5; extra == "dev"
Requires-Dist: ruff==0.16.2; extra == "dev"
Requires-Dist: mypy==1.19.0; extra == "dev"
Requires-Dist: hypothesis==6.150.0; extra == "dev"
Requires-Dist: vulture>=2.14; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Requires-Dist: pre-commit>=4; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=5; extra == "dev"
Provides-Extra: wandb
Requires-Dist: wandb>=0.17; extra == "wandb"
Dynamic: license-file

<p align="center">
  <img src="assets/social-preview.png" alt="The Narwhal logo, a black narwhal with a teal spiral tusk above the wordmark" width="100%">
</p>

<p align="center">
  <img src="https://img.shields.io/badge/license-Apache--2.0-blue" alt="Apache-2.0 license">
  <img src="https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue" alt="Python 3.11 through 3.13">
  <img src="https://img.shields.io/badge/style-ruff-261230" alt="Lint and format by ruff">
  <img src="https://img.shields.io/badge/types-mypy-blue" alt="Types checked with mypy">
</p>

Narwhal is a router and controller for disaggregated LLM inference. It profiles each engine, routes prefill and decode as separate legs, and changes the fleet split while the model weights remain resident.

The scheduling core independently implements Algorithms 1 through 3 from the [Arrow paper](https://arxiv.org/abs/2505.11916). Narwhal adds a target-state planner, admission control, engine-health handling, warm-standby failover, journals, preflight checks, and evaluation tools.

## Engine requirements

Every engine must serve the same model and exchange compatible KV cache with every peer. The shipped vLLM adapter uses NIXL and expects effective `kv_both` behaviour.

Run `narwhal-check` against the concrete image and fabric before serving. If the fleet declares an engine contract, run `narwhal-attest` beside each engine so the check can tie that contract to the current process. The `produce` and `consume` gates then move KV through every tested direction.

Terminate TLS at ingress because Narwhal accepts anonymous requests by default and expects a trusted fabric. Run one active router per fleet.

## Try it without GPUs

The CPU simulator replays a moving 90-second workload across the fleet architectures.

```bash
git clone https://github.com/athrael-soju/Narwhal
cd Narwhal
make demo
```

The local engine stubs exercise the router protocol, profiling, preflight, KV handoff, and live state.

```bash
make setup
make stub-fleet
```

Continue in [Quickstart](docs/Quickstart.md) from another terminal.

## Deploy a fleet

The normal bring-up sequence is short.

```bash
.venv/bin/narwhal-profile --fleet config/fleet.local.json
.venv/bin/narwhal-check --fleet config/fleet.local.json
.venv/bin/narwhal-serve --fleet config/fleet.local.json --port 8000
```

[Deploy](docs/Deploy.md) covers config creation, SLO calibration, engine compatibility checks, failover, and fleet deployment. Shipped hardware and model recipes live under [`presets/`](presets/README.md).

## What the router records

| Surface | Contents |
| --- | --- |
| `/arrow/state` | Current pools, load, health state, counters, and role changes |
| `/metrics` | Prometheus counters, gauges, and TTFT and TPOT histograms |
| Request journal | Per-request lengths, timing, placement, handoff, and verdict |
| State handoff | Roles, ejections, offline windows, and counters for restart or standby |
| Profile store | Measured prefill and decode curves for every engine |

Prompt and response text stays out of the standard journal. Bounded payload capture is available as an explicit sidecar.

## Commands

Installing `narwhal-inference` provides these entry points.

| Command | Purpose |
| --- | --- |
| `narwhal-attest` | Tie one engine contract to the current vLLM process. |
| `narwhal-profile` | Measure per-engine prefill and decode curves. |
| `narwhal-check` | Run the deployment gates. |
| `narwhal-serve` | Start the router or a warm standby. |
| `narwhal-bench` | Sweep offered rate, replay a trace, or score one journal. |
| `narwhal-canary` | Run exact-output probes beside an evaluation workload. |
| `narwhal-report` | Score a comparison artifact directory. |
| `narwhal-live-bench` | Drive interactive or scripted traffic. |
| `narwhal-fleet` | Run commands and deploy the checkout over SSH. |

`make check` runs lint, format, tests, and the documentation link checker.

## Documentation

- [Quickstart](docs/Quickstart.md) runs the full local path without GPUs.
- [How Narwhal works](docs/How-Narwhal-Works.md) explains request routing, control, health, and failover.
- [Deploy](docs/Deploy.md) connects a real fleet.
- [Fleet architectures](docs/Architectures.md) compares aggregate, static, cold-swap, and hot-swap layouts.
- [Configuration](docs/Configuration.md) defines every field and default.
- [Command-line reference](docs/CLI.md) defines every option, default, and precedence rule.
- [HTTP API](docs/Api.md) defines the seven routes and status behaviour.
- [Journals](docs/Journals.md) defines request, event, and payload records.
- [Serving KPIs](docs/KPIs.md) defines latency, attainment, and goodput.
- [Monitor a fleet](docs/Monitor.md) connects Prometheus, Grafana, W&B, and the watchdog.
- [Observability](docs/Observability.md) defines metrics, resets, alerts, and dashboard panels.
- [Benchmarking](docs/Benchmarking.md) covers SLO calibration and controlled comparisons.
- [Evals](docs/Evals.md) covers scored-run discipline and artifacts.
- [Supported hardware and models](docs/Supported-Hardware-and-Models.md) records verified pairs and limits.

`docs/` is also the source for the [GitHub wiki](https://github.com/athrael-soju/Narwhal/wiki). Documentation changes receive the same review as code.

## Evidence

*The Price of Order in Disaggregated Inference* uses the packaged topology walk to compare fleet architectures and controllers. Its artifact contains the campaign drivers, methodology, configs, and raw journals ([Zenodo](https://doi.org/10.5281/zenodo.22083692)). A preprint link will be added on publication.

The two seeded walks sent 203,313 requests and 2.86 billion tokens through one six-node fleet. Their journals contain no crash, restart, ejection, or panic event in any arm.

[Benchmarking](docs/Benchmarking.md) describes how to run the same comparison on another fleet.

## Project status

Narwhal is an independent implementation and has no affiliation with the Arrow authors. The public interfaces may change during the 0.x series.

See [CONTRIBUTING.md](CONTRIBUTING.md) for change requirements and [SECURITY.md](SECURITY.md) for private vulnerability reporting. Bugs and questions belong in [GitHub issues](https://github.com/athrael-soju/Narwhal/issues).

## Citation

Cite this software for Narwhal, the study for its measured results, and Arrow when referring to the scheduling algorithms. GitHub's **Cite this repository** button reads [CITATION.cff](CITATION.cff).

```bibtex
@software{soju2026narwhal,
  author  = {Athrael Soju},
  title   = {narwhal},
  version = {0.2.0},
  year    = {2026},
  url     = {https://github.com/athrael-soju/Narwhal/releases/tag/v0.2.0},
}

@misc{georgiou2026priceanarchydisaggregatedinference,
      title={The Price of Anarchy in Disaggregated Inference},
      author={Athos Georgiou},
      year={2026},
      eprint={2606.17081},
      archivePrefix={arXiv},
      primaryClass={cs.AR},
      url={https://arxiv.org/abs/2606.17081},
}

@unpublished{georgiou2026priceoforder,
  author = {Georgiou, Athos},
  title  = {The Price of Order in Disaggregated Inference},
  year   = {2026},
  note   = {In preparation},
}

@misc{wu2025arrow,
  author        = {Wu, Yu and Liu, Tongxuan and Zeng, Yuting and Wu, Siyu and Xiong, Jun and Dong, Xianzhe and Yang, Hailong and Zhang, Ke and Li, Jing},
  title         = {Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture},
  year          = {2025},
  eprint        = {2505.11916},
  archivePrefix = {arXiv},
  primaryClass  = {cs.DC},
  url           = {https://arxiv.org/abs/2505.11916},
}
```
