Metadata-Version: 2.4
Name: innards
Version: 0.3.0
Summary: See inside open-weight models while they run: read-only hooks on the model you loaded, one analyzer per architectural feature (Gated DeltaNet, LatentMoE, mHC, CSA/HCA, MLA), KV and memory predicted vs measured, and a live web view on the same machine.
Keywords: llm,kv-cache,transformers,attention,gqa,mla,deltanet,mixture-of-experts,observability,profiling,inference
Author: Jai
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Dist: httpx>=0.28.1
Requires-Dist: huggingface-hub>=1.5
Requires-Dist: innards[hf,view,aws,docs] ; extra == 'all'
Requires-Dist: boto3>=1.34 ; extra == 'aws'
Requires-Dist: pypdf>=5.0 ; extra == 'docs'
Requires-Dist: psutil>=5.9 ; extra == 'hf'
Requires-Dist: torch>=2.4 ; extra == 'hf'
Requires-Dist: transformers>=5.0 ; extra == 'hf'
Requires-Dist: fastapi>=0.115 ; extra == 'view'
Requires-Dist: uvicorn>=0.30 ; extra == 'view'
Requires-Python: >=3.10
Project-URL: Homepage, https://pypi.org/project/innards/
Project-URL: Documentation, https://pypi.org/project/innards/#description
Provides-Extra: all
Provides-Extra: aws
Provides-Extra: docs
Provides-Extra: hf
Provides-Extra: view
Description-Content-Type: text/markdown

# Innards

![Innards: one input, every architecture, zero slowdown](https://<public-host>/innards/infographic.png)

Innards shows what happens inside an open-weight model while it runs. Your code loads the original
checkpoint and runs the conversation; Innards attaches read-only hooks to the model you loaded, treats
each `generate()` call as one turn, and analyzes it on the same machine, beside generation.

- **Every model:** tokens, context, KV cache per layer, memory and timing, with the KV predicted from
  the config checked against the cache the engine really holds.
- **Each architectural feature at work:** one analyzer per feature (Gated DeltaNet, gated attention,
  MoE and LatentMoE, mHC, CSA/HCA, DSA, MLA), picked automatically from the model's config.
- **A live web view** served from the machine that runs the model: your laptop, or an AWS GPU instance
  through an SSH tunnel. There is no central site, no exporter and no API key.

Innards never loads or runs a model itself. [innards-exp](https://github.com/jaiganesan-n/innards_exp) is
the runner that loads the lineup's original checkpoints and drives conversation sets through the view.

## Install

<!-- test: install (checked by the fresh-environment wheel install in CI) -->
```bash
pip install innards                  # core: observe, analyzers, KV calculator, lineage, run store
pip install "innards[hf]"            # + torch, transformers, psutil
pip install "innards[view]"          # + FastAPI and uvicorn for the web view
pip install "innards[aws]"           # + boto3, to sync run folders to S3
pip install "innards[docs]"          # + pypdf, for PDF documents in the Input docs set
pip install "innards[all]"

uv add "innards[hf,view]"            # inside a uv project
```

The core install has two small dependencies (`httpx`, `huggingface-hub`) and works on Python 3.10+.

## Quickstart

<!-- test: requires=hf,network -->
```python
import innards
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-0.8B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B")  # your code loads the original checkpoint

watch = innards.observe(model, tokenizer, model_id="Qwen/Qwen3.5-0.8B", run="deltanet-study")

messages = [{"role": "user", "content": "Remember this: my name is Priya."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(**inputs, max_new_tokens=32)  # your code, unchanged: one turn

turn = watch.last_turn().record
print(turn.kv.measured_bytes == turn.kv.predicted_bytes)  # True
print(sorted(turn.features))  # ['gated_attention', 'gated_deltanet', 'kv']
watch.close()
```

`watch.view()` serves the web view from the same process (in a background thread, so a notebook keeps
working) and returns its address, for example `http://127.0.0.1:8000/`.

## What each turn records

The **light tier** is always on and costs metadata only: tokens and prefix reuse, the context, KV bytes
per layer and per layer type (measured from tensor shapes and dtypes, never values), memory against the
GPU budget (10 GB by default), TTFT, prefill and decode speed, and Innards' own overhead.

**Feature probes** attach only to the features the config says the model has. Each probe reports its
own measured overhead, supports sampling (`sample_every=N` tokens, or `layers=[...]`), and copies what
it reads to pinned CPU memory without blocking; a worker thread analyzes it.

| Analyzer | Reads | Shows |
| --- | --- | --- |
| `kv` (every model) | Cache object after each turn | KV per layer and layer type, growth, context, predicted vs measured, with gaps explained |
| `gated_deltanet` | β and decay per token and layer | Write strength, retention, half-life, fixed state next to growing KV |
| `gated_attention` | Output gate | Gate openness per head and token |
| `moe` (MoE and LatentMoE) | Router outputs, latent projection | Expert load, routing entropy, latent size vs hidden size |
| `mhc` | H_pre, H_post, H_res per layer | Stream mixing, doubly-stochastic check (row/column sums), signal gain across depth |
| `csa_hca` | Cache layout | Compressed vs window entries, indexer cache, tokens waiting to be compressed |
| `dsa` | Indexer keys | Indexer cache, share of the context each query attends to |
| `mla` | Cache layout | Latent vs expanded layout |

Features Innards recognizes but does not analyze yet (Mamba-2 layers, short convolutions) are monitored
at the light tier and marked "not analyzed yet"; multi-token prediction heads are marked "not run by
transformers". When a new architecture comes out, it gets a new analyzer (`innards.features.register`).

### Calling an analyzer yourself

`observe` runs every matching analyzer automatically. An analyzer can also be called on a captured turn;
`args` is what `observe` captured, not the output of `generate()`, which holds no hook captures:

<!-- test: requires=hf,network -->
```python
import innards
from innards import GatedDeltaNet
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-0.8B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B")
watch = innards.observe(model, tokenizer, store=False)
model.generate(**tokenizer("The keeper lit the lamp", return_tensors="pt"), max_new_tokens=8)

payload = GatedDeltaNet(model="Qwen/Qwen3.5-0.8B", args=watch.last_turn())
print(payload["findings"][0])  # Write strength β averaged ...
watch.close()
```

The same form works for every analyzer: `mHC(model="deepseek-ai/DeepSeek-V4-Flash", args=watch.last_turn())`.

## The web view

A light monitoring dashboard with two tabs and a sidebar.

- **Header:** model picker (the lineup, any Hub id or local folder), precision, Check and Load, a live
  indicator ("● Live · turn 7") and a memory gauge against the budget.
- **Sidebar:** what is monitored now (KV cache, context, memory, timing), the model's features
  (analyzed, not in this model, or not analyzed yet), and health: each probe's overhead, dropped events
  and errors.
- **Conversation:** run a conversation set turn by turn, or type your own messages; every reply shows
  tokens reused and prefilled, the KV change and speed.
- **Analysis:** stat tiles with sparklines, what changed from the version before, each feature at work,
  KV/context/memory per turn predicted vs measured with a per-layer heatmap, and the same set side by side
  with the version before. Hovering turn 5 highlights turn 5 in every chart.

Load, Check and Send need a runner (`innards-exp serve`), because Innards never loads a model. Without one,
the view monitors whatever your code generates, and opens any run folder:

<!-- test: text -->
```text
innards view                                   # ./innards_runs on http://127.0.0.1:8000
innards view ./innards_runs/deltanet-study     # one run folder, e.g. copied from another machine
ssh -L 8000:127.0.0.1:8000 ubuntu@<instance>   # on AWS: open the instance's view through a tunnel
innards sync up deltanet-study s3://bucket/innards      # before the instance stops
innards sync down s3://bucket/innards/deltanet-study    # on your laptop, then `innards view`
```

## Where results go

Results stay on the machine that runs the model, next to the run:

<!-- test: text -->
```text
innards_runs/<run>/run.json                          name, host, environment
innards_runs/<run>/<conversation>/conversation.json  model, config, features, conversation set, status
innards_runs/<run>/<conversation>/turns.jsonl        one TurnRecord (schema 2.0) per turn
```

Schema 2.0 records a run, a conversation and a turn, with one payload per analyzed feature under
`features`. Missing measurements are `null`, never zero. Readers accept 2.x and upgrade 1.x records.

<!-- test: check-output -->
```python
from innards.schema import TurnRecord

record = TurnRecord.from_dict(
    {
        "schema_version": "2.0",
        "run": "deltanet-study",
        "conversation": "20260929-081500-Qwen3.5-0.8B-a1b2",
        "turn": 7,
        "timestamp": "2026-09-29T08:15:42Z",
        "model": {"id": "Qwen/Qwen3.5-0.8B", "architecture": "hybrid", "features": ["gated_deltanet"]},
        "runtime": {"backend": "observe"},
        "kv": {"predicted_bytes": 2154496, "measured_bytes": 2154496, "measured_source": "hf_cache"},
    }
)
print(record.kv.measured_bytes, record.memory.peak_bytes)  # 2154496 None
```

## Checks before loading

Every check runs before anything is downloaded, and says what is wrong, the numbers and the fix:
architecture support in the installed transformers (never `trust_remote_code`), whether this device can
run the checkpoint's published precision (FP8 and FP4 need CUDA), whether weights (from safetensors
metadata) plus the KV for the chosen conversation set plus headroom fit the budget, gated-repo access,
and which features have analyzers. The runner checks memory again after loading and unloads cleanly on
out-of-memory.

<!-- test: check-output requires=hf -->
```python
from innards.validate import validate

report = validate("deepseek-ai/DeepSeek-V4-Flash", device="mps", check_access=False)
print(report.status)  # Blocked
print([check.id for check in report.blocked])  # ['precision', 'memory']
```

## Conversation sets

| Set | Contents | What it shows |
| --- | --- | --- |
| `basic_qa` | Short, independent questions | Baseline KV per token and speed |
| `with_context` | A chat that plants facts early and asks about them later | KV growth across turns, recall, prefix reuse |
| `input_docs` | Your own .txt, .md or .pdf, then questions about it | Large prefill, jumps in context and KV |
| `long_document` | One long built-in document, then questions about it | Capacity; sliding, global and recurrent layers |

Running the same set on both models of a pair makes the Analysis tab compare like with like.

## The lineup

Three pairs of a newer model and the version before it, original checkpoints only:

| Feature | Newer model | Version before | Where |
| --- | --- | --- | --- |
| Gated DeltaNet + gated attention | Qwen3.5-0.8B (or 2B) | Qwen3-0.6B (or 1.7B) | Local |
| LatentMoE | Nemotron 3 Super 120B-A12B | Nemotron 3 Nano 30B-A3B | AWS |
| mHC (+ CSA/HCA attention) | DeepSeek-V4-Flash | DeepSeek-V3.2 | AWS |

Pinned configs ship with Innards, so each pair's architecture, what changed and its KV predicted from the
config are available before either model runs (labeled as predicted):

<!-- test: check-output -->
```python
from innards.kv import predict
from innards.lineage.lineup import model_config

p = predict(model_config("deepseek-ai/DeepSeek-V4-Flash"), context_tokens=131072, kv_dtype="bf16")
print(p.architecture, round(p.total_bytes / 2**20))  # compressed 865
```

## KV calculator and lineage

`predict` gives architectural KV bytes, capacity and a per-layer breakdown from a config alone:

<!-- test: check-output -->
```python
from innards.kv import predict

gemma_like = {
    "model_type": "gemma3_text",
    "num_hidden_layers": 26,
    "num_attention_heads": 4,
    "num_key_value_heads": 1,
    "head_dim": 256,
    "sliding_window": 512,
    "sliding_window_pattern": 6,
    "max_position_embeddings": 32768,
}
p = predict(gemma_like, context_tokens=8192, kv_dtype="bf16")
print(p.by_layer_type)  # {'global': 33554432, 'sliding': 11511808, 'recurrent': 0}
```

`explain` says what an architecture changed from its predecessor, the problem it solved and the trade-off:

<!-- test: check-output -->
```python
from innards.lineage import chain, explain

print(explain("gqa").predecessor, "->", explain("gqa").family)  # mha -> gqa
print([entry.family for entry in chain("compressed")])  # ['compressed', 'dsa', 'mla', 'gqa', 'mha']
```

## CLI

<!-- test: cli -->
```bash
innards lineup                                               # the pairs and what each newer model adds
innards predict --model Qwen/Qwen3-0.6B --context 32768 --kv-dtype fp8
innards explain sliding_window
innards validate Qwen/Qwen3.5-0.8B --device cpu --set with_context
```

`innards view [folder]` serves the web view and `innards sync up|down` copies run folders to and from
S3 (see The web view).

## Zero slowdown

- **Hot path:** a `generate` wrapper and one forward pre-hook take timestamps, count steps and offer a
  small event to a bounded queue that never blocks (full means dropped and counted). No network calls,
  disk writes or device synchronization happen during generation.
- **Feature probes** reduce small tensors on the device and copy them to pinned CPU memory without
  blocking; only the worker waits for the copies. Every hook is timed, so each probe's overhead is
  measured and shown, and every hook is fail-open: an error is counted, never raised into the model.
- **Worker threads** do the rest: KV math and measurement, memory, analysis, writing the run folder.
- **Same output:** greedy replies are identical with and without the hooks (tested on the Qwen pair and
  on replicas of every analyzer's architecture).

**Budget:** throughput within 1% with the light tier, measured by `scripts/overhead_bench.py`
(same model and prompts, observed vs plain `generate`, interleaved runs, medians).

## Supported

**Architectures** (KV calculator and measurements): MHA, MQA, GQA, MLA (DeepSeek-V2/V3), DSA
(DeepSeek-V3.2, with its indexer keys), compressed attention (DeepSeek-V4 CSA/HCA), sliding window and
chunked layers (Gemma 2/3, GPT-OSS, OLMo 3, Llama 4 ...), hybrids (Qwen3.5/3.6 and Qwen3-Next Gated
DeltaNet, Nemotron-H and Nemotron 3, Granite 4.0-H, LFM2, Jamba, Bamba, Falcon-H1) and SSMs. Recurrent
state is sized in float32 where the engine keeps it so.

**Models:** open-weight only, as the vendor published them (FP4/FP8 checkpoints such as DeepSeek-V4
count as original). Community quantizations, distillations and conversions are out of scope; tiny
random-weight replicas appear only in the test suite.

**Engine:** Hugging Face Transformers 5.x in-process (feature probes need forward hooks), on CUDA,
Apple MPS or CPU. **Python:** 3.10 to 3.15.

## Deprecated in 0.3

Innards 0.2 loaded models itself. That API keeps working for one release with a `DeprecationWarning`,
and goes in 0.4: `Session` and `AsyncSession` (load the model yourself and call `innards.observe`),
`compare` (run the same conversation set on each model; the view compares them) and `innards chat`
(use `innards-exp serve`). The `int8` extra and `quantize="int8"` are removed: Innards measures original
checkpoints at their published precision. `innards.observe` is now the function above; the observer
internals moved to `innards.observer`.

## License

Apache-2.0
