How to read this page
Every time a coding agent (Claude Code, OpenCode, …) needs the model it sends the whole conversation as one prompt: the system prompt, every tool definition, and all messages so far.
The server keeps the last prompt it processed. If the new prompt starts with exactly the same tokens, it reuses that work and only computes ("prefills") the new tail. Time to first token is almost entirely prefill time, so the question for every request is: how much was new, and why?
The request list answers it at a glance (grey = reused, orange = computed fresh), grouped by conversation when several agents or sessions share the server. Click a request: the panel on the right explains it in words, What's in the prompt shows what the tokens are, Why it cost this shows where the prompt first differs from the previous one, and Tokens shows the exact tokens the model read.
One rule to remember: reuse works best when the new prompt extends the previous one. Normal turns and tool results do that. Some architectures (hybrid/recurrent models: DeltaNet or Mamba hybrids) reuse only when the new prompt extends the previous one; plain-attention models reuse everything up to the first change. Anything that changes an earlier message costs a re-prefill from there (or from the last checkpoint before it).
Glossary
- token
- the unit the model reads; ~4 characters of English, fewer for code and JSON.
- prefill
- processing prompt tokens before the first output token.
- reused / cached
- prompt tokens the server already had processed for an earlier request and did not compute again (Anthropic usage: cache_read_input_tokens; OpenAI usage: prompt_tokens_details.cached_tokens).
- TTFT
- time to first token: from sending the request until the first piece of the answer arrived. Includes prefill and any waiting for the GPU.
- decode
- generating the answer, token by token.
- slot
- one conversation's cached state on the server. With more than one, a small side request need not evict the main conversation.
- LCP / LRU
- how the server picked a slot: LCP = it found a slot whose cached prompt shares a prefix with the new one (the fraction kept is f_keep); LRU = nothing matched, it took the least recently used slot and started from scratch.
- end-extension
- a prompt that is the previous prompt plus new tokens at the end. The cheap case.
- tool round-trip
- the client sending a tool's result back to the model; an end-extension whose cost is the size of the result.
- template overhead
- tokens added by the chat template (markup around tool schemas, turn markers) on top of the raw text the client sent.
Overview
Timeline
When each request ran, one lane per server slot. Bar length is wall time; the darker start of a bar is the wait for the first token (mostly prefill). Bars that overlap were sharing the GPU, which slows both. Click a bar to select it.
Requests
One row per request that reached the tap. When several conversations (sessions of Claude Code, OpenCode, or other clients) share the server, rows are grouped by conversation; click a group row to fold it, or untick "group by conversation" for a single timeline. Anything sent by a script rather than an agent (curl, a benchmark script) is marked "via …". The bar shows the prompt: grey is what the server reused, orange is what it computed fresh (this is what you wait for). Hover a column title or a kind badge for its meaning; j/k move the selection.
| # | time | kind | promptreused | fresh | prompt | reused | fresh | TTFT | prefill | decode | cache | first change | check |
|---|