Metadata-Version: 2.5
Name: layerscale
Version: 0.9.0
Summary: Python client for the LayerScale inference server
Project-URL: Homepage, https://layerscale.ai
Author: LayerScale
License: MIT
Keywords: inference,layerscale,llm,sessions,streaming
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: anyio<5,>=4.15.1
Requires-Dist: httpx<1,>=0.28.1
Requires-Dist: pydantic<3,>=2.14.1
Requires-Dist: typing-extensions<5,>=4.16.0
Requires-Dist: websockets<18,>=17.3
Provides-Extra: dev
Requires-Dist: pyright>=1.1.414; extra == 'dev'
Requires-Dist: pytest-asyncio>=1.4.1; extra == 'dev'
Requires-Dist: pytest>=9.1.1; extra == 'dev'
Requires-Dist: python-dotenv>=1.2.4; extra == 'dev'
Requires-Dist: respx>=0.23.1; extra == 'dev'
Requires-Dist: ruff>=0.17.3; extra == 'dev'
Description-Content-Type: text/markdown

# layerscale

Python client for the LayerScale inference server: OpenAI's Chat Completions
and Responses APIs, Anthropic's Messages API, and LayerScale sessions. Sync
(`LayerScale`) and async (`AsyncLayerScale`) clients with identical surfaces,
typed requests (TypedDicts) and responses (pydantic v2), both server SSE
dialects, and a WebSocket session socket.

Requires Python 3.11+. Runtime dependencies: `httpx`, `pydantic` v2,
`typing-extensions`, `anyio`, `websockets` (imported lazily, only if you open
a WebSocket).

```bash
pip install layerscale
```

### Server version

0.9.0 talks to a LayerScale server from 2026-10-26 (`bef36ae2`) on.
`client.responses`, the `developer` role on chat completions, and the Claude
Code fields on messages (system messages, `tool_reference` results, deferred
tools) need a server that serves `/v1/responses`: server main from
2026-10-27 (`f601acd4`) on.
What changed is in [CHANGELOG.md](CHANGELOG.md).

## Quickstart

```python
from layerscale import LayerScale

client = LayerScale()  # defaults to http://127.0.0.1:8080

resp = client.chat.create(messages=[{"role": "user", "content": "Hello!"}])
print(resp.choices[0].message.content)
```

```python
from layerscale import AsyncLayerScale

async with AsyncLayerScale() as client:
    resp = await client.chat.create(messages=[{"role": "user", "content": "Hello!"}])
```

The base URL falls back to the `LAYERSCALE_BASE_URL` environment variable,
then to `http://127.0.0.1:8080` (the server's default bind).

### Authentication

By default the server's API is open and needs no key. Started with
`--api-key`, its Ed25519 license key (format `LSK-...`) is the API bearer
token: every route except the CORS
preflight and `/health`/`/healthz` (this includes `/metrics`) requires
`Authorization: Bearer <LSK-...>` (the server also accepts `x-api-key`) and
answers 401 `authentication_error` without it; the WebSocket upgrade is
refused with a plain-HTTP 401 before the handshake. Pass the key as the
`api_key` constructor option, or let it fall back to the `LAYERSCALE_API_KEY`
env var, then `LAYERSCALE_LICENSE_KEY` (the server's own canonical variable
for the key). The client sends it as `Authorization: Bearer <key>` on every
request, including the WebSocket handshake; see
[.env.example](.env.example).

### Evaluation mode

A server started without a license key runs in evaluation mode, with the free
key's limits: one live session (a second `sessions.create()` raises
`PermissionDeniedError`, 403 `permission_error`, `the free license permits at
most 1 live session(s); delete one first`), a context of at most 32,768 tokens
(`models.list()` reports it as `max_model_len`) and one GPU. It cannot be
started with `--api-key`, so it never asks for a key and ignores one sent.
Nothing in a response marks the mode; the server logs its tier at startup.
Every call in this client works against it, sessions within that cap. A server
in benchmark mode (started with a forced draft acceptance) has no caps, but
its replies are not the model's: only the speculative decoding statistics show
it, with `mean_acceptance_length` at the forced value.

## Endpoints

| Client call | Endpoint |
|---|---|
| `client.chat.create(...)` / `client.chat.stream(...)` | `POST /v1/chat/completions` |
| `client.responses.create(...)` / `client.responses.stream(...)` | `POST /v1/responses` (see [Responses](#responses-openai-responses-api) for the by-id routes) |
| `client.messages.create(...)` / `client.messages.stream(...)` | `POST /v1/messages` |
| `client.messages.count_tokens(...)` | `POST /v1/messages/count_tokens` |
| `client.models.list()` / `client.models.retrieve(id)` | `GET /v1/models`, `GET /v1/models/{id}` |
| `client.health.check()` | `GET /health` |
| `client.metrics.get()` | `GET /metrics` (Prometheus text) |
| `client.sessions.create/list/get/delete` | `POST/GET /v1/sessions`, `GET/DELETE /v1/sessions/{id}` |
| `client.sessions.push(id, data)` | `POST /v1/sessions/{id}/push` |
| `client.sessions.warm(id, ...)` | `POST /v1/sessions/{id}/warm` |
| `client.sessions.generate(id, ...)` / `generate_stream(id, ...)` | `POST /v1/sessions/{id}/generate` |
| `client.sessions.flash/list_flash/unflash` | `POST/GET /v1/sessions/{id}/flash`, `DELETE .../flash/{fid}` |
| `client.sessions.events(id)` | `GET /v1/sessions/{id}/events` (SSE) |
| `client.sessions.ws(id)` | `GET /v1/sessions/{id}/ws` (WebSocket) |

### Sampling defaults (all generation endpoints)

| Parameter | Default |
|---|---|
| `temperature` | the checkpoint's `generation_config.json`, else 1.0 |
| `top_p` | the checkpoint's `generation_config.json`, else 1.0 |
| `top_k` | the checkpoint's `generation_config.json`, else no cap; an integer of at least 1, or `0` / `-1` for no cap |
| `max_tokens` | no cap (required on `/v1/messages`) |

Each is taken from the checkpoint on its own, so a request that sets only
`temperature` still gets the checkpoint's `top_p` and `top_k`. `top_k` is
accepted on chat completions, messages, session `generate` and responses. Unsupported
parameters (`n`, penalties, `logit_bias`, `min_p`, the reply-constraint fields
other servers define such as `guided_json`, audio output through `modalities`
or `audio`, ...) are refused by the server with a 400 naming the parameter;
the client never forwards fields you didn't pass.

`skip_special_tokens` (default `True`, on chat completions, messages and
session `generate`) keeps the vocabulary's special tokens (an end of turn, the
next turn's header, padding) out of the reply's text; `False` returns every
generated token's text. With `separate_reasoning=False`, reasoning markers the
vocabulary flags special (Gemma 4's thought channel, Mistral's `[THINK]`) come
back only under `skip_special_tokens=False`.

### Context window

`client.models.list()` reports `max_model_len`, the longest sequence (prompt
plus completion) the server admits once the model's context, the license,
`--ctx` and the KV pool have each capped it, and `max_input_tokens`, one
less, the longest prompt. Size your own window from it. A prompt that does
not fit is refused with 400 before anything streams, and the session (if
any) is free again and keeps what it held: `APIStatusError.context_overflow`
is true for that refusal on either surface (`code ==
"context_length_exceeded"` on the OpenAI routes, `prompt is too long: N
tokens > M maximum` on `/v1/messages`), so compact the conversation and
retry. A reply that fills the context ends with `finish_reason="length"`
(`stop_reason="model_context_window_exceeded"` on `/v1/messages`).

## Chat (OpenAI-compatible)

```python
resp = client.chat.create(
    messages=[
        {"role": "system", "content": "Be brief."},
        {"role": "user", "content": "What's the AAPL price?"},
    ],
    tools=[{"type": "function", "function": {
        "name": "get_stock_price",
        "parameters": {"type": "object", "properties": {"symbol": {"type": "string"}}},
    }}],
    tool_choice="auto",   # "auto", "none", "required", or {"type": "function", "function": {"name": ...}}
)
choice = resp.choices[0]
if choice.finish_reason == "tool_calls":
    for call in choice.message.tool_calls or []:
        print(call.function.name, call.function.arguments)  # arguments is a JSON string
```

Notes:

- `model` is optional and **ignored**, the response always carries the
  server's model name.
- Roles are `system`, `user`, `assistant` and `tool`, plus OpenAI's
  `developer` (read as `system`) on a server that serves `/v1/responses`.
- On prose replies the `tool_calls` key is absent; with tool calls, `content`
  is the text ahead of the first call, or `None` when there was none.
- `tool_choice="required"` or a named function forces a call, on every
  template that calls tools, and runs the turn without thinking unless the
  request asks for it. `parallel_tool_calls=False` limits the turn to one
  call.
- `reasoning_effort` (`"none"` … `"max"`) or
  `chat_template_kwargs={"enable_thinking": ...}` switch a reasoning model's
  thinking. `logprobs=True` (plus `top_logprobs`, 0–20) fills
  `choices[0].logprobs`; when streaming, it arrives on the final chunk.
- User messages may carry `{"type": "image_url", "image_url": {"url":
  "data:image/png;base64,..."}}` parts (PNG or JPEG) on a model with image
  input; only base64 `data:` URLs are accepted. Audio parts are described in
  [Audio input](#audio-input).
- A call cut by `max_tokens`, the context or a stop string is no call: the
  reply ends `length` or `stop`, never `tool_calls`. `completion_tokens`
  counts every generated token, the end of sequence the reply stopped on
  included.
- `usage.prompt_tokens_details.cached_tokens` is always present and counts
  every prompt token served from cache: the tokens a session turn resumed
  from held KV, plus the full pages the prefix cache supplied to any request
  (`prefix_cached_tokens`, present only when nonzero, is that part).
  `prompt_tokens - cached_tokens` is what the request computed.
- A reply that speculative decoding drafted for carries its acceptance
  figures twice: `choices[0].speculative_decoding_stats` (histogram as a
  dict, `"j"` to the verify steps that accepted `j` drafts; streamed on the
  chunk with `finish_reason`) and the root `metrics.speculative_decoding`
  (histogram as a list; streamed on the usage chunk, so it needs
  `stream_options={"include_usage": True}`). Neither is present when nothing
  drafted.
- LayerScale extensions: `session_id` (bind the turn to a session; sent as
  the `x-layerscale-session-id` header, not in the body), `ignore_eos`,
  `seed`, `top_k`, `skip_special_tokens`, `chat_template_kwargs`.

### Structured outputs

`response_format` holds the reply to JSON, enforced token by token:
`{"type": "json_object"}` for any object, or `{"type": "json_schema",
"json_schema": {"name", "schema", "strict"}}` for one value valid against
`schema` (enforced whatever `strict` says; no `schema` is any JSON value).
The JSON comes back as `content`, streamed as it is written, and the reply
ends `stop` the moment the value is complete (`length` leaves a valid
prefix).

```python
import json

resp = client.chat.create(
    messages=[{"role": "user", "content": "The largest city in France, as JSON."}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "city",
            "schema": {"type": "object", "properties": {"name": {"type": "string"}}, "required": ["name"]},
        },
    },
)
city = json.loads(resp.choices[0].message.content or "{}")
```

Reasoning stays free and the format holds from the answer on. With tools on
offer and `tool_choice="auto"` the reply is a call or the JSON; a forced
`tool_choice` is a call. The format is per turn, a session carries nothing of
it over. Unsupported schema keywords (`not`, `if`/`then`/`else`,
`patternProperties`, `multipleOf`, `uniqueItems: true`, ...) are refused by
name, and the Metal backend refuses `response_format`. On messages the same
holds through `output_config={"format": {"type": "json_schema", "schema":
...}}` (or the deprecated top-level `output_format`, not both). `/health`
counts constrained requests in `structured_requests`.

### Audio input

On a checkpoint with an audio tower (Gemma 4 E2B and E4B, on the CPU and CUDA
backends), a user message takes a clip as a content part:

```python
import base64
from pathlib import Path

data = base64.b64encode(Path("question.wav").read_bytes()).decode()
resp = client.chat.create(
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Answer the question in the clip."},
            {"type": "input_audio", "input_audio": {"data": data, "format": "wav"}},
        ],
    }],
)
```

`{"type": "audio_url", "audio_url": {"url": "data:audio/wav;base64,..."}}` is
the same clip. WAV and MP3 are read (the file's first bytes decide, `format`
is a hint); a clip is at most 30 seconds and a request at most 16 clips. Audio
in a session turn or a warm turn, audio blocks on messages, and audio output
(`modalities`, `audio`) are refused by name.

### Reasoning

A checkpoint that thinks before it answers returns its reasoning apart from
the answer, as `message.reasoning` (a string without the block's markers, or
`None` when the reply had none) and, when streaming, as `delta.reasoning`
chunks ahead of the content; `usage.completion_tokens_details.reasoning_tokens`
counts its tokens. Send it back on the assistant message as `reasoning` (the
older `reasoning_content` is accepted too) and the template renders it as the
reference does. Three options, each default true: `include_reasoning=False`
leaves the reasoning out of the reply (still generated and counted),
`separate_reasoning=False` leaves it in `content` as the model wrote it,
markers included, and `stream_reasoning=False` sends it as one delta once
complete. Checkpoints without a reasoning mode never carry the field.

### Chat streaming, SSE dialect A

Untyped `data:` frames terminated by `data: [DONE]` (consumed by the client).
Chat streams send **no heartbeats**, and no bytes at all during prefill;
the only stream with `: heartbeat` comments is the session events stream
(the parser drops SSE comments defensively everywhere):

```python
for chunk in client.chat.stream(
    messages=[{"role": "user", "content": "Hello!"}],
    stream_options={"include_usage": True},
):
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")
    elif chunk.usage is not None:
        print(f"\n{chunk.usage.total_tokens} tokens")  # final usage chunk, choices == []
```

Tool calls stream as exactly two chunks per call: an opener with
`id`/`type`/`name` and empty `arguments`, then the full arguments string
(never split). With `stream_options={"include_usage": True}` every chunk
carries `usage: null` until the final usage chunk (which has `choices: []`).

## Responses (OpenAI Responses API)

`client.responses` speaks OpenAI's Responses API: input and output are lists
of typed items, a stored response is continued by its id, and the stream
sends semantic events. It is the wire Codex and the OpenAI Agents SDK speak.
The types and the resource layout are named as in OpenAI's SDK, under
`layerscale.types.responses` (`Response`, `ResponseCreateParams`,
`ResponseInputParam`, `ResponseStreamEvent`, `FunctionToolParam`,
`responses.input_items.list`, `responses.input_tokens.count`), so code
written for it moves over with a base URL change. This package does not
depend on `openai`.

**Server requirement**: a LayerScale server that serves `/v1/responses`:
server main from 2026-10-27 (`f601acd4`) on; an older server
answers 404 `unknown endpoint` on these routes.

```python
res = client.responses.create(
    instructions="Be brief.",
    input="Why is the sky blue?",
    max_output_tokens=200,
)
print(res.output_text, res.status, res.usage)
```

`res.output_text` joins the message's `output_text` parts, as in OpenAI's
SDK. `output` holds a reasoning item when the reply reasoned (the reasoning
whole, in `content`), the assistant message, and one item per tool call.
`status` is `completed`, or `incomplete` (`incomplete_details.reason ==
"max_output_tokens"`) when the budget or the context cut the reply.

| Client call | Endpoint |
|---|---|
| `client.responses.create(...)` / `.stream(...)` | `POST /v1/responses` |
| `client.responses.retrieve(id, include=...)` | `GET /v1/responses/{id}` |
| `client.responses.delete(id)` | `DELETE /v1/responses/{id}` |
| `client.responses.input_items.list(id, after=, limit=, order=)` | `GET /v1/responses/{id}/input_items` |
| `client.responses.input_tokens.count(...)` | `POST /v1/responses/input_tokens` |

There is no `cancel` (the server runs no background responses and always
answers 400), no `compact` and no WebSocket mode (both refused by name; the
stream sends the same events over SSE).

Request params are `TypedDict`s with a `Param` suffix, as in OpenAI's SDK;
response objects are pydantic models. A response's `output` items can go
back into `input` as they are: the client writes each model as the JSON it
came from. An item, content part or event of a type this client does not
know comes through as `UnknownResponseItem` / `UnknownStreamEvent` with every
field it carried, so keep a fallback branch when you switch on `type`.

### Request fields

Used: `input` (a string, or items: messages with `input_text`,
`input_image` as a `data:` URL, `input_audio`; every call and output kind
below; `reasoning` items; `item_reference`), `instructions` (not carried over
from a previous response), `previous_response_id`, `store` (default `True`),
`max_output_tokens`, `temperature`, `top_p`, `tools`, `tool_choice` (`auto`,
`none`, `required`, a named function or custom tool, `shell`, `apply_patch`,
`allowed_tools`), `parallel_tool_calls`, `text.format` (`text`,
`json_object`, `json_schema` held as on chat completions), `reasoning.effort`
and `reasoning.summary`, `include` (`reasoning.encrypted_content`,
`message.output_text.logprobs`), `top_logprobs` and `metadata`. LayerScale's
own: `top_k` and `seed` in the body, and `session_id`, which the client sends
as the `x-layerscale-session-id` header. Typed so ported code type-checks,
and accepted without effect: `user`, `safety_identifier`, `prompt_cache_key`,
`prompt_cache_retention`, `prompt_cache_options`, `client_metadata`,
`service_tier`, `truncation` (`auto` runs as `disabled`),
`context_management`, `max_tool_calls`, `stream_options`, `text.verbosity`.
Refused by name, and not typed: `background=True`, `conversation`, `prompt`,
`personality`, `response_format` (use `text.format`), `input_file` parts.

### Tools

Every tool kind a client runs is offered to the model as a function in the
chat template's own tool format, and its call comes back as the kind's own
item, so a history renders again byte for byte and the prefix cache serves
it:

| Tool | Call item | Its output, sent back |
|---|---|---|
| `function` | `function_call` (`arguments` the JSON string the model wrote) | `function_call_output` (a string, or `input_text` / `input_image` parts) |
| `custom` (a grammar is described, not enforced) | `custom_tool_call` (`input`) | `custom_tool_call_output` |
| `namespace` (members as `<namespace>__<name>`) | the member's kind, with `namespace` | the member's |
| `shell` (local environment) | `shell_call` (`action.commands`) | `shell_call_output` |
| `apply_patch` | `apply_patch_call` (`operation` with a V4A `diff`) | `apply_patch_call_output` |
| `local_shell` | `local_shell_call` (`action.command`, an argv) | `local_shell_call_output` |
| `tool_search` (execution `client`) | `tool_search_call` | `tool_search_output` (its tools join the prompt's tool list; `defer_loading` honoured) |

`web_search` / `web_search_preview` are accepted and never run; the other
hosted tools are refused by name. A function's `strict` is accepted and not
enforced. A tool loop:

```python
from layerscale.types.responses import ResponseFunctionToolCall

res = client.responses.create(input="What is the weather in Paris?", tools=[weather_tool])
while calls := [item for item in res.output if isinstance(item, ResponseFunctionToolCall)]:
    res = client.responses.create(
        previous_response_id=res.id,
        tools=[weather_tool],
        input=[{"type": "function_call_output", "call_id": c.call_id, "output": run(c)} for c in calls],
    )
print(res.output_text)
```

### Responses streaming

```python
for event in client.responses.stream(input="Hi"):
    if event.type == "response.output_text.delta":
        print(event.delta, end="")
    elif event.type == "response.completed":
        print(event.response.usage)
```

Frames are `event:` + `data:` with `type` and a `sequence_number` from 0;
there is no `[DONE]`. The order: `response.created`, `response.in_progress`,
then per output item `response.output_item.added`, its part's
`response.content_part.added`, the deltas (`response.reasoning_text.delta`,
with `response.reasoning_summary_text.delta` beside it when
`reasoning.summary` was asked for; `response.output_text.delta`; each call
kind's own, such as `response.function_call_arguments.delta`), their `.done`,
the part's and the item's `.done`, and last `response.completed` or
`response.incomplete`, which carries the whole response. A call's item is
added the moment its name reads; its payload follows in one delta when it
closes. The stream ends after the last event. The status line waits for the
first token, so a refusal before it is an HTTP error (`APIStatusError`).
Inside the stream, the `error` event raises `APIStreamError` with `code`,
`param` and `context_overflow`: a prompt past the context fails this way
(`created`, `in_progress`, `error` with `code == "context_length_exceeded"`),
as OpenAI's stream reports it, and an engine fault after the first token as
`server_error`. The `response.failed` that follows is not read.

### Stored responses and sessions

`previous_response_id` and `store` decide what the model sees: a response
stored with `store=True` (the default) is kept with its conversation, and a
request naming it renders that conversation and then its own `input` exactly
as the whole history sent at once renders, so the prefix cache serves the
earlier turns (`input_tokens_details.cached_tokens`). A LayerScale session
decides where a conversation's KV stays: resident, past prefix-cache
eviction, with tool loops (custom and shell calls included) resumed exactly.
The two are independent, and this client keeps them so: it creates no session
per chain and never sets `store`.

- **Memory limits**: the server keeps at most 16,384 responses and 512 MiB of
  them, least recently used out, and forgets them on restart. A kept response
  holds its whole conversation, so it continues after earlier responses of it
  are evicted or deleted.
- **Long-lived agents** take one of two paths. Send `store=False` with
  `include=["reasoning.encrypted_content"]` and replay the items yourself
  (`input=[*history, *res.output, next_message]`), as Codex and OpenCode do;
  nothing is kept on the server. Or keep `store` and catch
  `previous_response_not_found` (a 400, `param == "previous_response_id"`,
  also for an unknown `item_reference`) and send the whole history again, as
  Copilot does:

```python
from layerscale import BadRequestError

try:
    res = client.responses.create(previous_response_id=last.id, input=next_items)
except BadRequestError as exc:
    if exc.code != "previous_response_not_found":
        raise
    res = client.responses.create(input=[*history, *next_items])
```

- **Both at once**: `sessions.create()` once, then every turn
  `responses.create(session_id=..., previous_response_id=..., input=...)`.
  Deleting a session does not delete stored responses, and deleting a
  response does not touch the session.
- **Evaluation mode**: a keyless server allows one live session; a second
  `sessions.create()` is a 403 (see [Evaluation mode](#evaluation-mode)).
  Stored responses have no such cap.
- **`encrypted_content`** is `lsr1.` and the reasoning in base64, an encoding
  rather than encryption: treat it as the reasoning text. One this server did
  not produce is refused.
- **Usage**: `input_tokens_details.cached_tokens` counts the prefix cache and
  the session together (this route has no `prefix_cached_tokens` split),
  `cache_write_tokens` is always 0, `output_tokens_details.reasoning_tokens`
  counts the reasoning.

Chat-only fields (`skip_special_tokens`, `separate_reasoning`,
`chat_template_kwargs`) are not read on this route. The official `openai`
Python SDK works against `/v1/responses` too
(`OpenAI(base_url="http://127.0.0.1:8080/v1", api_key=...)`, with any
`model`: its `responses.stream()` helper requires one, the server ignores
it), with `top_k` in `extra_body` and the session as an
`x-layerscale-session-id` header in `extra_headers`. Its stream yields the
`error` event and `response.failed` instead of raising; this client raises
`APIStreamError`.

## Messages (Anthropic-compatible)

```python
resp = client.messages.create(
    messages=[{"role": "user", "content": "Weather in Paris?"}],
    max_tokens=256,                      # required by the server
    system="Be brief.",                  # string or a list of text blocks
    tools=[{"name": "get_weather", "input_schema": {"type": "object"}}],
    tool_choice={"type": "auto"},        # "auto", "none", "any", or {"type": "tool", "name": ...}
)
for block in resp.content:
    if block.type == "text":
        print(block.text)
    elif block.type == "tool_use":
        # input is the parsed JSON arguments, or the raw string if the
        # model emitted invalid JSON
        print(block.name, block.input)
```

Roles are `user` and `assistant`, with the system prompt in the top-level
`system` field; a server that serves `/v1/responses` also takes `system`
messages inside `messages`, as Claude Code sends them to a model it does not
know. On such a server the Messages surface also takes what Claude Code
sends: a `tool_result` whose content holds `text`, `image` and
`tool_reference` blocks, tools with `type: "custom"`, `defer_loading` (kept
out of the system prompt until a `tool_reference` loads it), `strict` and
`cache_control`, and unknown `output_config` keys. A `document` block, and an
image to a checkpoint without image input, are refused with exactly
`capability_rejected: document_block` / `capability_rejected: image_block`, on
which Claude Code sends the request again without them; Anthropic's typed
tools (`web_search_20250305`, ...) are refused by name.

Notes: `top_k` and `skip_special_tokens` work as on chat completions, and
`output_config={"format": ...}` holds the text to a JSON schema (see
[Structured outputs](#structured-outputs)); audio blocks are refused.
`tool_choice` `any` or a named tool forces a call on every template that calls tools (a forced choice runs
without thinking, and thinking turned on beside one is a 400), and
`"disable_parallel_tool_use": True` beside the type limits the turn to one
call. User content may carry `image` blocks with a `base64` source on a
model with image input. `usage` carries Anthropic's split:
`cache_read_input_tokens` counts the prompt tokens served from cache (held
session KV and prefix-cache pages alike), `input_tokens` is the rest, so the
two add up to the prompt; `cache_creation_input_tokens` is always 0 and
`prefix_cached_tokens`, present only when nonzero, is the prefix cache's
part of the reads; `output_tokens` counts the end of sequence a reply stopped
on. `stop_reason` adds `model_context_window_exceeded` to
Anthropic's four, for a reply the context ended before `max_tokens`, and a
call cut by `max_tokens` or a stop sequence ends `max_tokens` /
`stop_sequence`, not `tool_use`. A `tool_use` block's
`input` is usually a parsed JSON object, but falls back to the raw arguments
string when the model's arguments were not valid JSON, guard with
`isinstance(block.input, dict)` before indexing into it. Inside a
`tool_result`'s content array every part must carry an explicit
`type: "text"` (unlike top-level blocks, where it defaults), a part without
one is a 400. Errors from this endpoint use the Anthropic envelope
(`{"type": "error", "error": {...}}`), the exception classes parse both
envelopes.

`client.messages.count_tokens(...)` returns the prompt length the same
request would run with (system, messages, tools, a forced `tool_choice`'s
opening, image placeholders and the thinking setting) as `input_tokens`,
counted on the tokenizer alone; `max_tokens` and the sampling fields are not
needed.

### Thinking

On a checkpoint that reasons, the reply opens with a `ThinkingBlock`
(`thinking`, `signature`) ahead of the text and the calls; the signature is
an opaque id to keep with the block and send back (it is never checked).
`thinking={"type": "enabled", "budget_tokens": N}` (at least 1024, not
enforced), `{"type": "adaptive"}` or `{"type": "disabled"}` and
`output_config={"effort": ...}` (`"low"` off, `"medium"` … `"max"` on) switch
it, as `reasoning_effort` does on chat completions; with neither, the
template's default applies. `"display": "omitted"` beside the type sends the
block with empty text. Echo `thinking` blocks (and `redacted_thinking`, which
is dropped) on assistant messages as the reply sent them.

### Messages streaming, SSE dialect B

Typed `event:` + `data:` frames, **no `[DONE]` sentinel**:

```python
for event in client.messages.stream(messages=[...], max_tokens=256):
    if event.type == "content_block_delta" and event.delta.type == "text_delta":
        print(event.delta.text, end="")
```

Event order: `message_start` (the `ping` the server sends after it is dropped,
as Anthropic's SDKs drop it) → (`content_block_start` / `content_block_delta`
/ `content_block_stop`: the thinking block first, its `ThinkingDelta`s closed
by one `SignatureDelta`; then text; then one block per tool call with a
single `InputJsonDelta`)* → `message_delta` → `message_stop`. The
`message_delta` usage carries the final figures, `input_tokens` included;
`message_start`'s are provisional (the whole prompt, zero cache reads). A
mid-stream failure arrives as an `event: error` frame and raises
`APIStreamError`. An event type this client does not know comes through as an
`UnknownStreamEvent` with every field it carried, so keep a fallback branch.

## Health

`client.health.check()` never raises:

```python
health = client.health.check()
print(health.ok, health.status)          # True/False, 200 | 503 | 0 (connection error)
if health.body:
    print(health.body.status, health.body.sessions)  # "ok" | "unavailable", counters
```

Health lives at `GET /health` (also `/healthz`), **not** under `/v1`. The
body's counters include `background_yields` and the structured-output ones
(`structured_requests`, `structured_forced_runs`, `structured_forced_tokens`,
`structured_refused_tokens`).

`client.metrics.get()` returns the server's Prometheus metrics
(`GET /metrics`, `text/plain; version=0.0.4`) verbatim as a string. Unlike
`/health`, the route is not exempt from license bearer auth, so under
`--api-key` a scraper needs the key too.

## Sessions

Sessions are explicit, durable, and shared by both chat surfaces via
`session_id`, which the client sends as the `x-layerscale-session-id`
request header (the server refuses a `session_id` body field). A session
nobody uses for an hour is deleted as `delete` would delete it; every
request that names it restarts the clock (a turn, `get`, `warm`, `push`,
`generate`, a flash change, opening an event stream), the listing does not,
and a session is never expired while a request, pending data or an event
stream is on it. A client that comes back within the hour adopts its
sessions with `get`: 200 means live (`kv_tokens` says how much is held, and
`last_used` when it was last touched), 404 means gone.

```python
session = client.sessions.create(id="ticker-feed", window=4096)  # all fields optional

reply = client.chat.create(messages=[{"role": "user", "content": "Summarise the feed."}], session_id=session.id)

# wait=True polls until the pushed batch is ingested and KV-resident;
# the push-free alternative is watching the events stream / WS `data_updated`.
client.sessions.push(session.id, ["AAPL 187.44 +0.5%", "MSFT 420.21 -0.1%"], wait=True)

answer = client.sessions.generate(session.id, prompt="What's trending?", max_tokens=32)
print(answer.text, answer.usage.prompt_tokens)

client.sessions.delete(session.id)
```

- `push` takes raw text, a string or a non-empty list of strings. The ack is
  immediate; backpressure never blocks or errors (overflow drops the oldest
  unprocessed entries, counted in `dropped`). `wait=True` (with
  `wait_timeout=30.0`, `wait_interval=0.1`) is a client-side convenience
  that polls `get()` until the server's residency predicate holds —
  `pending == 0`, `data_version` stamped past the ack's, and
  `kv_tokens + 1 >= data_end` (`pending == 0` alone is not proof of
  ingestion), and raises `TimeoutError` on the deadline; the server never
  blocks on push.
- `generate` is **prompt-only** (no messages/tools) and ephemeral, the Q/A
  tail is evicted by the next durable change. Optional `fast_answer`
  (list of candidate strings) and `gap_threshold` enable the speculative
  ready-position exit; hits come back with `speculative: True` /
  `logit_gap`, flash-cache hits with `flash: True` / `flash_id` /
  `confidence`, both with zero usage and no `total_tokens`.
  `reasoning_effort` and `chat_template_kwargs` switch the question's
  thinking as on chat completions (a question whose setting changes the
  generation prompt is never served from the flash cache).
- `delete` raises `ConflictError` (409) while a foreground request is in
  flight, and may take up to ~2 s while background ingest is cancelled.

### Warm turns

`client.sessions.warm(id, messages=..., tools=...)` prefills a conversation
into the session ahead of its next turn and generates nothing. Send it
whenever part of the next prompt is known before the turn can be sent (tool
results as each call finishes, the conversation while the user is still
typing); the turn then resumes past whatever the warm turn computed.

```python
warm = client.sessions.warm(session.id, messages=messages, tools=tools)
print(warm.finish_reason, warm.prompt_tokens, warm.cached_tokens, warm.prefix_cached_tokens)
```

The body is the chat request the next turn would send; only the messages,
the tools and the thinking settings are read. It runs as background work:
the session's next turn cancels it at a step boundary rather than getting a
409, and a newer warm turn takes over from an older one the same way.
`finish_reason` is `"warmed"` (every token has KV), `"cancelled"` (a turn
took over; what was computed is kept) or `"covered"` (the session already
held it all). A warm turn sent while a turn is in flight raises
`ConflictError`; a conversation past the context is a 400 overflow like a
turn's. The call returns once the warm turn ends.

### Streaming generate

Dialect A: `{"text": ...}` frames, then a `{"done": true, ...}` result frame,
then `data: [DONE]` (consumed by the client):

```python
from layerscale.types import GenerationDone

for chunk in client.sessions.generate_stream(session.id, prompt="Trend?"):
    if isinstance(chunk, GenerationDone):
        print(f"\n[{chunk.finish_reason}] v{chunk.data_version}")
    else:
        print(chunk.text, end="")
```

### Flash queries

Standing questions re-evaluated in the background after every ingested batch
(at most 20 per session; duplicates are a 409):

```python
fq = client.sessions.flash(session.id, "Is the market bullish or bearish?", max_tokens=16)
listed = client.sessions.list_flash(session.id)   # .data, .data_version
client.sessions.unflash(session.id, fq.id)
```

A flash query's `value` / `data_version` / `confidence` / `evaluated_at`
fields exist only once an answer has been computed; `fresh` means the answer
matches the current `data_version`.

### Events stream (SSE dialect B)

```python
for event in client.sessions.events(session.id):
    if event.type == "connected":       # always first: session_id, data_version, flash_queries
        ...
    elif event.type == "flash_ready":   # replayed cached answers, then live changes
        print(event.query, "→", event.value)
    elif event.type == "data_updated":  # every durable settle
        print("v", event.data_version, "tokens", event.tokens, "pending", event.pending)
```

Undecodable frames are dropped (lenient decoding); heartbeat comments
(`: heartbeat`, sent after 15 s of event silence, this is the only stream
that has them) are ignored. When the stream ends (session deleted, subscriber
too far behind, shutdown), reconnect and resync from `connected` + replay.

### WebSocket

```python
from layerscale import WsFlashReady, WsPushAck

with client.sessions.ws(session.id) as ws:   # async: `async with`, `await ws.push(...)`
    ws.push(["line one", "line two"])        # → push_ack event
    ws.ping()                                # JSON ping → pong event
    for event in ws:                         # first event is always `connected`
        if isinstance(event, WsPushAck):
            print("acked at v", event.data.data_version)
        elif isinstance(event, WsFlashReady):
            print(event.data.query, "→", event.data.value)
```

The server sends a protocol ping (`hb`) every 15 s; the `websockets` library
answers pings automatically, so the socket stays alive without extra code.
JSON `error` events are non-fatal (the connection stays open). `close()`
sends the JSON `{"type": "close"}` message, then the WebSocket close frame.

The upgrade request carries the client's configured headers (Authorization,
`default_headers`, User-Agent) just like every HTTP request. A refused
upgrade, the server answers plain HTTP instead of `101`, raises the same
typed errors as the HTTP surface: `AuthenticationError` for a missing or
invalid license key (401, refused before the handshake), `NotFoundError` for
an unknown session (404), `BadRequestError` for a malformed path (400).

The server refuses any inbound frame over 16 MiB before reading its payload
and tears the connection down, no JSON `error` event, the socket just
drops, so split larger pushes into smaller batches.

## Errors

```
LayerScaleError
├── APIError
│   ├── APIConnectionError
│   │   └── APITimeoutError
│   ├── APIStreamError          # mid-SSE error frame: .message, .type, .code, .param, .context_overflow, .body
│   └── APIStatusError          # .status_code, .message, .type, .code, .param, .context_overflow, .body, .response
│       ├── BadRequestError           (400)
│       ├── AuthenticationError       (401)
│       ├── PermissionDeniedError     (403)
│       ├── NotFoundError             (404)
│       ├── ConflictError             (409)
│       ├── UnprocessableEntityError  (422)
│       ├── RateLimitError            (429)
│       └── InternalServerError       (5xx)
```

Both server envelopes are parsed: `{"error": {"message", "type"}}` on every
route (with `param` and `code` on a context overflow, and always on the
`/v1/responses` routes, `None` where none applies: `code ==
"previous_response_not_found"`, `param == "previous_response_id"` for a
stored response the server does not hold), and `{"type": "error", "error":
{"type", "message"}, "request_id"}` on the routes under `/v1/messages`.
`context_overflow` is set for a prompt past the context on either surface
(see [Context window](#context-window)). The Responses stream's flat `error`
event (`{"type": "error", "code", "message", "param"}`) is raised as an
`APIStreamError` whose `body` is the OpenAI envelope built from it, its
`type` `invalid_request_error` for `context_length_exceeded` and
`server_error` otherwise.

### Retries

- **Only GET requests retry**, on connection errors, 408, and the
  body-budget 503 (`"server is holding too many request bodies"`), with
  exponential backoff (`max_retries`, default 2). Other 503s (e.g. the
  not-ready `/health` answer or a wedged session) are not transient and are
  **not** retried.
- **POST and DELETE never retry.** A retried DELETE whose first attempt
  succeeded but whose response was lost would surface as a spurious 404.
- **409 is never retried**, on this server it means duplicate-create or
  session-busy, not a transient conflict.
- Streams never retry.

## Timeouts & per-request options

Default `httpx` timeout: **600 s** total with **5 s** connect. Override per
client (`LayerScale(timeout=30.0)`); `timeout=0` or `timeout=None` disables
the timeout entirely (matching the TypeScript client's `timeoutMs: 0`).
Streaming requests skip the default read timeout, they keep the 5 s connect
bound but the read side is open-ended by design (session event streams idle
between events).

Every resource method also accepts a keyword-only `request_options` mapping
with per-request overrides:

```python
client.models.list(request_options={"timeout": 5.0, "max_retries": 0, "headers": {"X-Trace": "1"}})
```

To cancel an in-flight streaming request, close the `Stream`/`AsyncStream`
(or leave its `with` block), the server notices the disconnect and cancels
generation; for non-streaming calls the `timeout` override is the
cancellation mechanism.

Note the server closes idle kept-alive connections after ~30 s, the client
treats the resulting connection errors as retryable for GET requests.

## Examples

- [`examples/example_http.py`](examples/example_http.py), health, models,
  chat with tool calling, session create/push/generate, flash queries, SSE.
- [`examples/example_ws.py`](examples/example_ws.py), async client, WebSocket
  push and live flash results.
- [`examples/example_responses.py`](examples/example_responses.py), an agent's
  tool loop on the Responses API inside a session (without one on a keyless
  server's cap), then a streamed turn.

## Development

```bash
pip install -e '.[dev]'
ruff check src/ examples/ tests/
pyright                      # src, tests and examples, at Python 3.11
pytest                       # offline: the server is mocked with respx
LAYERSCALE_TEST_URL=http://127.0.0.1:8080 pytest tests/live
                             # against a real server (LAYERSCALE_TEST_EVALUATION=1 adds a keyless
                             # server's caps); the live tests skip without LAYERSCALE_TEST_URL
python -m build
```
