Metadata-Version: 2.5
Name: layerscale
Version: 0.8.0
Summary: Python client for the LayerScale inference server
Project-URL: Homepage, https://layerscale.ai
Author: LayerScale
License: MIT
Keywords: inference,layerscale,llm,sessions,streaming
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: anyio<5,>=4.15.1
Requires-Dist: httpx<1,>=0.28.1
Requires-Dist: pydantic<3,>=2.14.0
Requires-Dist: typing-extensions<5,>=4.16.0
Requires-Dist: websockets<18,>=17.2
Provides-Extra: dev
Requires-Dist: pyright>=1.1.414; extra == 'dev'
Requires-Dist: pytest-asyncio>=1.4.0; extra == 'dev'
Requires-Dist: pytest>=9.1.1; extra == 'dev'
Requires-Dist: python-dotenv>=1.2.4; extra == 'dev'
Requires-Dist: respx>=0.23.1; extra == 'dev'
Requires-Dist: ruff>=0.16.10; extra == 'dev'
Description-Content-Type: text/markdown

# layerscale

Python client for the LayerScale inference server. Sync (`LayerScale`) and
async (`AsyncLayerScale`) clients with identical surfaces, typed requests
(TypedDicts) and responses (pydantic v2), both server SSE dialects, and a
WebSocket session socket.

Requires Python 3.11+. Runtime dependencies: `httpx`, `pydantic` v2,
`typing-extensions`, `anyio`, `websockets` (imported lazily, only if you open
a WebSocket).

```bash
pip install layerscale
```

## Quickstart

```python
from layerscale import LayerScale

client = LayerScale()  # defaults to http://127.0.0.1:8080

resp = client.chat.create(messages=[{"role": "user", "content": "Hello!"}])
print(resp.choices[0].message.content)
```

```python
from layerscale import AsyncLayerScale

async with AsyncLayerScale() as client:
    resp = await client.chat.create(messages=[{"role": "user", "content": "Hello!"}])
```

The base URL falls back to the `LAYERSCALE_BASE_URL` environment variable,
then to `http://127.0.0.1:8080` (the server's default bind).

### Authentication

The server needs its Ed25519 license key (format `LSK-...`) to start, but by
default its API is open and needs no key. Started with `--api-key`, the
license key is the API bearer token: every route except the CORS
preflight and `/health`/`/healthz` (this includes `/metrics`) requires
`Authorization: Bearer <LSK-...>` (the server also accepts `x-api-key`) and
answers 401 `authentication_error` without it; the WebSocket upgrade is
refused with a plain-HTTP 401 before the handshake. Pass the key as the
`api_key` constructor option, or let it fall back to the `LAYERSCALE_API_KEY`
env var, then `LAYERSCALE_LICENSE_KEY` (the server's own canonical variable
for the key). The client sends it as `Authorization: Bearer <key>` on every
request, including the WebSocket handshake; see
[.env.example](.env.example).

## Endpoints

| Client call | Endpoint |
|---|---|
| `client.chat.create(...)` / `client.chat.stream(...)` | `POST /v1/chat/completions` |
| `client.messages.create(...)` / `client.messages.stream(...)` | `POST /v1/messages` |
| `client.messages.count_tokens(...)` | `POST /v1/messages/count_tokens` |
| `client.models.list()` / `client.models.retrieve(id)` | `GET /v1/models`, `GET /v1/models/{id}` |
| `client.health.check()` | `GET /health` |
| `client.metrics.get()` | `GET /metrics` (Prometheus text) |
| `client.sessions.create/list/get/delete` | `POST/GET /v1/sessions`, `GET/DELETE /v1/sessions/{id}` |
| `client.sessions.push(id, data)` | `POST /v1/sessions/{id}/push` |
| `client.sessions.warm(id, ...)` | `POST /v1/sessions/{id}/warm` |
| `client.sessions.generate(id, ...)` / `generate_stream(id, ...)` | `POST /v1/sessions/{id}/generate` |
| `client.sessions.flash/list_flash/unflash` | `POST/GET /v1/sessions/{id}/flash`, `DELETE .../flash/{fid}` |
| `client.sessions.events(id)` | `GET /v1/sessions/{id}/events` (SSE) |
| `client.sessions.ws(id)` | `GET /v1/sessions/{id}/ws` (WebSocket) |

Sampling defaults everywhere: `temperature` **0.6**, `top_p` **0.9**,
no `max_tokens` cap (it is required on `/v1/messages`). Unsupported parameters are refused by the server with a
400 naming the parameter, the client never forwards fields you didn't pass.

### Context window

`client.models.list()` reports `max_model_len`, the longest sequence (prompt
plus completion) the server admits once the model's context, the license,
`--ctx` and the KV pool have each capped it, and `max_input_tokens`, one
less, the longest prompt. Size your own window from it. A prompt that does
not fit is refused with 400 before anything streams, and the session (if
any) is free again and keeps what it held: `APIStatusError.context_overflow`
is true for that refusal on either surface (`code ==
"context_length_exceeded"` on the OpenAI routes, `prompt is too long: N
tokens > M maximum` on `/v1/messages`), so compact the conversation and
retry. A reply that fills the context ends with `finish_reason="length"`
(`stop_reason="model_context_window_exceeded"` on `/v1/messages`).

## Chat (OpenAI-compatible)

```python
resp = client.chat.create(
    messages=[
        {"role": "system", "content": "Be brief."},
        {"role": "user", "content": "What's the AAPL price?"},
    ],
    tools=[{"type": "function", "function": {
        "name": "get_stock_price",
        "parameters": {"type": "object", "properties": {"symbol": {"type": "string"}}},
    }}],
    tool_choice="auto",   # "auto", "none", "required", or {"type": "function", "function": {"name": ...}}
)
choice = resp.choices[0]
if choice.finish_reason == "tool_calls":
    for call in choice.message.tool_calls or []:
        print(call.function.name, call.function.arguments)  # arguments is a JSON string
```

Notes:

- `model` is optional and **ignored**, the response always carries the
  server's model name.
- On prose replies the `tool_calls` key is absent; with tool calls, `content`
  is the text ahead of the first call, or `None` when there was none.
- `tool_choice="required"` or a named function forces a call, on every
  template that calls tools, and runs the turn without thinking unless the
  request asks for it. `parallel_tool_calls=False` limits the turn to one
  call.
- `reasoning_effort` (`"none"` … `"max"`) or
  `chat_template_kwargs={"enable_thinking": ...}` switch a reasoning model's
  thinking. `logprobs=True` (plus `top_logprobs`, 0–20) fills
  `choices[0].logprobs`; when streaming, it arrives on the final chunk.
- User messages may carry `{"type": "image_url", "image_url": {"url":
  "data:image/png;base64,..."}}` parts on a model with image input; only
  base64 `data:` URLs are accepted.
- `usage.prompt_tokens_details.cached_tokens` is always present and counts
  every prompt token served from cache: the tokens a session turn resumed
  from held KV, plus the full pages the prefix cache supplied to any request
  (`prefix_cached_tokens`, present only when nonzero, is that part).
  `prompt_tokens - cached_tokens` is what the request computed.

### Reasoning

A checkpoint that thinks before it answers returns its reasoning apart from
the answer, as `message.reasoning` (a string without the block's markers, or
`None` when the reply had none) and, when streaming, as `delta.reasoning`
chunks ahead of the content; `usage.completion_tokens_details.reasoning_tokens`
counts its tokens. Send it back on the assistant message as `reasoning` (the
older `reasoning_content` is accepted too) and the template renders it as the
reference does. Three options, each default true: `include_reasoning=False`
leaves the reasoning out of the reply (still generated and counted),
`separate_reasoning=False` leaves it in `content` as the model wrote it,
markers included, and `stream_reasoning=False` sends it as one delta once
complete. Checkpoints without a reasoning mode never carry the field.
- LayerScale extensions: `session_id` (bind the turn to a session; sent as
  the `x-layerscale-session-id` header, not in the body), `ignore_eos`,
  `seed`, `chat_template_kwargs`.

### Chat streaming, SSE dialect A

Untyped `data:` frames terminated by `data: [DONE]` (consumed by the client).
Chat streams send **no heartbeats**, and no bytes at all during prefill;
the only stream with `: heartbeat` comments is the session events stream
(the parser drops SSE comments defensively everywhere):

```python
for chunk in client.chat.stream(
    messages=[{"role": "user", "content": "Hello!"}],
    stream_options={"include_usage": True},
):
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")
    elif chunk.usage is not None:
        print(f"\n{chunk.usage.total_tokens} tokens")  # final usage chunk, choices == []
```

Tool calls stream as exactly two chunks per call: an opener with
`id`/`type`/`name` and empty `arguments`, then the full arguments string
(never split). With `stream_options={"include_usage": True}` every chunk
carries `usage: null` until the final usage chunk (which has `choices: []`).

## Messages (Anthropic-compatible)

```python
resp = client.messages.create(
    messages=[{"role": "user", "content": "Weather in Paris?"}],
    max_tokens=256,                      # required by the server
    system="Be brief.",                  # string or a list of text blocks
    tools=[{"name": "get_weather", "input_schema": {"type": "object"}}],
    tool_choice={"type": "auto"},        # "auto", "none", "any", or {"type": "tool", "name": ...}
)
for block in resp.content:
    if block.type == "text":
        print(block.text)
    elif block.type == "tool_use":
        # input is the parsed JSON arguments, or the raw string if the
        # model emitted invalid JSON
        print(block.name, block.input)
```

Notes: `top_k` is refused by the server; `tool_choice` `any` or a named
tool forces a call on every template that calls tools (a forced choice runs
without thinking, and thinking turned on beside one is a 400), and
`"disable_parallel_tool_use": True` beside the type limits the turn to one
call. User content may carry `image` blocks with a `base64` source on a
model with image input. `usage` carries Anthropic's split:
`cache_read_input_tokens` counts the prompt tokens served from cache (held
session KV and prefix-cache pages alike), `input_tokens` is the rest, so the
two add up to the prompt; `cache_creation_input_tokens` is always 0 and
`prefix_cached_tokens`, present only when nonzero, is the prefix cache's
part of the reads. `stop_reason` adds `model_context_window_exceeded` to
Anthropic's four, for a reply the context ended before `max_tokens`. A `tool_use` block's
`input` is usually a parsed JSON object, but falls back to the raw arguments
string when the model's arguments were not valid JSON, guard with
`isinstance(block.input, dict)` before indexing into it. Inside a
`tool_result`'s content array every part must carry an explicit
`type: "text"` (unlike top-level blocks, where it defaults), a part without
one is a 400. Errors from this endpoint use the Anthropic envelope
(`{"type": "error", "error": {...}}`), the exception classes parse both
envelopes.

`client.messages.count_tokens(...)` returns the prompt length the same
request would run with (system, messages, tools, a forced `tool_choice`'s
opening, image placeholders and the thinking setting) as `input_tokens`,
counted on the tokenizer alone; `max_tokens` and the sampling fields are not
needed.

### Thinking

On a checkpoint that reasons, the reply opens with a `ThinkingBlock`
(`thinking`, `signature`) ahead of the text and the calls; the signature is
an opaque id to keep with the block and send back (it is never checked).
`thinking={"type": "enabled", "budget_tokens": N}` (at least 1024, not
enforced), `{"type": "adaptive"}` or `{"type": "disabled"}` and
`output_config={"effort": ...}` (`"low"` off, `"medium"` … `"max"` on) switch
it, as `reasoning_effort` does on chat completions; with neither, the
template's default applies. `"display": "omitted"` beside the type sends the
block with empty text. Echo `thinking` blocks (and `redacted_thinking`, which
is dropped) on assistant messages as the reply sent them.

### Messages streaming, SSE dialect B

Typed `event:` + `data:` frames, **no `[DONE]` sentinel**:

```python
for event in client.messages.stream(messages=[...], max_tokens=256):
    if event.type == "content_block_delta" and event.delta.type == "text_delta":
        print(event.delta.text, end="")
```

Event order: `message_start` → (`content_block_start` / `content_block_delta`
/ `content_block_stop`: the thinking block first, its `ThinkingDelta`s closed
by one `SignatureDelta`; then text; then one block per tool call with a
single `InputJsonDelta`)* → `message_delta` → `message_stop`. The
`message_delta` usage carries the final figures, `input_tokens` included;
`message_start`'s are provisional (the whole prompt, zero cache reads). A
mid-stream failure arrives as an `event: error` frame and raises
`APIStreamError`.

## Health

`client.health.check()` never raises:

```python
health = client.health.check()
print(health.ok, health.status)          # True/False, 200 | 503 | 0 (connection error)
if health.body:
    print(health.body.status, health.body.sessions)  # "ok" | "unavailable", counters
```

Health lives at `GET /health` (also `/healthz`), **not** under `/v1`.

`client.metrics.get()` returns the server's Prometheus metrics
(`GET /metrics`, `text/plain; version=0.0.4`) verbatim as a string. Unlike
`/health`, the route is not exempt from license bearer auth, so under
`--api-key` a scraper needs the key too.

## Sessions

Sessions are explicit, durable, and shared by both chat surfaces via
`session_id`, which the client sends as the `x-layerscale-session-id`
request header (the server refuses a `session_id` body field). A session
nobody uses for an hour is deleted as `delete` would delete it; every
request that names it restarts the clock (a turn, `get`, `warm`, `push`,
`generate`, a flash change, opening an event stream), the listing does not,
and a session is never expired while a request, pending data or an event
stream is on it. A client that comes back within the hour adopts its
sessions with `get`: 200 means live (`kv_tokens` says how much is held, and
`last_used` when it was last touched), 404 means gone.

```python
session = client.sessions.create(id="ticker-feed", window=4096)  # all fields optional

reply = client.chat.create(messages=[{"role": "user", "content": "Summarise the feed."}], session_id=session.id)

# wait=True polls until the pushed batch is ingested and KV-resident;
# the push-free alternative is watching the events stream / WS `data_updated`.
client.sessions.push(session.id, ["AAPL 187.44 +0.5%", "MSFT 420.21 -0.1%"], wait=True)

answer = client.sessions.generate(session.id, prompt="What's trending?", max_tokens=32)
print(answer.text, answer.usage.prompt_tokens)

client.sessions.delete(session.id)
```

- `push` takes raw text, a string or a non-empty list of strings. The ack is
  immediate; backpressure never blocks or errors (overflow drops the oldest
  unprocessed entries, counted in `dropped`). `wait=True` (with
  `wait_timeout=30.0`, `wait_interval=0.1`) is a client-side convenience
  that polls `get()` until the server's residency predicate holds —
  `pending == 0`, `data_version` stamped past the ack's, and
  `kv_tokens + 1 >= data_end` (`pending == 0` alone is not proof of
  ingestion), and raises `TimeoutError` on the deadline; the server never
  blocks on push.
- `generate` is **prompt-only** (no messages/tools) and ephemeral, the Q/A
  tail is evicted by the next durable change. Optional `fast_answer`
  (list of candidate strings) and `gap_threshold` enable the speculative
  ready-position exit; hits come back with `speculative: True` /
  `logit_gap`, flash-cache hits with `flash: True` / `flash_id` /
  `confidence`, both with zero usage and no `total_tokens`.
  `reasoning_effort` and `chat_template_kwargs` switch the question's
  thinking as on chat completions (a question whose setting changes the
  generation prompt is never served from the flash cache).
- `delete` raises `ConflictError` (409) while a foreground request is in
  flight, and may take up to ~2 s while background ingest is cancelled.

### Warm turns

`client.sessions.warm(id, messages=..., tools=...)` prefills a conversation
into the session ahead of its next turn and generates nothing. Send it
whenever part of the next prompt is known before the turn can be sent (tool
results as each call finishes, the conversation while the user is still
typing); the turn then resumes past whatever the warm turn computed.

```python
warm = client.sessions.warm(session.id, messages=messages, tools=tools)
print(warm.finish_reason, warm.prompt_tokens, warm.cached_tokens, warm.prefix_cached_tokens)
```

The body is the chat request the next turn would send; only the messages,
the tools and the thinking settings are read. It runs as background work:
the session's next turn cancels it at a step boundary rather than getting a
409, and a newer warm turn takes over from an older one the same way.
`finish_reason` is `"warmed"` (every token has KV), `"cancelled"` (a turn
took over; what was computed is kept) or `"covered"` (the session already
held it all). A warm turn sent while a turn is in flight raises
`ConflictError`; a conversation past the context is a 400 overflow like a
turn's. The call returns once the warm turn ends.

### Streaming generate

Dialect A: `{"text": ...}` frames, then a `{"done": true, ...}` result frame,
then `data: [DONE]` (consumed by the client):

```python
from layerscale.types import GenerationDone

for chunk in client.sessions.generate_stream(session.id, prompt="Trend?"):
    if isinstance(chunk, GenerationDone):
        print(f"\n[{chunk.finish_reason}] v{chunk.data_version}")
    else:
        print(chunk.text, end="")
```

### Flash queries

Standing questions re-evaluated in the background after every ingested batch
(at most 20 per session; duplicates are a 409):

```python
fq = client.sessions.flash(session.id, "Is the market bullish or bearish?", max_tokens=16)
listed = client.sessions.list_flash(session.id)   # .data, .data_version
client.sessions.unflash(session.id, fq.id)
```

A flash query's `value` / `data_version` / `confidence` / `evaluated_at`
fields exist only once an answer has been computed; `fresh` means the answer
matches the current `data_version`.

### Events stream (SSE dialect B)

```python
for event in client.sessions.events(session.id):
    if event.type == "connected":       # always first: session_id, data_version, flash_queries
        ...
    elif event.type == "flash_ready":   # replayed cached answers, then live changes
        print(event.query, "→", event.value)
    elif event.type == "data_updated":  # every durable settle
        print("v", event.data_version, "tokens", event.tokens, "pending", event.pending)
```

Undecodable frames are dropped (lenient decoding); heartbeat comments
(`: heartbeat`, sent after 15 s of event silence, this is the only stream
that has them) are ignored. When the stream ends (session deleted, subscriber
too far behind, shutdown), reconnect and resync from `connected` + replay.

### WebSocket

```python
from layerscale import WsFlashReady, WsPushAck

with client.sessions.ws(session.id) as ws:   # async: `async with`, `await ws.push(...)`
    ws.push(["line one", "line two"])        # → push_ack event
    ws.ping()                                # JSON ping → pong event
    for event in ws:                         # first event is always `connected`
        if isinstance(event, WsPushAck):
            print("acked at v", event.data.data_version)
        elif isinstance(event, WsFlashReady):
            print(event.data.query, "→", event.data.value)
```

The server sends a protocol ping (`hb`) every 15 s; the `websockets` library
answers pings automatically, so the socket stays alive without extra code.
JSON `error` events are non-fatal (the connection stays open). `close()`
sends the JSON `{"type": "close"}` message, then the WebSocket close frame.

The upgrade request carries the client's configured headers (Authorization,
`default_headers`, User-Agent) just like every HTTP request. A refused
upgrade, the server answers plain HTTP instead of `101`, raises the same
typed errors as the HTTP surface: `AuthenticationError` for a missing or
invalid license key (401, refused before the handshake), `NotFoundError` for
an unknown session (404), `BadRequestError` for a malformed path (400).

The server refuses any inbound frame over 16 MiB before reading its payload
and tears the connection down, no JSON `error` event, the socket just
drops, so split larger pushes into smaller batches.

## Errors

```
LayerScaleError
├── APIError
│   ├── APIConnectionError
│   │   └── APITimeoutError
│   ├── APIStreamError          # mid-SSE error frame
│   └── APIStatusError          # .status_code, .message, .type, .code, .context_overflow, .body, .response
│       ├── BadRequestError           (400)
│       ├── AuthenticationError       (401)
│       ├── PermissionDeniedError     (403)
│       ├── NotFoundError             (404)
│       ├── ConflictError             (409)
│       ├── UnprocessableEntityError  (422)
│       ├── RateLimitError            (429)
│       └── InternalServerError       (5xx)
```

Both server envelopes are parsed: `{"error": {"message", "type"}}` on every
route (with `param` and `code` on a context overflow), and `{"type":
"error", "error": {"type", "message"}, "request_id"}` on the routes under
`/v1/messages`. `context_overflow` is set for a prompt past the context on
either surface (see [Context window](#context-window)).

### Retries

- **Only GET requests retry**, on connection errors, 408, and the
  body-budget 503 (`"server is holding too many request bodies"`), with
  exponential backoff (`max_retries`, default 2). Other 503s (e.g. the
  not-ready `/health` answer or a wedged session) are not transient and are
  **not** retried.
- **POST and DELETE never retry.** A retried DELETE whose first attempt
  succeeded but whose response was lost would surface as a spurious 404.
- **409 is never retried**, on this server it means duplicate-create or
  session-busy, not a transient conflict.
- Streams never retry.

## Timeouts & per-request options

Default `httpx` timeout: **600 s** total with **5 s** connect. Override per
client (`LayerScale(timeout=30.0)`); `timeout=0` or `timeout=None` disables
the timeout entirely (matching the TypeScript client's `timeoutMs: 0`).
Streaming requests skip the default read timeout, they keep the 5 s connect
bound but the read side is open-ended by design (session event streams idle
between events).

Every resource method also accepts a keyword-only `request_options` mapping
with per-request overrides:

```python
client.models.list(request_options={"timeout": 5.0, "max_retries": 0, "headers": {"X-Trace": "1"}})
```

To cancel an in-flight streaming request, close the `Stream`/`AsyncStream`
(or leave its `with` block), the server notices the disconnect and cancels
generation; for non-streaming calls the `timeout` override is the
cancellation mechanism.

Note the server closes idle kept-alive connections after ~30 s, the client
treats the resulting connection errors as retryable for GET requests.

## Examples

- [`examples/example_http.py`](examples/example_http.py), health, models,
  chat with tool calling, session create/push/generate, flash queries, SSE.
- [`examples/example_ws.py`](examples/example_ws.py), async client, WebSocket
  push and live flash results.
