Metadata-Version: 2.4
Name: sakuradite
Version: 0.1.0
Summary: Structured decision inference with native PyTorch and MLX, embedded or over HTTP
License-Expression: Apache-2.0
Keywords: inference,classification,decisions,llm,pytorch,mlx
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: filelock<5,>=4.0.1
Requires-Dist: huggingface-hub<2,>=1.32
Requires-Dist: httpx<1,>=0.27
Requires-Dist: orjson<4,>=3.10
Requires-Dist: pillow<13,>=10
Requires-Dist: pydantic<3,>=2.9
Requires-Dist: typer<1,>=0.15
Provides-Extra: server
Requires-Dist: uvicorn<1,>=0.30; extra == "server"
Provides-Extra: torch
Requires-Dist: accelerate<2,>=1; extra == "torch"
Requires-Dist: safetensors<1,>=0.4; extra == "torch"
Requires-Dist: torch<3,>=2.6; extra == "torch"
Requires-Dist: torchvision<1,>=0.21; extra == "torch"
Requires-Dist: tokenizers<1,>=0.23.2; extra == "torch"
Requires-Dist: transformers<6,>=5.17; extra == "torch"
Provides-Extra: mlx
Requires-Dist: mlx<1,>=0.32.2; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "mlx"
Requires-Dist: mlx-lm<1,>=0.31.3; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "mlx"
Requires-Dist: mlx-vlm<0.8,>=0.7.1; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "mlx"
Requires-Dist: tokenizers<1,>=0.23.2; extra == "mlx"
Provides-Extra: dev
Requires-Dist: ruff<1,>=0.12; extra == "dev"
Requires-Dist: mypy<2,>=1.16; extra == "dev"
Requires-Dist: build<2,>=1.2; extra == "dev"
Requires-Dist: twine<7,>=6; extra == "dev"
Requires-Dist: typesafe-sdk==0.6.0; extra == "dev"
Dynamic: license-file

# Sakuradite

Structured decisions from open-weight language models, embedded in Python or
served over HTTP. Sakuradite scores answer candidates with native model forwards;
it is not a chat or text-generation server.

Questions return typed results:

- `Noul`: a probability for a true/false question.
- `Choice`: a selected key, candidate probabilities, and confidence.
- `Score`: an expected zero-based position in an ordered criteria list,
  candidate probabilities, a legend, and confidence.

Probabilities are normalized over the supplied candidates, not calibrated
guarantees of correctness.

## Install

Use Python 3.12+ on Linux or macOS. Install a published release with
`python -m pip install 'sakuradite[server,torch]'`, or install from a source checkout:

```sh
python -m pip install .                  # HTTP client and Typer CLI
python -m pip install '.[torch]'         # Embedded PyTorch / Transformers
python -m pip install '.[mlx]'           # Embedded MLX, Apple Silicon macOS only
python -m pip install '.[server,torch]'  # Uvicorn daemon with PyTorch
```

Choose the runtime you need; the base install includes neither model weights
nor an inference runtime. Use `.[server,mlx]` for an MLX daemon. CUDA execution
also needs a compatible PyTorch installation, GPU, and driver.

JSON uses `orjson`. Native inference requires Rust-backed fast tokenizers.
Serving uses the package's ASGI application and Uvicorn, without FastAPI or
SGLang in its request path. The optional MLX vision runtime installs its own
upstream dependencies, which may include web-framework packages.

## Embedded Python

Pass an actual Hugging Face repository ID or a local checkpoint directory.
Hub models are downloaded on first use; access-controlled models require an
existing authorized Hugging Face login or `HF_TOKEN`.

```python
from sakuradite import Choice, Noul, Sakuradite, Score

with Sakuradite("Qwen/Qwen3-0.6B", backend="torch") as client:
    result = client.decide(
        state="The package arrived damaged and the customer needs a replacement.",
        questions={
            "replacement": Noul(instructions="Does the customer need a replacement?"),
            "route": Choice(criteria={"shipping": "Delivery issues", "billing": "Payments"}),
            "urgency": Score(criteria=["Low urgency", "Medium urgency", "High urgency"]),
        },
    )
    print(result.model_dump_json(indent=2))
```

`state`, instructions, and criteria entries accept text or structured JSON
objects/lists. Reuse an open client for repeated requests: an embedded client
owns one model and releases it when the context exits.

The same API is awaitable inside an event loop:

```python
import asyncio

from sakuradite import Noul, Sakuradite


async def main():
    async with Sakuradite("Qwen/Qwen3-0.6B", backend="torch") as client:
        result = await client.decide(
            state="The delivery is late.",
            questions={"late": Noul(instructions="Is the delivery late?")},
        )
        print(result.nouls["late"].noul)


asyncio.run(main())
```

Use `with` in synchronous code and `async with` in asynchronous code. If you
manage the lifecycle manually, call `close()` or `await close()` respectively.
A closed client cannot be reopened. Closing an HTTP client does not unload
models from the daemon.

For an image/text checkpoint on Apple Silicon:

```python
from pathlib import Path

from sakuradite import Noul, Sakuradite

with Sakuradite("LiquidAI/LFM2-VL-450M", backend="mlx") as client:
    result = client.decide(
        state="Inspect the supplied photograph.",
        images=[Path("photo.png")],
        questions={"red": Noul(instructions="Is a red object visible?")},
    )
    print(result.nouls["red"].noul)
```

## Daemon and CLI

Start the native Uvicorn server in one terminal:

```sh
sakuradite serve --host 127.0.0.1 --port 7777
```

In another terminal, inspect, download, and load a model:

```sh
sakuradite inspect Qwen/Qwen3-0.6B
sakuradite pull Qwen/Qwen3-0.6B
sakuradite load Qwen/Qwen3-0.6B --backend torch
sakuradite list
sakuradite ps
```

`inspect` reads metadata without downloading weights. `pull` downloads without
loading; `load` downloads if necessary and retains the model in device memory.
These commands connect to the running daemon. Use the global `--base-url`
option before the command when it runs elsewhere.

Save this question map as `questions.json`:

```json
{
  "late": {
    "type": "noul",
    "instructions": "Is the delivery late?"
  }
}
```

```sh
sakuradite run Qwen/Qwen3-0.6B --backend torch \
  --state "The delivery is late." --questions questions.json
sakuradite unload Qwen/Qwen3-0.6B
sakuradite rm Qwen/Qwen3-0.6B
```

`run` loads the model if necessary and prints a JSON decision result. Use
`--state-file` for UTF-8 text or JSON, or pipe state through standard input.
`unload` releases device memory; `rm` deletes an unloaded managed checkpoint.
Bare `sakuradite` starts an interactive shell in a terminal. Commands are also
available through `python -m sakuradite`; use `--help` for options.

## HTTP client

Load the model on the daemon first, then use the same decision API:

```python
from sakuradite import Noul, Sakuradite

with Sakuradite(
    "Qwen/Qwen3-0.6B",
    base_url="http://127.0.0.1:7777",
) as client:
    result = client.decide(
        state="The delivery is late.",
        questions={"late": Noul(instructions="Is the delivery late?")},
    )
    print(result.model_dump_json(indent=2))
```

Native HTTP requests use `POST /v1/decisions`:

```sh
curl --fail-with-body http://127.0.0.1:7777/v1/decisions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","state":"The delivery is late.","questions":{"late":{"type":"noul","instructions":"Is the delivery late?"}}}'
```

`GET /v1/models` lists ready models. `/healthz` reports liveness; `/readyz`
returns 503 until a model is ready. `/docs` serves the generated API reference;
`/openapi.json` exposes its OpenAPI 3.1 contract. `/v1/systemone` accepts the text-only TypeSafe System
One request contract; native image input uses `/v1/decisions`.

Non-loopback serving requires `SAKURADITE_API_KEY`; clients send it as a
bearer token or set `api_key=`. Administrative endpoints use a separate
`SAKURADITE_ADMIN_KEY`. On POSIX, by default, the daemon generates an owner-only
`admin.key` in its home directory, read automatically by local clients using
the default loopback URL and the same home. Other URLs need an explicit
administration key. Use HTTPS through a trusted reverse proxy for remote use.
Windows daemons require an explicit `SAKURADITE_ADMIN_KEY`; generate a strong
random value and set it in the daemon and administrative clients. This remains
separate from `SAKURADITE_API_KEY`.

Model load/download operations return an operation ID; poll with
`client.models.wait(operation)` or `GET /admin/v1/operations/{operation_id}`.
The daemon retains the latest 256 completed operations plus all pending/running
operations. Older operation IDs return 404; loaded models remain resident.
Persist results you need for auditing. Failed operations raise `APIError` with
the corresponding error code and status, rather than always reporting 503.
Model downloads requiring approved Hugging Face access use `HF_TOKEN` or
`hf auth login` on the **daemon host**, not the inference or administration key.

## API reference

With the default loopback daemon and no inference key configured, open
**http://127.0.0.1:7777/docs**. The reference is generated from the same route
declarations and Pydantic schemas used by the daemon, with grouped endpoints,
request/response fields, text and image examples, all three answer primitives,
status codes, model-operation polling, and SSE event examples. Expand a response
or example and follow schema links for field descriptions and constraints.

The page is read-only semantic HTML: no JavaScript, external CDN, frontend build,
or network assets are required. It works offline when saved. Installing the
Python package includes the library, CLI, and daemon, **not a desktop app or
Node.js toolchain**. Clients can consume `/openapi.json` directly; operation IDs
are stable and the relative server URL supports reverse-proxy path prefixes.

When `SAKURADITE_API_KEY` is configured, **both documentation endpoints require
the inference bearer token**, just like `/v1/*`. The separate admin token only
authorizes `/admin/*`; viewing documentation does not grant administrative access.
A plain browser visit cannot set the bearer header. Fetch and save the page with
an authorized HTTP client instead:

```sh
curl --fail-with-body http://127.0.0.1:7777/docs \
  -H "Authorization: Bearer $SAKURADITE_API_KEY" -o sakuradite-api.html
curl --fail-with-body http://127.0.0.1:7777/openapi.json \
  -H "Authorization: Bearer $SAKURADITE_API_KEY" -o openapi.json
```

Save both files in the same directory to retain the raw-JSON link when reading
offline. Never put tokens in URLs. Without an inference key, access remains
restricted to loopback clients on a loopback-bound daemon. `/healthz` and
`/readyz` remain unauthenticated; no cross-origin browser access is enabled.

## Streaming decisions

Pass `stream=True` to receive each completed answer before the full request
finishes. This is **answer streaming, not token generation**: the engine scores
candidates in parallel, and answers become available as native batches finish.
Output-token usage remains zero.

```python
from sakuradite import Noul, Sakuradite

with Sakuradite("Qwen/Qwen3-0.6B", base_url="http://127.0.0.1:7777") as client:
    with client.decide(
        state="The delivery is late and the package is damaged.",
        questions={
            "late": Noul(instructions="Is the delivery late?"),
            "damaged": Noul(instructions="Is the package damaged?"),
        },
        stream=True,
    ) as stream:
        for event in stream:
            if event.type == "answer":
                print(event.key, event.answer)
        print(stream.result.model_dump_json(indent=2))
```

For asynchronous code, use `async with await client.decide(..., stream=True)`
and `async for event in stream`. The same stream API works with an embedded
model. A stream supports one consumer. Exhausting it closes its transport
automatically; use its context manager when stopping early. Closing the client
also closes its active streams. `stream.result` is available only after the
`complete` event.

Over HTTP, send `"stream": true` to `POST /v1/decisions` and use a client that
does not buffer the response:

```sh
curl --no-buffer --fail-with-body http://127.0.0.1:7777/v1/decisions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","state":"The delivery is late.","questions":{"late":{"type":"noul"}},"stream":true}'
```

The response is `text/event-stream`. Each SSE frame has an `event` name and
one JSON `data` payload whose `type` matches that name:

- `answer`: `model`, question `key`, typed `answer`, and that question's `usage`.
- `complete`: the complete `result`, identical in shape to a non-streaming response.
- `error`: an `error` with `code` and `message`, plus `status_code`; terminal.

Failures before the first answer retain their ordinary HTTP status and JSON
error body. Failures after streaming starts use an `error` event; the Python
SDK raises `APIError` with its code and status. A truncated or inconsistent
stream raises `TransportError`, never a successful partial result. Streams
are not automatically retried or resumable. Input-token usage counts logical
question inputs, not the physical tokens saved by cache reuse.

Disable response buffering at the reverse proxy and allow the configured
model deadline for upstream reads. The daemon sends `X-Accel-Buffering: no`,
bounds pending events, cancels disconnected requests, and limits blocked
stream writes to 30 seconds by default (`ServerConfig.stream_write_timeout`).
Native work already running drains before its model or request cache is released.
The TypeSafe-compatible `/v1/systemone` endpoint remains non-streaming.

## Runtime limits and configuration

- PyTorch uses native Transformers loaders for text and compatible image/text
  checkpoints, on CPU, CUDA, or Apple MPS. MLX supports text on Apple Silicon
  and native SmolVLM/Idefics3 and LFM2-VL image/text checkpoints through MLX-VLM.
- Support depends on the checkpoint's native backend registry, fast tokenizer,
  chat template, and model processor. Custom repository code is not executed.
  Registry membership alone does not prove a checkpoint works; there is no
  claim of support for every architecture or quantization format.
- With only one runtime installed, select `backend="torch"` or `backend="mlx"`
  explicitly. Automatic selection prefers MLX for eligible text and image checkpoints
  on Apple Silicon and otherwise selects PyTorch.
- For compatible vision checkpoints, pass `images=[Path("image.png")]`
  to `decide()` after importing `Path` from `pathlib`, or repeat `--image` in the
  CLI. Up to eight JPEG/PNG images are accepted as `Path` objects, bytes, or
  base64 data URLs. Remote image URLs are not fetched.
- MLX image inference has been checked against native full forwards with
  `HuggingFaceTB/SmolVLM-256M-Instruct` and `LiquidAI/LFM2-VL-450M`, including
  multiple images and image-sensitive probabilities. Other MLX vision families
  are rejected explicitly; select PyTorch for those compatible checkpoints.
- Qwen3.5 is currently disabled in MLX-LM because release 0.31.3 has a known
  recurrent q/k normalization defect. Use the Torch backend instead; registry
  availability is not treated as proof of correct native math.
- `revision=` pins a Hub revision; an immutable commit is best for reproducibility.
  `offline=True` or CLI `--offline` requires locally available model assets.
  Device and dtype can be selected when opening or loading a model.

Runtime data, downloads, and the model catalog live in `~/.sakuradite` by
default. Override this with `SAKURADITE_HOME`, Python `home=Path(...)`, or the
global CLI `--home` option. The SDK reads defaults from `config.toml` there;
explicit constructor values override those defaults. `SAKURADITE_MODEL` and
`SAKURADITE_BASE_URL` can select a model and HTTP endpoint without hardcoding
them; unset `SAKURADITE_BASE_URL` when using embedded inference.

On CPU-limited GPU servers, prevent image preprocessing from creating more CPU
threads than the container can run. Set thread limits before starting Python:

```sh
OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 sakuradite serve
```

This limits CPU preprocessing threads, not CUDA parallelism. CPU-only inference
can benefit from more threads. Latency depends on the model, input length, images,
question count, concurrent callers, and available hardware; loading and the first
request also incur initialization costs.

Request compilation, image preparation and inference share one model-owning
worker. This keeps mutable native processor state serialized without competing
preparation threads. Admission, deadlines and scoped cleanup cover the complete
request, including compilation.

Questions and concurrent requests use bounded native batches. PyTorch selects
only the needed output positions when the native model supports indexed logits,
without replacing its vocabulary head, scaling, or soft-capping. If a tokenizer
has no padding token, its existing end-of-sequence token is used for masked right
padding; no vocabulary entries or model weights are added. Eligible native
attention models share an identical text prefix across a request's native
batches when enough duplicate computation can be removed. Eligibility depends on the actual cache,
attention, rotary-position and quantization configuration, not the model's brand.
Validated hybrid cache layouts include Qwen3.5 text, LFM2 and Nemotron-H. The
Qwen3.5 multimodal wrapper can share prefixes for text-only requests; image
prefixes are never shared. BF16 linear-attention/state-space hybrids keep unsplit
native prefill because splitting can amplify low-precision rounding.
Qwen3.5-family and Nemotron Nano shared-prefix measurements explicitly use
`dtype="float16"`. LongRoPE reuse requires the prefix and complete inputs to stay
in the same frequency regime. Eligibility uses all matching questions in the
request, not just the first batch. Immutable request-owned seeds are cloned for
each branch and released after all native rows finish, even for slow stream
consumers. They are never shared between requests. Retained seeds require
additional memory; batch-token budgeting still uses complete input lengths.
Other recurrent layouts, quantized caches, shared-KV layers and dynamically
changing rotary frequencies do not use this optimization; native batching and
supported output-position selection still apply.
Native batch shapes, precision and fused kernels can change floating-point
results; probabilities are not promised to be bitwise invariant across them.
Repeated images are decoded once per request. Verified Idefics3/SmolVLM and
LFM2-VL processors also reuse native resized/normalized pixels and image-token
expansions for questions sharing the same ordered images. Homogeneous Torch rows
retain native padding and independent output storage; heterogeneous image batches
still use the complete native processor. Image data is never cached across requests
and is released when the owning native work drains.

MLX text scoring uses each architecture's native prompt cache to prefill long inputs and
evaluate only the final token's output head. Short prompts stay on a single
forward to avoid extra dispatch overhead. This preserves native tied or
quantized embeddings, recurrent state, head scaling and soft-capping.
Compatible float32 MLX attention models additionally use native variable-length
batches and request-wide prefix forks. The new optimizations are disabled for
low-precision weights after measured probability drift; this is not a reason to
assume float32 is faster overall. Recurrent/rotating caches keep their native
execution path. MLX vision currently processes unpadded question rows
sequentially and reuses native image features within a request; it does not
claim general batched multimodal execution. Compatible float32 vision inputs
of at least 512 processed tokens use native language-cache prefill and evaluate
only the final output-head position. BF16/FP16 vision keeps the unsplit forward.
Each row still receives its own token positions and masks while request-owned
image preprocessing and vision features are reused.

An upstream registry entry is not a pretrained-checkpoint test. In particular,
MoE "active parameter" counts are not weight-memory requirements: all experts
must fit the selected native runtime's placement and quantization constraints.
Smaller checkpoints are the practical starting point for local latency tuning.

### CUDA runtime tuning

The validated Linux CUDA stack uses PyTorch 2.10.0 with CUDA 12.8 and
Transformers 5.17.0. Install the CUDA wheels before the runtime extra:

```sh
python -m pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0 \
  --index-url https://download.pytorch.org/whl/cu128
python -m pip install '.[torch,server]' 'transformers==5.17.0' \
  'kernels==0.16.2' 'fla-core==0.5.2'
```

Native Mamba checkpoints additionally benefit from `mamba-ssm==2.3.2.post1`
and `causal-conv1d==1.7.0`. Their compiled wheels must match Python, PyTorch,
CUDA and the C++ ABI; follow the upstream projects' installation instructions.
These CUDA extensions are not installed by the base package or the Torch extra.
Missing native kernels can substantially increase latency.

The H200 measurements use a 32,768-token native batch budget. The default is
8,192 to suit smaller devices. Set a larger budget explicitly when loading a
daemon model, rather than assuming the measured hardware configuration is the
default:

```python
from sakuradite import Sakuradite
from sakuradite.config import ModelConfig

with Sakuradite(base_url="http://127.0.0.1:7777") as client:
    operation = client.models.load(
        ModelConfig(
            model="LiquidAI/LFM2.5-2.6B",
            backend="torch",
            device="cuda",
            max_batch_tokens=32768,
        )
    )
    client.models.wait(operation)
```

A larger batch budget increases potential memory consumption and is not always
faster. Model loading and kernel initialization are separate from warm-request
latency; concurrent callers also include queueing time.

Tune the row limit alongside the token budget. For Qwen2.5-0.5B-Instruct in FP16
on an RTX 4090, `max_batch_size=64` with `max_batch_tokens=32768` gave the best
tested mixed-workload completion time; 128 rows with 65,536 tokens was slower.
This is workload-specific guidance, not a recommendation for every checkpoint.
Measure individual request latency as well as whole-batch completion: larger
batches can keep short requests waiting. Defaults remain 16 rows and 8,192 tokens.

Model weights are downloaded separately and remain subject to each model's
license and access conditions.

## License

Sakuradite is licensed under Apache-2.0; see [LICENSE](LICENSE). This does not
change the licenses or access requirements of any model weights.
