Metadata-Version: 2.4
Name: callosum-sdk
Version: 0.2.3
Summary: AsyncOpenAI-compatible client for Callosum spot inference
Project-URL: Homepage, https://github.com/callosumtechnologies/callosum-sdk
Project-URL: Repository, https://github.com/callosumtechnologies/callosum-sdk
Project-URL: Issues, https://github.com/callosumtechnologies/callosum-sdk/issues
Author: Callosum Technologies
License: Apache-2.0
License-File: LICENSE
Keywords: batch,callosum,inference,openai
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Requires-Dist: httpx>=0.27
Requires-Dist: openai>=2.53.0
Provides-Extra: dev
Requires-Dist: pyright>=1.1.403; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest-cov>=6; extra == 'dev'
Requires-Dist: pytest>=8.3; extra == 'dev'
Requires-Dist: ruff>=0.12; extra == 'dev'
Description-Content-Type: text/markdown

# callosum-sdk

callosum-sdk provides an AsyncOpenAI-compatible client for latency-insensitive
models running on spot capacity. A familiar chat-completions call becomes a
background submit/poll operation without holding an HTTP connection across a
GPU cold start.

## Install

~~~bash
pip install callosum-sdk
~~~

## Quickstart

~~~python
import asyncio
import os

from callosum_sdk import AsyncOpenAI


async def main() -> None:
    async with AsyncOpenAI(
        api_key=os.environ["CALLOSUM_API_KEY"],
        base_url=os.environ.get("CALLOSUM_BASE_URL", "https://api.callosum.ai/v1"),
    ) as client:
        response = await client.chat.completions.create(
            model="qwen3.5-4b",
            messages=[{"role": "user", "content": "What is the corpus callosum?"}],
        )
        print(response.choices[0].message.content)


asyncio.run(main())
~~~

Run `uv run demo-readme` from a source checkout to execute that exact Quickstart
block. Set `CALLOSUM_BASE_URL` to target a staging gateway or port-forward;
otherwise it defaults to the public API.

The SDK sends the public slug as `spot/qwen3.5-4b`, which selects the
gateway's spot proxy. The gateway maps that slug through its deployed model
catalogue and forwards to the spot orchestrator's background API. The returned
response still reports `qwen3.5-4b`, keeping the routing namespace
transparent to application code.

## Available models

Pass the slug alone and it is served on spot capacity; the SDK adds the
namespace.

| Slug | Model | Shape |
|---|---|---|
| `qwen3.5-4b` | Qwen3.5-4B | One GPU per replica |
| `deepseek-v4-flash` | DeepSeek-V4-Flash-0731 | 304B mixture-of-experts, four GPUs per replica |

The slug set is the gateway's, not the SDK's: a syntactically valid slug with no
deployed serving profile returns `unknown_model`, so a new model becomes
available without an SDK release.

### Passthrough models

Some models are not ours to schedule: the gateway proxies them to an always-on
upstream and returns the reply on the same connection. Name one by its namespace
and the SDK sends it straight through, so the choice is a model name rather than
a different client.

| Slug | Upstream |
|---|---|
| `cerebras/gpt-oss-120b` | Cerebras |
| `cerebras/zai-glm-4.7` | Cerebras |

~~~python
response = await client.chat.completions.create(
    model="cerebras/gpt-oss-120b",
    messages=[{"role": "user", "content": "What is the corpus callosum?"}],
)
~~~

The rule is the namespace: a slug with no `/` is spot-backed and goes through the
background submit/poll protocol, while a slug carrying one names a passthrough
upstream and goes to the gateway's ordinary chat completions route. That is the
same line the gateway itself draws, so the two cannot disagree.

Because a passthrough answers immediately, it keeps what the background protocol
has to give up — `stream=True` works:

~~~python
stream = await client.chat.completions.create(
    model="cerebras/gpt-oss-120b",
    messages=[{"role": "user", "content": "Explain the corpus callosum."}],
    stream=True,
)
async for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")
~~~

`deepseek-v4-flash` is a reasoning model that answers directly unless asked to
deliberate. Request deliberation per call:

~~~python
response = await client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[{"role": "user", "content": "What is the corpus callosum?"}],
    extra_body={"chat_template_kwargs": {"thinking": True}},
)
print(response.choices[0].message.reasoning_content)
print(response.choices[0].message.content)
~~~

The deliberation arrives as `reasoning_content` on the message, leaving
`content` as the answer alone. It is an additional field rather than part of the
typed OpenAI surface, so it is readable but invisible to type checkers.

## Behaviour and limits

- `chat.completions.create()` is non-streaming: it submits immediately, polls
  every five seconds, and returns a typed OpenAI `ChatCompletion` at completion.
- The default one-hour client wait can be changed with
  `request_timeout_seconds`; the request continues in the orchestrator if a
  local caller is cancelled.
- Polls remain authenticated and tenant-scoped, but do not consume the
  customer's request-per-minute API limit.
- `model_prefix="spot/"` is idempotent. Passing an already namespaced model does
  not produce `spot/spot/...`.
- The API key identifies the tenant. A response id submitted by one tenant
  cannot be polled by another.
- Raw Files and Batch API batching remains available through the explicit
  `BatchOpenAI` class for non-spot workload batches.

See [the spot client guide](docs/spot/README.md) for the routing contract and
[the batching guide](docs/batching/README.md) for `BatchOpenAI`.
