Metadata-Version: 2.5
Name: tokentriage
Version: 1.0.0
Summary: Route your OpenAI client through your assigned TokenTriage gateway and read request metadata, with typed customer, management and jobs clients, tracing and optional local routing. Zero runtime dependencies.
Project-URL: Homepage, https://tokentriage.com/docs/sdks/python/
Project-URL: Documentation, https://tokentriage.com/docs/sdks/overview/
Project-URL: Changelog, https://tokentriage.com/docs/sdks/python/
Project-URL: Support, https://tokentriage.com/contact/
Project-URL: Status, https://tokentriage.com/status/
Author: TokenTriage
License-Expression: Apache-2.0 AND LicenseRef-TokenTriage-Commercial
License-File: LICENSE
License-File: LICENSE-ENGINE
License-File: NOTICE
Keywords: instrumentation,llm,observability,proxy,tokentriage,tracing
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: crewai
Requires-Dist: crewai>=0.30; extra == 'crewai'
Provides-Extra: dev
Requires-Dist: build>=1.0; extra == 'dev'
Requires-Dist: cryptography>=42; extra == 'dev'
Requires-Dist: hypothesis>=6; extra == 'dev'
Requires-Dist: mypy>=1.0; extra == 'dev'
Requires-Dist: opentelemetry-api>=1.0; extra == 'dev'
Requires-Dist: opentelemetry-sdk>=1.0; extra == 'dev'
Requires-Dist: pip-audit>=2.10; extra == 'dev'
Requires-Dist: pyright>=1.1; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: pyyaml>=6; extra == 'dev'
Requires-Dist: ruff>=0.16; extra == 'dev'
Requires-Dist: tomli>=2; (python_version < '3.11') and extra == 'dev'
Requires-Dist: wasmtime>=20; extra == 'dev'
Provides-Extra: httpx
Requires-Dist: httpx>=0.27; extra == 'httpx'
Provides-Extra: langchain
Requires-Dist: langchain-core>=1.0; extra == 'langchain'
Requires-Dist: langchain-openai>=1.0; extra == 'langchain'
Provides-Extra: management
Requires-Dist: httpx<1,>=0.28; extra == 'management'
Provides-Extra: openai
Requires-Dist: httpx<1,>=0.28; extra == 'openai'
Requires-Dist: openai<4,>=3.14.1; extra == 'openai'
Provides-Extra: otel
Requires-Dist: opentelemetry-api>=1.0; extra == 'otel'
Provides-Extra: requests
Requires-Dist: requests>=2.31; extra == 'requests'
Provides-Extra: router
Requires-Dist: httpx>=0.27; extra == 'router'
Requires-Dist: wasmtime>=20; extra == 'router'
Description-Content-Type: text/markdown

# TokenTriage for Python

Route your OpenAI client through your assigned TokenTriage gateway and read the
request ID from its response. Your provider client keeps its usual response,
stream and error types.

## Get the four values

Your backend needs four values. Get them one of two ways.

**Set them up yourself in the portal** (you need permission to manage provider
connections and keys):

1. Open your application, then its gateway, then the gateway's **Connect** tab.
   Pick the environment at the top of the page (for example Development).
2. **Connect a provider.** Save your OpenAI or Anthropic key as a provider
   connection and finish gateway setup. The page shows when the gateway can
   serve requests; until then it shows no request example.
3. **Get your runtime key.** Create a runtime key for this gateway and
   environment. It starts with `ttv_` and is shown once: copy it now. Lost keys
   are rotated or replaced on **Manage runtime keys**.
4. **Send your first request** shows the gateway endpoint: that is the gateway
   base URL (not the portal's address). **Connection details** lists the
   Gateway ID (`gw_…`) and Environment ID (`env_…`).

If you already use the `tokentriage` CLI and are signed in, `tokentriage cloud
credential create` can mint the runtime key instead of step 3; it is optional
and needs the CLI installed. See the [CLI reference](https://tokentriage.com/docs/reference/cli/).

**Or ask your workspace admin** for the gateway base URL, a runtime key for your
gateway and environment, the gateway ID and the environment ID.

## Install and run your first request

You need Python 3.10 or newer. In a new directory:

```bash
python -m venv .venv && . .venv/bin/activate
pip install "tokentriage[openai]"
export TOKENTRIAGE_GATEWAY_BASE='https://…'   # the gateway endpoint
export TOKENTRIAGE_RUNTIME_KEY='ttv_…'
export TOKENTRIAGE_GATEWAY_ID='gw_…'
export TOKENTRIAGE_ENVIRONMENT_ID='env_…'
```

Save the example below as `first_request.py`, then run `python first_request.py`.

<!-- sdk-example:first_request.py -->
```python
import os
import sys

from openai import APIError
from tokentriage.client import TokenTriage, TokenTriageError, error_details

NAMES = ("TOKENTRIAGE_GATEWAY_BASE", "TOKENTRIAGE_RUNTIME_KEY",
         "TOKENTRIAGE_GATEWAY_ID", "TOKENTRIAGE_ENVIRONMENT_ID")
missing = [name for name in NAMES if not os.environ.get(name)]
if missing:
    sys.exit(f"Set {', '.join(missing)} before running this example.")
base, runtime_key, gateway_id, environment_id = (os.environ[name] for name in NAMES)

try:
    with TokenTriage(base_url=base, api_key=runtime_key) as connection:
        client = connection.openai(gateway_id=gateway_id, environment_id=environment_id)
        raw = client.chat.completions.with_raw_response.create(
            model="gateway",
            messages=[{"role": "user", "content": "Say hello in one sentence."}],
            max_tokens=128,
        )
        answer = raw.parse().choices[0].message.content
except (TokenTriageError, APIError) as error:
    # A configuration error is raised locally and never reaches the gateway.
    details = error_details(error)
    sys.exit(f"Request failed: category={details.category} code={details.code} "
             f"status={details.status} request_id={details.request_id}")
except Exception as error:
    sys.exit(f"Request failed: {error!r}")

request_id = raw.headers.get("x-tt-request-id")
model = raw.headers.get("x-tt-model")
print(answer)
print("Request ID:", request_id)
print("Model:", model)
if model in (None, "gateway"):
    print("Warning: x-tt-model is missing or still 'gateway'; check the gateway's applied release.",
          file=sys.stderr)
if request_id is None:
    sys.exit("No x-tt-request-id was returned; the response did not come from a TokenTriage gateway.")

# A successful run prints, for example:
#   Hello! How can I help you today?
#   Request ID: req_...
#   Model: <provider model chosen by the applied release>
```
<!-- sdk-example:end -->

`model="gateway"` lets the gateway choose the provider model from its deployed
configuration (its *release*); `x-tt-model` reports the choice. The runtime key
is the only secret your code holds: with a saved provider connection you do not
set `OPENAI_API_KEY`. The runtime key only sends model requests; a management
key (`ttm_…`) is for the customer and management APIs. Keep both on your
server, never in a browser.

The SDK reads `TOKENTRIAGE_GATEWAY_BASE` and `TOKENTRIAGE_RUNTIME_KEY` (or the
`base_url` and `api_key` arguments) and `TOKENTRIAGE_MODE` (`managed`, the
default, or `self_hosted` for your own gateway). It never reads the gateway or
environment ID from the environment: the example reads them and passes them in.
Set the two variables together: if only one is set the constructor refuses
(`gateway_base_missing` or `runtime_key_missing`) unless an argument supplies
the missing value. Explicit arguments win over the environment. The older
`TOKENTRIAGE_BASE_URL` / `TOKENTRIAGE_API_KEY` still work, but are read only when
neither of the two new variables is set (an empty one counts as unset).

The `openai` extra installs `openai>=3.14.1,<4` and `httpx`; the core has no
runtime dependencies.

## What works at launch

| Request | Saved provider connection (OpenAI, Anthropic) | Managed tier |
|---|---|---|
| Chat Completions | Yes | `gpt-4o-mini`, `gpt-4o`, haiku |
| Chat Completions, streamed | Yes | `gpt-4o-mini`, `gpt-4o`; haiku answers 503 `enforcement_unavailable` |
| Responses | Not at launch | responses-mini, not streamed (a streamed request answers 503 `enforcement_unavailable`) |
| Embeddings | Not at launch | embeddings-small (streaming does not apply) |

Every row goes through the OpenAI client, sync (`connection.openai(...)`) or
async (`connection.async_openai(...)`, an `AsyncOpenAI`), including a saved
Anthropic connection: the gateway talks to Anthropic. The native Anthropic
Messages client, Gemini, batches, files and fine-tuning are not supported at
launch.

```python
async with TokenTriage() as connection:  # reads the two variables
    client = connection.async_openai(gateway_id=gateway_id, environment_id=environment_id)
    answer = await client.chat.completions.create(
        model="gateway", messages=[{"role": "user", "content": "Hello"}], max_tokens=128)
```

## When a request fails

`connection.error_details(error)` reads any failure without replacing it and
returns `code`, `status`, `category`, `request_id`, `provider_request_id`,
`retryable`, `safe_to_replay` and `retry_after_s`.

| Where it failed | How you see it | Codes and fixes |
|---|---|---|
| In the SDK, before anything is sent | `tokentriage.client.ConfigurationError` (category `configuration`, no status) from the constructor or `openai()` | `gateway_base_missing`, `runtime_key_missing`: set both variables. `mode_invalid`: `TOKENTRIAGE_MODE` is `managed` or `self_hosted`, never empty. `managed_base_insecure`: use the `https` endpoint. `provider_key_not_allowed`: remove the `api_key` argument. `configuration_error`: no gateway base or key found. |
| In the SDK, while sending | `ConfigurationError`, or `openai.APIConnectionError` whose `__cause__` is one | `destination_not_allowed`: the request was redirected away from the gateway base. `client_closed`: the connection was closed. |
| At the gateway | `openai.APIStatusError` with the gateway's status and `code` | `provider_credential_not_allowed`, `ambiguous_credential`, `gateway_required`, `gateway_not_found`, `environment_not_allowed`, `model_override_unsupported`; 401 `account_key_absent`, `account_key_inactive` or `account_key_unknown` (the runtime key is missing, revoked or not recognised: check `TOKENTRIAGE_RUNTIME_KEY` or the runtime key you passed, and issue a new runtime key in the portal if it was revoked); 503 `release_unavailable` (no deployed release to serve: check the gateway's deployment status) or `enforcement_unavailable` |
| At the provider | `openai.APIStatusError` passed through unchanged | the provider's own status; `code` is the provider's code or `provider_error` |

Fixes for the gateway codes: [read the refusal you got
instead](https://tokentriage.com/docs/sdks/first-request/#4-read-the-refusal-you-got-instead).
The exception class says which layer raised it; `code` says why.

## Retries and billing

Provider clients are built with `max_retries=0` and the SDK has no retry loop.
A retried inference request is a new billable attempt, so raising `max_retries`
is an explicit choice to accept those charges. A timeout or dropped connection
leaves the outcome unknown: the request may have completed and been billed.

`retryable` means a retry might succeed, not that it is free. It follows the
gateway's own `retryable` flag when the error body carries one. Otherwise it is
`False` when the body says `safe_to_replay: false`, `False` for
`enforcement_unavailable` unless the server sent `Retry-After` (or
`retry_after`), and `True` for 408, 429, 500, 502, 503 and 504.
`safe_to_replay` is `True` only when the body says so. Never replay a partly
delivered stream.

## Timeouts, cancellation and closing

- **Timeouts.** The SDK sets none, so the `openai` package's default applies.
  Pass `timeout=` in seconds to `connection.openai(...)` or per request.
- **Cancellation.** Cancel the asyncio task that awaits an async request or
  stream; `error_details` reports category `cancelled`. A sync request stops
  only at its timeout; close a sync stream with `stream.close()`. Cancelling
  locally does not tell you whether the gateway served, or billed, the request.
- **Closing.** `connection.close()` (or leaving `with`) closes the HTTP clients
  the SDK created for sync clients and refuses later requests with
  `client_closed`. Use `await connection.aclose()` (or `async with`) after async
  use. An `http_client` you passed in stays yours to close.

## Request IDs

On success, read `x-tt-request-id` from the raw response's `headers` (or
`connection.metadata(response).request_id`); `x-tt-model`, `x-tt-release`,
`x-tt-gateway` and `x-tt-environment` come with it. On failure, use
`error_details(error).request_id`. It is `None` when no gateway response
arrived, and a stream that fails part-way may raise an error without it, so
keep the ID from the stream's first response. With an ID, find the request on
the gateway's Requests tab in the portal; without one, look there by time. A
request ID is for diagnosis; it does not deduplicate a retry.

## What else is in the package

| You want to | Use |
|---|---|
| Keep your own OpenAI constructor or `httpx` client | `OpenAI(**connection.openai_options(gateway_id=..., environment_id=...))`, or `async_openai_options()` |
| Label routed requests | `tokentriage.trace()`, `step()` and `headers()` |
| Manage credentials, gateways and jobs, or read spend, requests and comparisons | [The management and customer clients](https://tokentriage.com/docs/sdks/management/) |
| Record a comparison review | `customer.comparisons.review(...)`; `note` takes `bytes` (a `str` is refused). See the [evaluations reference](https://tokentriage.com/docs/reference/managed-evaluations/) |
| Route in your process | `tokentriage.router` (`router` extra); see the [Python SDK reference](https://tokentriage.com/docs/sdks/python/) and, for required guardrails, [guardrails](https://tokentriage.com/docs/guardrails/overview/) |
| Observe requests, use native chat, or add OpenTelemetry spans | [Request controls and observations](https://tokentriage.com/docs/sdks/production/) |

Provider, HTTP, local router, framework and OpenTelemetry integrations use
optional extras. Existing clients whose transport the SDK cannot see cannot be
wrapped safely; build them with `openai_options()`.

## Upgrading from 1.0.0rc1

rc1 code keeps working with the new package. Moving your traffic to hosted
saved mode is a separate step: get the four values above, replace your
hand-built client with `connection.openai(gateway_id=..., environment_id=...)`,
send `model="gateway"`, remove the provider key and any `X-Tokentriage-Key`
header, and check that a request prints a request ID. The 1.0.0 entry in the
package's CHANGELOG lists every change.

## Documentation and support

- Documentation: <https://tokentriage.com/docs/sdks/python/> (walkthrough:
  [first request](https://tokentriage.com/docs/sdks/first-request/); async and
  streaming: [provider clients](https://tokentriage.com/docs/sdks/adapters/))
- Support: <https://tokentriage.com/contact/>; status: <https://tokentriage.com/status/>

The source repository is private; report problems through support.

The SDK code is Apache-2.0; see `LICENSE`. The bundled decision engine
`enginecore.wasm` is proprietary InferOps LLC software under `LICENSE-ENGINE`.
See `NOTICE`.
