Metadata-Version: 2.4
Name: assay-sdk
Version: 0.3.0
Summary: Python SDK for the Assay LLM tracing and evaluation platform
Keywords: evals,llm,observability,opentelemetry,tracing
License-Expression: Apache-2.0
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Typing :: Typed
Requires-Dist: httpx==0.28.1
Requires-Dist: opentelemetry-api==1.44.0
Requires-Dist: opentelemetry-sdk==1.44.0
Requires-Python: >=3.10
Project-URL: Documentation, https://github.com/marioweid/assay#readme
Project-URL: Issues, https://github.com/marioweid/assay/issues
Project-URL: Repository, https://github.com/marioweid/assay
Description-Content-Type: text/markdown

# assay-sdk

The Python package for [Assay](https://github.com/marioweid/assay), a self-hosted LLM
tracing and evaluation platform.

```bash
uv add assay-sdk
```

The distribution is named `assay-sdk` and imported as `assay`.

## Tracing

Content capture is off by default. Enable it explicitly when the traced input and output may be
sent to Assay:

```python
import assay

assay.init(
    endpoint="http://localhost:8080",
    api_key="asy_...",
    application="support-bot",
    capture=True,
)


@assay.trace
def answer(question: str) -> str:
    return "Assay evaluates AI systems."
```

Use `assay.span(...)` for explicit spans and call `set_input`, `set_output`, `set_context`, or
`set_reference` on the active span. Call `assay.flush()` before a short-lived process exits and
`assay.shutdown()` when the tracing lifecycle ends.

## API Client

```python
import assay

with assay.Client("http://localhost:8080", admin_token="...") as client:
    applications = client.applications.list()
```

Management, dataset, scorer, and run operations use an admin token. Trace inspection and scoring
use a project API key.

## CLI

Set `ASSAY_ENDPOINT` and the relevant credential, then use the management and evaluation commands:

```bash
assay projects list
assay apps list --project PROJECT_ID
assay datasets import APPLICATION_ID --file regression.jsonl
assay run create APPLICATION_ID --dataset DATASET_ID --scorers groundedness,correctness
assay run watch RUN_ID --gate groundedness:0.8
assay traces score --scorer correctness TRACE_ID
assay scores export APPLICATION_ID --failed --format jsonl
assay datasets from-trace DATASET_ID TRACE_ID --scorer groundedness
assay metrics APPLICATION_ID --scorer groundedness
```

Commands emit JSON. `assay run watch` returns exit code 1 when a run fails or a gate is not met,
which makes it suitable for CI checks.

M6 commands are available from this checkout via `uv run assay ...`. Metrics and score export
require admin authentication and default to 30 days. `--start` and `--end` accept timezone-bearing
timestamps for ranges up to 366 days. Trace imports preserve the selected scorer's latest evidence
and reject duplicate trace/scorer pairs without overwriting existing items.

The optional `tests/test_live_workflow.py` acceptance test requires `ASSAY_LIVE_TEST_ENDPOINT`
and `ASSAY_ADMIN_TOKEN`. It makes real judge calls using synthetic data and deletes its project.
