Metadata-Version: 2.5
Name: failmode
Version: 0.0.1
Summary: Instrument LLM agents and ship their traces to failmode, which finds, clusters and regression-tests agent failures.
Project-URL: Homepage, https://github.com/srushti-lohiya/failmode
Project-URL: Repository, https://github.com/srushti-lohiya/failmode
Project-URL: Documentation, https://github.com/srushti-lohiya/failmode/tree/main/sdk#readme
Project-URL: Issues, https://github.com/srushti-lohiya/failmode/issues
Project-URL: Changelog, https://github.com/srushti-lohiya/failmode/releases
Author: Srushti Lohiya
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,evaluation,llm,observability,opentelemetry,tracing
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: opentelemetry-api>=1.27
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.27
Requires-Dist: opentelemetry-proto>=1.27
Requires-Dist: opentelemetry-sdk>=1.27
Requires-Dist: protobuf>=5
Requires-Dist: pydantic>=2.7
Requires-Dist: typer>=0.12
Description-Content-Type: text/markdown

# failmode (Python SDK + CLI)

Instrument an LLM agent in a few lines and send its traces to a
[failmode](https://github.com/srushti-lohiya/failmode) server, which detects failures, groups
them into named failure modes and turns them into regression tests.

> **Status: pre-alpha (0.0.x).** Tracing and upload work; failure detection,
> clustering and the regression-test runner land in later releases. APIs may change.

## Install

> **Requires Python 3.11 or newer.** Check with `python3 --version`. On macOS, the
> `python3` that ships with the system (Xcode Command Line Tools) is 3.9, which is too old.

```bash
pip install failmode
```

No Python 3.11+ yet? [uv](https://docs.astral.sh/uv/) can install one for you:

```bash
uv venv --python 3.12 && source .venv/bin/activate
uv pip install failmode
```

### Troubleshooting

**`ERROR: Could not find a version that satisfies the requirement failmode (from versions: none)`**
This almost always means your Python is older than 3.11. pip hides releases that need a
newer Python, so it looks as if the package doesn't exist. Check `python --version` inside
the environment you're installing into, then create one with Python 3.11+ (see above).
uv reports the same problem more directly: `failmode==… depends on Python>=3.11`.

## Instrument an agent

```python
import failmode

failmode.init(
    project="support-bot",
    endpoint="http://localhost:8000",  # or FAILMODE_ENDPOINT
    api_key="fm_...",  # or FAILMODE_API_KEY
    agent_version="a1b2c3d",  # e.g. your git SHA; used to compare releases
)


@failmode.tool
def lookup_order(order_id: str) -> dict:
    return {"order_id": order_id, "status": "shipped"}


@failmode.retrieval
def search_docs(query: str, k: int = 4) -> list[dict]:
    return [{"id": "faq-12", "text": "Refunds take 5 days.", "score": 0.82}]


@failmode.agent
def run(message: str) -> str:
    docs = search_docs(message)
    with failmode.llm_call(model="claude-haiku", provider="anthropic") as call:
        reply = "..."  # call your model here
        call.record_llm(
            messages=[{"role": "user", "content": message}],
            output=reply,
            input_tokens=812,
            output_tokens=64,
        )
    lookup_order("A-1043")
    return reply


run("Where is my order A-1043?")
failmode.flush()
```

What gets recorded:

| API | Step kind | Captured |
|---|---|---|
| `@failmode.agent` | `agent` | arguments, return value; the outermost call is the trace root |
| `@failmode.tool` | `tool` | tool name, arguments, result, exceptions |
| `@failmode.retrieval` | `retrieval` | query, returned documents (strings, dicts, LangChain-style docs, `(doc, score)` tuples) |
| `failmode.llm_call()` | `llm` | model, provider, messages, output, token usage, cost |
| `failmode.step()` | any | a generic block, for anything else |

Decorators work on sync and async functions. Exceptions are recorded on the step and
re-raised unchanged. Each payload is capped at 64 KB.

### Existing OpenTelemetry setups

`failmode.init()` uses its own `TracerProvider`, so it doesn't replace your OTel
configuration. You can also skip the SDK entirely: the server accepts standard
OTLP/HTTP at `/v1/traces` and understands the OpenTelemetry GenAI semantic conventions
and OpenInference attributes.

### Offline / no server

```python
failmode.init(project="support-bot", jsonl_path="traces.jsonl")
```

Spans are written as OTLP/JSON lines. Send them later:

```bash
failmode upload traces.jsonl --endpoint http://localhost:8000 --api-key fm_...
```

## CLI

```text
failmode version            print the SDK version
failmode ping               check the server is ready
failmode upload FILE        send a JSONL file of traces
```

## License

Apache-2.0
