Metadata-Version: 2.4
Name: agentx-python
Version: 0.8.14
Summary: Official Python SDK for AgentX (https://www.agentx.so/)
Home-page: https://github.com/AgentX-ai/AgentX-python
Author: Robin Wang and AgentX Team
Author-email: contact@agentx.so
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: urllib3>=1.26.11
Requires-Dist: certifi
Requires-Dist: requests
Requires-Dist: pydantic>=2.0.0
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.1.0; extra == "langchain"
Provides-Extra: crewai
Requires-Dist: crewai>=0.80.0; extra == "crewai"
Provides-Extra: openai-agents
Requires-Dist: openai-agents>=0.0.3; extra == "openai-agents"
Provides-Extra: openai
Requires-Dist: openai>=1.0.0; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.25.0; extra == "anthropic"
Provides-Extra: google-adk
Requires-Dist: google-adk>=1.0.0; extra == "google-adk"
Provides-Extra: google-genai
Requires-Dist: google-genai>=1.0.0; extra == "google-genai"
Provides-Extra: litellm
Requires-Dist: litellm>=1.0.0; extra == "litellm"
Provides-Extra: llamaindex
Requires-Dist: llama-index-core>=0.10.0; extra == "llamaindex"
Provides-Extra: autogen
Requires-Dist: autogen-agentchat>=0.4.0; extra == "autogen"
Requires-Dist: autogen-core>=0.4.0; extra == "autogen"
Provides-Extra: databricks
Requires-Dist: mlflow>=3.6.0; extra == "databricks"
Provides-Extra: all
Requires-Dist: langchain-core>=0.1.0; extra == "all"
Requires-Dist: crewai>=0.80.0; extra == "all"
Requires-Dist: openai-agents>=0.0.3; extra == "all"
Requires-Dist: openai>=1.0.0; extra == "all"
Requires-Dist: anthropic>=0.25.0; extra == "all"
Requires-Dist: google-adk>=1.0.0; extra == "all"
Requires-Dist: google-genai>=1.0.0; extra == "all"
Requires-Dist: litellm>=1.0.0; extra == "all"
Requires-Dist: llama-index-core>=0.10.0; extra == "all"
Requires-Dist: autogen-agentchat>=0.4.0; extra == "all"
Requires-Dist: autogen-core>=0.4.0; extra == "all"
Requires-Dist: mlflow>=3.6.0; extra == "all"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license-file
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

![Logo](https://agentx-resources.s3.us-west-1.amazonaws.com/AgentX-logo-387x60.png)

[![PyPI version](https://img.shields.io/pypi/v/agentx-python)](https://pypi.org/project/agentx-python/)
[![Python versions](https://img.shields.io/pypi/pyversions/agentx-python)](https://pypi.org/project/agentx-python/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)

The official Python SDK for **[AgentX](https://www.agentx.so/)** - an evaluation, tracing, and monitoring framework for AI agents, plus a client for AgentX's own hosted agents.

Also see [SDK Developer Docs](https://developers.agentx.so), [API Reference Docs](https://docs.agentx.so/reference)

---

## Contents

- [Why AgentX](#why-agentx)
- [Installation](#installation)
- [Authentication](#authentication)
- [Quick start](#quick-start)
- [Custom agent evaluations](#custom-agent-evaluations) - LLM-as-a-judge, cosine / Jaccard similarity, any framework
- [Production tracing](#production-tracing) - record live agent runs from any framework
- [Monitor](#monitor) - automatic production monitoring, patterns and signals
- [Self-host](#self-host) - run Trace/Evaluate/Monitor on your own machine instead of the hosted dashboard
- [Agents & conversations](#agents--conversations) - chat with and orchestrate AgentX's own hosted agents
- [Links](#links)

---

## Why AgentX

- **Agent Evaluations** - score **any** agent (LangChain, CrewAI, AutoGen, LlamaIndex, OpenAI, Anthropic, HTTP, or plain Python) against a dataset with LLM-as-a-judge ratings plus optional cosine, Jaccard, and BLEU/ROUGE similarity metrics. Configurable judge prompt/model, per-question judge guidelines, smoke testing for phrasing robustness, and a durable multi-judge analysis pass for a qualitative report.
- **Production tracing** - one decorator or context manager records every agent run (input, output, latency, tool calls, token usage, and a real span tree) into your workspace, for any framework.
- **Monitor** - check live traces against detection patterns or a sampled LLM judge, and read back triage-ready signals, no dashboard setup required.
- **Prompt registry** - make AgentX the source of truth for your own agent's prompts: pull a version at runtime, tag eval runs and live traces with it, let a judge propose a rewrite from your worst-rated results.
- **Self-host** - run the whole Trace/Evaluate/Monitor stack locally, bring your own LLM keys, no account required.
- **Bring any LLM** - works across major open and closed-source vendors, for evaluation, tracing, and AgentX's own hosted agents alike.
- **AgentX's own hosted agents** - a simple `Agent → Conversation → Message` mental model, chain-of-thought built in, multi-agent workforces, MCP support, and A2A publishing, for when you want AgentX to run the agent too, not just evaluate/trace/monitor it.

---

## Installation

```bash
pip install --upgrade agentx-python
```

Requires Python 3.9 or newer.

#### Run the self-host governance suite locally

```bash
agentx-trace-eval --dev
```

(See [Self-host](#self-host) below for what this downloads and how to point the SDK at it.)

---

## Quick start

Evaluate your own agent - any framework, or plain Python - against a dataset:

```python
from agentx import AgentX

client = AgentX.from_env()

def my_agent(case):
    return call_my_agent(case.query)  # your agent's own code, any framework

report = (
    client.evaluations
    .run(dataset_id="evds_…", subject={"kind": "custom_agent", "framework": "raw_python"})
    .execute(my_agent)
    .finalize()
    .analyze()
)

print(report.average_rating)   # LLM-graded score, 0-10
print(report.summary)          # AI-generated narrative from .analyze()
```

That's it. The rest of this section covers building the dataset, framework adapters, similarity metrics, and judge configuration.

---

## Custom agent evaluations

Evaluate **any** AI agent - LangChain, CrewAI, AutoGen, LlamaIndex, OpenAI, Anthropic, HTTP endpoints, or plain Python - using AgentX as the scoring and reporting backend. Includes optional **cosine**, **Jaccard**, and **BLEU/ROUGE** similarity metrics alongside LLM-graded ratings.

```python
report = (
    client.evaluations
    .run(dataset_id="evds_…", subject={"kind": "custom_agent", "framework": "raw_python"})
    .execute(my_agent_fn)
    .finalize()
    .analyze()
)

print(report.average_rating)       # LLM-graded score, 0-10
print(report.cosine_similarity)    # embedding cosine, 0-1 (None if not enabled)
print(report.jaccard_similarity)   # token-set overlap, 0-1 (None if not enabled)

print(report.summary)              # AI-generated narrative from .analyze()
print(report.recommendations)      # list of prioritized, actionable fixes
```

`.analyze()` also generates a full qualitative report (strengths, weaknesses, instruction adherence, reasoning quality, and recommendations), running the same durable, multi-judge pipeline as the dashboard's "Analyze" button. `analyze(mode=..., quality_mode=..., judges=[...])` controls how items are scored and by which models. See [AI analysis report](EVALUATIONS.md#ai-analysis-report) in the full guide for the complete field and parameter reference.

Ask a case's question several extra ways each run, LLM-paraphrased server-side, to catch agents that break on phrasing rather than substance, and override the judge's prompt/model per config, see [Smoke testing](EVALUATIONS.md#smoke-testing-phrasing-robustness) and [Configuring the judge](EVALUATIONS.md#configuring-the-judge) in the full guide.

Since AgentX doesn't own your agent's code, `client.evaluations.prompts` lets AgentX become your prompt's _source of truth_ instead - the same problem LangSmith's Prompt Hub and Langfuse's Prompt Management solve. Pull a version at runtime, tag your eval runs (or live traces) with it, and let a judge propose a rewrite from your real worst-rated results - a human always has to approve before it publishes:

```python
prompt = client.evaluations.prompts.get("support-agent-system-prompt")  # or prompt.id
# use prompt.text as your own agent's system prompt

client.evaluations.run(
    dataset_id="evds_…",
    subject={"kind": "custom_agent", "metadata": {"promptName": prompt.name}},
).execute(my_agent_fn)
```

See [Prompt registry](EVALUATIONS.md#prompt-registry) in the full guide, or [self-host's docs](https://docs.agentx.so/improve/prompt-management) for the "Suggest improvement" dashboard flow (self-host only - no hosted-SaaS equivalent yet).

On self-host, a finalized run can also **gate a CI job**: `run.gate(fail_under=7, no_regression=True)` (on the run context `.execute()` returns) checks the run's average rating against an absolute floor and/or the dataset's previous run, prints per-check verdicts into the CI log, and returns a `GateResult` - `sys.exit(gate.exit_code)` blocks the merge on regression. Recorded gates appear in the dashboard's CI Gates tab. See [self-host's CI docs](https://docs.agentx.so/integrations/self-host-ci) for the GitHub Actions recipe.

See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide - dataset builder, framework adapters, similarity metrics, smoke testing, judge configuration, prompt registry, and the complete API reference.

---

## Production tracing

Record live agent runs into your workspace with a single decorator or context manager - no changes to your agent's logic. Traces appear in the **Live Traces** tab and can be evaluated against your test datasets with [`tracer.evaluate_trace()`](TRACING.md#tracerevaluate_trace).

```python
from agentx import AgentX

client = AgentX.from_env()
tracer = client.tracer

@tracer.trace("customer-support-agent", framework="langchain", model="gpt-4o")
def handle_query(query: str) -> str:
    return chain.invoke(query)

# Every call is automatically traced: input, output, latency, tool calls, token usage
handle_query("How do I reset my password?")
tracer.flush(timeout=10)  # ensure delivery before the process exits
```

Prefer full control over what gets captured? Use the context manager instead:

```python
with tracer.trace("rag-agent", framework="langchain") as span:
    span.input = {"query": query, "user_id": user_id}

    kb_result = search_knowledge_base(query)
    span.add_tool_call("search_knowledge_base", input=query, output=kb_result, latency_ms=190)

    span.output = llm.invoke(f"Context: {kb_result}\n\nQuery: {query}")
```

To time a tool call and capture its failures automatically, wrap the execution itself instead of reporting it after the fact:

```python
with tracer.trace_tool_call("search_knowledge_base", input=query) as t:
    t.output = search_knowledge_base(query)
```

An exception escaping the block records the call as failed (`success=False` plus the error text, which is what Monitor's built-in "Tool failure" check and the dashboard's Tool quality column read) and then propagates unchanged.

### Framework integrations

Each integration auto-captures LLM calls, tool calls, and token usage - including prompt-caching
token counts (Anthropic's cache write/read, OpenAI/LiteLLM's cached tokens, Google GenAI's cached
content), reported as their own `cache_read_tokens`/`cache_write_tokens` fields alongside the
regular totals, no extra config needed. Self-host's cost estimate prices these separately from a
regular input token when you've set optional cache rates on that model. Install the matching
extra:

| Framework             | Install                                      | Integration               |
| --------------------- | -------------------------------------------- | ------------------------- |
| LangChain             | `pip install "agentx-python[langchain]"`     | `AgentXCallbackHandler`   |
| CrewAI                | `pip install "agentx-python[crewai]"`        | `AgentXCrewObserver`      |
| OpenAI Agents SDK     | `pip install "agentx-python[openai-agents]"` | `AgentXTracingProcessor`  |
| OpenAI (raw client)   | `pip install "agentx-python[openai]"`        | `patch_openai_client`     |
| Anthropic             | `pip install "agentx-python[anthropic]"`     | `patch_anthropic_client`  |
| Google ADK            | `pip install "agentx-python[google-adk]"`    | `AgentXADKPlugin`         |
| Google GenAI (Gemini) | `pip install "agentx-python[google-genai]"`  | `patch_genai_client`      |
| LiteLLM               | `pip install "agentx-python[litellm]"`       | `AgentXLiteLLMLogger`     |
| LlamaIndex            | `pip install "agentx-python[llamaindex]"`    | `AgentXLlamaIndexHandler` |
| AutoGen               | `pip install "agentx-python[autogen]"`       | `AgentXAutoGenObserver`   |

Two more platforms are covered by **pull importers** rather than in-process hooks, each with its
own CLI: `agentx-moveworks` (Moveworks Data API sync, no extra needed) and `agentx-databricks`
(`pip install "agentx-python[databricks]"`, MLflow/Databricks trace sync).

Or plain Python - wrap any function with `@tracer.trace(...)` and it just works, no framework
required. Tracing is **platform agnostic**: each integration stamps its platform label
automatically, a plain trace auto-detects the one orchestration framework imported in the
process, and `framework="any-name"` labels platforms AgentX has never heard of - the label
drives the dashboard's framework filter and Monitor's Platforms chart. See
[Platform detection](TRACING.md#platform-detection).

Running specialist agents in parallel with a `ThreadPoolExecutor`? Wrap each worker body in `tracer.use_span(span)` so their steps land on the parent trace instead of becoming independent traces - see [TRACING.md](TRACING.md) for the full pattern.

See **[TRACING.md](TRACING.md)** for the complete guide - session grouping, error handling, async support, and the full API reference.

---

## Monitor

Automatic production monitoring: check traces against detection patterns and get back triage-ready signals. A **pattern** is a first-class SDK resource with a real id, just like a `Dataset` or `EvaluationSettings`: create one once, then reference it by id at trace time.

```python
pattern = client.monitor.patterns.builder(
    name="Promises a refund",
    detector_kind="semantic",
    semantic_prompt="The response promises a refund.",
    severity="high",
).publish()

with client.tracer.trace("support-agent", monitor=True, pattern_ids=[pattern.id]) as span:
    span.output = call_llm(query)
```

`monitor=True` checks the trace immediately, no dashboard setup required. `pattern_ids` restricts detection to exactly those patterns; omit it to run the full default sweep instead (built-in checks like empty response, trace error, and latency regression, plus every pattern enabled for the workspace).

This works independently of the dashboard's per-agent monitoring toggle (Governance > Observe > Agents), which still auto-checks every trace from an agent once enabled there, with no code changes needed either way.

Read the resulting alerts back with `client.monitor.signals.list()`/`.get()`, no dashboard required:

```python
for signal in client.monitor.signals.list(severity="high"):
    print(signal.summary, signal.occurrence_count)
```

Per-agent monitoring settings (enable/disable, detection categories, notification channels) are `client.monitor.profile.get()`/`.update()`:

```python
client.monitor.profile.update("agent_123", info_detection_enabled=False)
```

On self-host, coverage mode, sample rate, retention, and the built-in latency threshold are project-level defaults set in the dashboard's Platform Settings; `update()` still accepts them for wire compatibility, but only the per-agent fields above take effect there.

Self-host also has **online evaluators**: a real LLM judge scoring a sample of live traffic continuously, distinct from a pattern's rule-matching. A score below `alert_threshold` raises a signal the same way a failing pattern does, deduped and triage-ready in `client.monitor.signals`.

```python
evaluator = client.monitor.online_evaluators.builder(
    name="Helpfulness",
    evaluation_settings_id=settings.id,
    sample_rate=0.1,
    alert_threshold=5,
).publish()

client.monitor.online_evaluators.ratings(evaluator.id, window="7d")
```

An online evaluator and a dataset run's grading config are two profiles of one **LLM Judge Scorer**, and `client.monitor.judge_scorers` manages both in a single call - one rubric, one id, used offline and online:

```python
scorer = client.monitor.judge_scorers.builder(
    name="Helpfulness",
    acceptance_criteria="Concrete, correct, cites the policy.",
    live=True, sample_rate=0.1, alert_threshold=5,   # the online profile
).publish()

client.evaluations.run(dataset_id=dataset.id, subject={...}, scorer_id=scorer.id)
client.monitor.judge_scorers.ratings(scorer.id, window="7d")
```

The two clients above are that entity's per-profile legacy views; they keep working and share the same ids. See [LLM Judge Scorers](EVALUATIONS.md#llm-judge-scorers---reusable-grading-configs) for the full surface, including which self-host engine builds serve it.

An online evaluator can also judge **whole conversations** instead of single traces: pass `scope="session"` and the engine scores each multi-turn session once it's been idle for `idle_seconds`, re-scoring if the conversation resumes. And to close the loop with reality, two ground-truth streams feed the dashboard's Judge Calibration view (which measures how often AgentX's automated verdicts agree with what actually happened): `client.outcomes.report(...)` for after-the-fact system results (a reopened ticket, a human confirmation), and `client.feedback.report(...)` for end-user votes forwarded from your own app's UI - a "down" raises a "Negative user feedback" signal directly, no sampling or judge call involved. All self-host features.

```python
client.monitor.online_evaluators.builder(
    name="Conversation resolution",
    evaluation_settings_id=settings.id,
    scope="session",     # judge the whole session, not each trace
    idle_seconds=120,    # score once the conversation has been quiet this long
    alert_threshold=5,
).publish()

client.outcomes.report(
    trace_id=trace_id,
    outcome="reopened",
    is_negative=True,
    reason="Customer reopened the ticket within 3 days",
)

client.feedback.report(
    trace_id=trace_id,
    rating="down",                            # "up" or "down"
    comment="It never answered my question",  # optional, the user's own words
    end_user_id=current_user.id,              # optional, opaque to AgentX
)
```

See **[TRACING.md](TRACING.md)** for the complete Monitor guide.

---

## Self-host

Prefer to run Trace/Evaluate/Monitor locally instead of the hosted dashboard - no account, bring your own LLM keys? This SDK ships a launcher for [AgentX-trace-eval](https://github.com/AgentX-ai/AgentX-trace-eval), a separate, portable governance engine:

```bash
agentx-trace-eval --dev
```

The first run downloads the engine (and dashboard) into `~/.agentx/bin` and prints a local API key; every run after that just starts it. Point this SDK at it instead of the hosted API:

```bash
export AGENTX_API_BASE_URL=http://localhost:4700/api/v1
export AGENTX_API_KEY=<printed by agentx-trace-eval on first run>
```

`agentx-trace-eval` isn't this SDK's own code - the engine itself is a separate, compiled binary, downloaded on demand rather than bundled into this package, so installing `agentx-python` doesn't get any heavier for the (much more common) case of just talking to the hosted AgentX API. Each SDK release pins the engine release it was tested against and converges the install to that pin, so upgrading the SDK upgrades the engine too. Two flags to know: `--update` (consumed by this launcher) force-reinstalls the resolved engine release, while `--upgrade` passes through to `agentx-server` and re-downloads the dashboard bundle before serving. See that repo's README for what's included, and the `AGENTX_INSTALL_DIR`/`AGENTX_TRACE_EVAL_VERSION`/`AGENTX_TRACE_EVAL_SKIP_WEB` env vars to control where/what it installs.

---

## Agents & conversations

Beyond evaluation, tracing, and monitoring, this SDK is also a client for AgentX's own hosted agents - build, chat with, and orchestrate them directly.

```python
agent = client.list_agents()[0]
conversation = agent.new_conversation()

# Blocking - returns the full response once it's ready
print(conversation.chat("What can you help me with?"))

# Streaming - yields ChatResponse objects as the model produces them
for chunk in conversation.chat_stream("Hello!"):
    if chunk.text:
        print(chunk.text, end="")
```

Each `ChatResponse` chunk exposes the agent's `text` and, where applicable, its `cot` (chain-of-thought) reasoning, along with any retrieved references and tasks. `agent.list_conversations()` / `conversation.list_messages()` resume history instead of starting fresh.

A **workforce** is a team of agents coordinated by a designated manager, mixing LLM vendors and routing work between specialists:

```python
workforce = client.list_workforces()[0]
conversation = workforce.new_conversation()

for chunk in workforce.chat_stream(conversation.id, "How can you help me with this project?"):
    if chunk.text:
        print(chunk.text, end="")
```

---

## Links

- **Dashboard** - [app.agentx.so](https://app.agentx.so)
- **Website** - [agentx.so](https://www.agentx.so/)
- **PyPI** - [agentx-python](https://pypi.org/project/agentx-python/)
- **Tracing docs** - [TRACING.md](TRACING.md)
- **Evaluations docs** - [EVALUATIONS.md](EVALUATIONS.md)
- **Monitor docs** - [docs.agentx.so/sdk/monitor](https://docs.agentx.so/sdk/monitor)
- **Self-host** - [AgentX-trace-eval](https://github.com/AgentX-ai/AgentX-trace-eval)
