Metadata-Version: 2.1
Name: exemplar-harness-sdk
Version: 0.2.20
Summary: Exemplar harness SDK — session ingest, memory, skills, prompts, HITL, MCP tools, and Relay policy adapters
License: LicenseRef-Proprietary
Keywords: exemplar,harness,eval,llm,agents,telemetry,observability,relay,policy
Author: Exemplar Dev LLC
Requires-Python: >=3.10,<3.14
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Provides-Extra: ag2
Provides-Extra: agno
Provides-Extra: all
Provides-Extra: anthropic
Provides-Extra: autogen
Provides-Extra: claude-agent
Provides-Extra: crewai
Provides-Extra: deepagents
Provides-Extra: google-adk
Provides-Extra: google-genai
Provides-Extra: haystack
Provides-Extra: langchain
Provides-Extra: langgraph
Provides-Extra: litellm
Provides-Extra: llamaindex
Provides-Extra: openai
Provides-Extra: portkey
Provides-Extra: pydantic-ai
Provides-Extra: semantic-kernel
Provides-Extra: smolagents
Requires-Dist: ag2 (>=0.14) ; extra == "ag2" or extra == "all"
Requires-Dist: agno (>=1.0) ; extra == "agno" or extra == "all"
Requires-Dist: anthropic (>=0.40) ; extra == "anthropic" or extra == "all"
Requires-Dist: autogen-agentchat (>=0.4) ; extra == "autogen" or extra == "all"
Requires-Dist: autogen-ext[openai] (>=0.4) ; extra == "autogen" or extra == "all"
Requires-Dist: claude-agent-sdk (>=0.2) ; extra == "claude-agent" or extra == "all"
Requires-Dist: crewai (>=0.80) ; extra == "crewai" or extra == "all"
Requires-Dist: deepagents (>=0.6) ; (python_version >= "3.11" and python_version < "3.14") and (extra == "deepagents" or extra == "all")
Requires-Dist: exemplar-core (>=0.1.10,<0.2)
Requires-Dist: google-adk (>=0.1) ; extra == "google-adk" or extra == "all"
Requires-Dist: google-genai (>=1.52.0) ; extra == "google-genai" or extra == "all"
Requires-Dist: haystack-ai (>=2.0) ; extra == "haystack" or extra == "all"
Requires-Dist: httpx (>=0.27.0)
Requires-Dist: langchain-core (>=0.3) ; extra == "langchain" or extra == "langgraph" or extra == "deepagents" or extra == "all"
Requires-Dist: langchain-google-genai (>=2.0) ; extra == "langchain" or extra == "langgraph" or extra == "deepagents" or extra == "all"
Requires-Dist: langchain-mcp-adapters (>=0.1) ; extra == "langchain" or extra == "langgraph" or extra == "deepagents" or extra == "all"
Requires-Dist: langgraph (>=0.2) ; extra == "langgraph" or extra == "deepagents" or extra == "all"
Requires-Dist: litellm (>=1.55) ; extra == "litellm" or extra == "all"
Requires-Dist: llama-index-core (>=0.10.20) ; extra == "llamaindex" or extra == "all"
Requires-Dist: mcp (>=1.13) ; extra == "langchain" or extra == "litellm" or extra == "portkey" or extra == "anthropic" or extra == "claude-agent" or extra == "agno" or extra == "langgraph" or extra == "deepagents" or extra == "openai" or extra == "autogen" or extra == "google-adk" or extra == "google-genai" or extra == "pydantic-ai" or extra == "smolagents" or extra == "all" or extra == "all"
Requires-Dist: openai (>=1.0) ; extra == "openai" or extra == "ag2" or extra == "all"
Requires-Dist: portkey-ai (>=1.9) ; extra == "portkey" or extra == "all"
Requires-Dist: pydantic-ai-slim[google] (>=0.2) ; extra == "pydantic-ai" or extra == "all"
Requires-Dist: semantic-kernel (>=1.0) ; extra == "semantic-kernel" or extra == "all"
Requires-Dist: smolagents (>=1.0) ; extra == "smolagents" or extra == "all"
Project-URL: Documentation, https://exemplar.dev
Project-URL: Homepage, https://exemplar.dev
Project-URL: Issue Tracker, https://github.com/Exemplar-Dev/exemplar-harness-sdk/issues
Project-URL: Repository, https://github.com/Exemplar-Dev/exemplar-harness-sdk
Description-Content-Type: text/markdown

# exemplar-harness-sdk

Python SDK for Exemplar: **session ingest**, **long-term memory**, **skills**, **prompts**, **HITL**, and **Relay** policy adapters for runtime agent SDKs.

Quick start covers **Agno**, **OpenAI SDK**, **Google ADK**, and **Claude Agent SDK**. More frameworks — see [Framework extras](#framework-extras). **Relay** policy adapters — see [Relay policy](#relay-policy) and [`examples/relay/`](examples/relay/README.md).

**Related package:** terminal CLI — [`exemplar-cli`](https://pypi.org/project/exemplar-cli/) ([docs](exemplar-cli/README.md), [examples](examples/cli/README.md)).

## Contents

- [Install](#install)
- [Quick start](#quick-start)
  - [Agno](#agno)
    - [Session ingest](#agno-session-ingest)
    - [Memory](#agno-memory)
    - [Skills](#agno-skills)
    - [Prompts](#agno-prompts)
  - [OpenAI SDK](#openai-sdk)
    - [Session ingest](#openai-sdk-session-ingest)
    - [Memory](#openai-sdk-memory)
    - [Skills](#openai-sdk-skills)
    - [Prompts](#openai-sdk-prompts)
  - [Google ADK](#google-adk)
    - [Session ingest](#google-adk-session-ingest)
    - [Memory](#google-adk-memory)
    - [Skills](#google-adk-skills)
    - [Prompts](#google-adk-prompts)
  - [Claude Agent SDK](#claude-agent-sdk)
    - [Session ingest](#claude-agent-sdk-session-ingest)
    - [Memory](#claude-agent-sdk-memory)
    - [Skills](#claude-agent-sdk-skills)
    - [Prompts](#claude-agent-sdk-prompts)
- [Session ingest](#session-ingest)
  - [Example 1 — Framework helper (Agno)](#example-1--framework-helper-agno)
  - [Example 2 — Direct ingest (no framework)](#example-2--direct-ingest-no-framework)
- [Memory](#memory)
  - [Example 1 — Add and recall](#example-1--add-and-recall)
  - [Example 2 — Search and update](#example-2--search-and-update)
- [Skills](#skills)
  - [Example 1 — Create and list](#example-1--create-and-list)
  - [Example 2 — Get and search](#example-2--get-and-search)
- [Prompts](#prompts)
  - [Example 1 — Create and run](#example-1--create-and-run)
  - [Example 2 — Build for your own agent framework](#example-2--build-for-your-own-agent-framework)
  - [Example 3 — List and get](#example-3--list-and-get)
- [HITL (human-in-the-loop)](#hitl-human-in-the-loop)
  - [Example 1 — Approval gate (create + wait in one call)](#example-1--approval-gate-create--wait-in-one-call)
  - [Example 2 — Free-text input and single choice](#example-2--free-text-input-and-single-choice)
  - [Example 3 — Non-blocking create, check later, cancel](#example-3--non-blocking-create-check-later-cancel)
  - [Example 4 — As a tool in any agent framework](#example-4--as-a-tool-in-any-agent-framework)
- [Relay policy](#relay-policy)
  - [Per-SDK examples](#relay-per-sdk-examples)
  - [Hook-free evaluate](#relay-hook-free-evaluate)
- [CLI (separate package)](#cli-separate-package)
- [Advanced usage](#advanced-usage)
  - [Session ingest — Session helper with auto judge](#session-ingest--session-helper-with-auto-judge)
  - [Memory — Generic hook in any agent loop](#memory--generic-hook-in-any-agent-loop)
  - [Skills — Install folders for agent runtimes](#skills--install-folders-for-agent-runtimes)
  - [Prompts — Publish, build for frameworks, or run inline](#prompts--publish-build-for-frameworks-or-run-inline)
- [Framework extras](#framework-extras)
- [Reference](#reference)
  - [Optional install extras](#optional-install-extras)
  - [Envelope v1](#envelope-v1)

## Install

```bash
pip install exemplar-harness-sdk

# One agent framework (pick what you use)
pip install "exemplar-harness-sdk[agno]"
pip install "exemplar-harness-sdk[openai]"
pip install "exemplar-harness-sdk[google-adk]"
pip install "exemplar-harness-sdk[claude-agent]"

# Everything
pip install "exemplar-harness-sdk[all]"
```

```bash
export EXEMPLAR_API_KEY="eis_your_org_api_key"
```

Licensed for **non-commercial use** only. Commercial use requires a separate license from [Exemplar Dev LLC](https://exemplar.dev). See [LICENSE](LICENSE).

---

## Quick start

Pick your agent framework below. Each section shows **session ingest**, **memory**, **skills**, and **prompts** for that stack.

Reuse the same `session_id` for every turn in a conversation. Session helpers call `Harness.ingest()` for you (Google ADK: call `ingest_adk_session` after the run). Skills and prompts are shared platform APIs — wire them into your agent with the patterns below.

```mermaid
flowchart LR
  Agent[Your agent] --> Integration[SDK integration]
  Integration --> Harness[Harness.ingest]
  Harness --> API[Exemplar platform API]
  API --> Eval[Harness eval]
```

### Agno

```bash
pip install "exemplar-harness-sdk[agno]"
```

<a id="agno-session-ingest"></a>

#### Session ingest

```python
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from exemplar_harness import Harness
from exemplar_harness.integrations.agno import harness_agno_post_hook

harness = Harness.from_env()
agent = Agent(
    name="support-bot",
    model=OpenAIChat(id="gpt-4o"),
    post_hooks=[
        harness_agno_post_hook(
            harness,
            session_id="sess-abc",
            agent_id="support-bot",
            source_app="my-app",
        )
    ],
)
agent.run("Summarize our refund policy.")
```

#### Relay policy (tool hooks)

Same Harness client — share `agent_id` / `session_id` with ingest. Full per-SDK demos: [`examples/relay/`](examples/relay/README.md).

```python
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from exemplar_harness import Harness
from exemplar_harness.integrations.agno import harness_agno_post_hook

harness = Harness.from_env(agent_id="support-bot")
relay = harness.relay(surface="agno", source_app="my-app")

def get_weather(city: str) -> str:
    """Return a short weather summary for the city."""
    return f"Sunny in {city}"

agent = Agent(
    name="support-bot",
    model=OpenAIChat(id="gpt-4o"),
    tools=[get_weather],
    tool_hooks=[relay.agno_tool_hook(session_id="sess-abc")],
    post_hooks=[
        harness_agno_post_hook(
            harness,
            session_id="sess-abc",
            agent_id="support-bot",
            source_app="my-app",
        )
    ],
)
```

Runnable example: [`examples/relay/agno.py`](examples/relay/agno.py)

<a id="agno-memory"></a>

#### Memory

```python
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from exemplar_harness import Harness
from exemplar_harness.integrations.agno import harness_agno_post_hook
from exemplar_harness.integrations.memory.agno import (
    make_agno_memory_helper,
    harness_agno_memory_hooks,
)

harness = Harness.from_env()
mem = make_agno_memory_helper(
    harness, user_id="user-123", session_id="sess-abc", app_id="my-app"
)
pre, post = harness_agno_memory_hooks(mem)

agent = Agent(
    name="support-bot",
    model=OpenAIChat(id="gpt-4o"),
    pre_hooks=[pre],
    post_hooks=[
        post,
        harness_agno_post_hook(harness, session_id="sess-abc", agent_id="support-bot"),
    ],
)
agent.run("What do you know about my formatting preferences?")
```

Live example: [`examples/live/agno_memory_demo.py`](examples/live/agno_memory_demo.py)

<a id="agno-skills"></a>

#### Skills

Install the skill folder for Agent Skills–compatible runtimes (`SKILL.md` + files).
Prompt-injection frameworks can also pass `skill.instructions` (markdown body only).

```python
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from exemplar_harness import Harness

harness = Harness.from_env()
skills = harness.skills()

# Folder-first: dest/<name>/SKILL.md (+ references/, scripts/, assets/)
skills.install(".agents/skills", names=["refund-policy"])

skill = skills.get("refund-policy")
agent = Agent(
    name="support-bot",
    model=OpenAIChat(id="gpt-4o"),
    instructions=[skill.instructions],
)
agent.run("Can a customer return an item after 20 days?")
```

Live example: [`examples/live/agno_skills_demo.py`](examples/live/agno_skills_demo.py)

<a id="agno-prompts"></a>

#### Prompts

```python
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from exemplar_harness import Harness

harness = Harness.from_env()
built = harness.prompts().build("support-summary", variables={"topic": "returns"})

system = next((m["content"] for m in built.messages if m["role"] == "system"), "")
user = next((m["content"] for m in built.messages if m["role"] == "user"), "")

agent = Agent(
    name="support-bot",
    model=OpenAIChat(id="gpt-4o"),
    instructions=[system] if system else None,
)
agent.run(user)
```

### OpenAI SDK

```bash
pip install "exemplar-harness-sdk[openai]"
```

<a id="openai-sdk-session-ingest"></a>

#### Session ingest

```python
from openai import OpenAI
from exemplar_harness import Harness
from exemplar_harness.integrations.openai import HarnessOpenAICallback, sdk_chat_completion

harness = Harness.from_env()
callback = HarnessOpenAICallback(
    harness,
    session_id="sess-abc",
    agent_id="support-bot",
    source_app="my-app",
)
client = OpenAI()
sdk_chat_completion(
    harness,
    callback,
    client,
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarize our refund policy."}],
)
```

<a id="openai-sdk-memory"></a>

#### Memory

```python
from openai import OpenAI
from exemplar_harness import Harness
from exemplar_harness.integrations.memory.openai import (
    make_openai_memory_helper,
    sdk_chat_completion_with_memory,
)

harness = Harness.from_env()
helper = make_openai_memory_helper(
    harness, user_id="user-123", session_id="sess-abc", app_id="my-app"
)
client = OpenAI()
sdk_chat_completion_with_memory(
    helper,
    client,
    model="gpt-4o",
    messages=[{"role": "user", "content": "What do you know about my formatting preferences?"}],
)
```

Live example: [`examples/live/openai_memory_demo.py`](examples/live/openai_memory_demo.py)

<a id="openai-sdk-skills"></a>

#### Skills

```python
from openai import OpenAI
from exemplar_harness import Harness

harness = Harness.from_env()
skills = harness.skills()
skills.install(".agents/skills", names=["refund-policy"])

skill = skills.get("refund-policy")
client = OpenAI()
client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": skill.instructions},
        {"role": "user", "content": "Can a customer return an item after 20 days?"},
    ],
)
```

Live example: [`examples/live/openai_skills_demo.py`](examples/live/openai_skills_demo.py)

#### Relay policy

OpenAI **Agents** SDK (not Chat Completions) uses tool guardrails:

```python
harness = Harness.from_env(agent_id="support-bot")
relay = harness.relay(surface="openai_agents", source_app="my-app")
# Agent(..., tool_input_guardrails=[relay.openai_tool_input(session_id=...)],
#            tool_output_guardrails=[relay.openai_tool_output(session_id=...)])
```

Runnable example: [`examples/relay/openai_agents.py`](examples/relay/openai_agents.py)

<a id="openai-sdk-prompts"></a>

#### Prompts

```python
from openai import OpenAI
from exemplar_harness import Harness
from exemplar_harness.integrations.openai import HarnessOpenAICallback, sdk_chat_completion

harness = Harness.from_env()
built = harness.prompts().build("support-summary", variables={"topic": "returns"})
callback = HarnessOpenAICallback(harness, session_id="sess-abc", agent_id="support-bot")
client = OpenAI()
sdk_chat_completion(
    harness,
    callback,
    client,
    model=built.model or "gpt-4o",
    messages=built.messages,
)
```

### Google ADK

```bash
pip install "exemplar-harness-sdk[google-adk]"
```

<a id="google-adk-session-ingest"></a>

#### Session ingest

```python
import asyncio

from exemplar_harness import Harness
from exemplar_harness.integrations.google_adk import ingest_adk_session
from google.adk.agents import LlmAgent
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.genai import types

harness = Harness.from_env()
session_id, app_name, user_id = "sess-abc", "my_app", "user-1"

agent = LlmAgent(
    model="gemini-2.5-flash",
    name="support_bot",  # must be a valid Python identifier
    instruction="Answer concisely.",
)
sessions = InMemorySessionService()


async def run_and_ingest() -> None:
    await sessions.create_session(app_name=app_name, user_id=user_id, session_id=session_id)
    runner = Runner(agent=agent, app_name=app_name, session_service=sessions)
    message = types.Content(role="user", parts=[types.Part(text="Summarize our refund policy.")])
    async for _ in runner.run_async(user_id=user_id, session_id=session_id, new_message=message):
        pass
    session = await sessions.get_session(app_name=app_name, user_id=user_id, session_id=session_id)
    ingest_adk_session(
        harness,
        session.model_dump(mode="json", exclude_none=True),
        session_id=session_id,
        agent_id="support-bot",
        source_app="my-app",
    )


asyncio.run(run_and_ingest())
```

<a id="google-adk-memory"></a>

#### Memory

```python
from exemplar_harness import Harness
from exemplar_harness.integrations.memory.google_adk import (
    make_google_adk_memory_helper,
    prepare_user_message,
    record_session_turn_with_memory,
)

harness = Harness.from_env()
mem = make_google_adk_memory_helper(
    harness, user_id="user-123", session_id="sess-abc", app_id="my-app"
)

question = "What do you know about my formatting preferences?"
user_text = prepare_user_message(mem, question)
# ... run your ADK agent with user_text, capture model_response ...
# record_session_turn_with_memory(mem, user_message=question, model_response=model_response)
```

Live example: [`examples/live/google_adk_memory_demo.py`](examples/live/google_adk_memory_demo.py)

<a id="google-adk-skills"></a>

#### Skills

```python
from exemplar_harness import Harness
from google.adk.agents import LlmAgent

harness = Harness.from_env()
skills = harness.skills()
skills.install(".agents/skills", names=["refund-policy"])

skill = skills.get("refund-policy")
agent = LlmAgent(
    model="gemini-2.5-flash",
    name="support_bot",
    instruction=skill.instructions,
)
```

Live example: [`examples/live/google_adk_skills_demo.py`](examples/live/google_adk_skills_demo.py)

#### Relay policy

```python
harness = Harness.from_env(agent_id="support-bot")
relay = harness.relay(surface="adk", source_app="my-app")
agent = LlmAgent(
    model="gemini-2.5-flash",
    name="support_bot",
    tools=[...],
    before_tool_callback=relay.adk_before_tool(session_id="sess-abc"),
    after_tool_callback=relay.adk_after_tool(session_id="sess-abc"),
)
```

Runnable example: [`examples/relay/adk.py`](examples/relay/adk.py)

<a id="google-adk-prompts"></a>

#### Prompts

```python
from exemplar_harness import Harness
from google.adk.agents import LlmAgent
from google.genai import types

harness = Harness.from_env()
built = harness.prompts().build("support-summary", variables={"topic": "returns"})

system = next((m["content"] for m in built.messages if m["role"] == "system"), "Answer concisely.")
user = next((m["content"] for m in built.messages if m["role"] == "user"), "")

agent = LlmAgent(model="gemini-2.5-flash", name="support_bot", instruction=system)
message = types.Content(role="user", parts=[types.Part(text=user)])
# pass `message` into Runner.run_async(...), then ingest_adk_session as above
```

### Claude Agent SDK

```bash
pip install "exemplar-harness-sdk[claude-agent]"
```

<a id="claude-agent-sdk-session-ingest"></a>

#### Session ingest

```python
import asyncio

from exemplar_harness import Harness
from exemplar_harness.integrations.claude_agent import HarnessClaudeAgentHandler

harness = Harness.from_env()
handler = HarnessClaudeAgentHandler(
    harness,
    session_id="sess-abc",
    agent_id="support-bot",
    source_app="my-app",
)
asyncio.run(handler.run_query("Summarize our refund policy."))
```

<a id="claude-agent-sdk-memory"></a>

#### Memory

```python
import asyncio

from exemplar_harness import Harness
from exemplar_harness.integrations.claude_agent import HarnessClaudeAgentHandler
from exemplar_harness.integrations.memory.claude_agent import (
    make_claude_agent_memory_helper,
    prepare_prompt,
    record_agent_result_with_memory,
)

harness = Harness.from_env()
handler = HarnessClaudeAgentHandler(harness, session_id="sess-abc", agent_id="support-bot")
mem = make_claude_agent_memory_helper(
    harness, user_id="user-123", session_id="sess-abc", app_id="my-app"
)


async def run() -> None:
    question = "What do you know about my formatting preferences?"
    prompt = prepare_prompt(mem, question)
    result, _ = await handler.run_query(prompt)
    record_agent_result_with_memory(mem, prompt=question, result=result)


asyncio.run(run())
```

Live example: [`examples/live/claude_agent_memory_demo.py`](examples/live/claude_agent_memory_demo.py)

<a id="claude-agent-sdk-skills"></a>

#### Skills

```python
import asyncio

from exemplar_harness import Harness
from exemplar_harness.integrations.claude_agent import (
    HarnessClaudeAgentHandler,
    merge_claude_agent_options,
)

harness = Harness.from_env()
skills = harness.skills()
# Claude Code also discovers folders under .claude/skills via CLI install/pull
skills.install(".claude/skills", names=["refund-policy"])

skill = skills.get("refund-policy")
handler = HarnessClaudeAgentHandler(harness, session_id="sess-abc", agent_id="support-bot")


async def run() -> None:
    from claude_agent_sdk import ClaudeAgentOptions

    options = merge_claude_agent_options(
        handler,
        ClaudeAgentOptions(system_prompt=skill.instructions),
    )
    await handler.run_query("Can a customer return an item after 20 days?", options=options)


asyncio.run(run())
```

Live example: [`examples/live/claude_agent_skills_demo.py`](examples/live/claude_agent_skills_demo.py)

#### Relay policy

```python
from claude_agent_sdk import ClaudeAgentOptions

harness = Harness.from_env(agent_id="support-bot")
relay = harness.relay(surface="claude_sdk", source_app="my-app")
options = ClaudeAgentOptions(
    hooks=relay.claude_hooks(session_id="sess-abc"),
)
```

Runnable example: [`examples/relay/claude_sdk.py`](examples/relay/claude_sdk.py)

<a id="claude-agent-sdk-prompts"></a>

#### Prompts

```python
import asyncio

from exemplar_harness import Harness
from exemplar_harness.integrations.claude_agent import (
    HarnessClaudeAgentHandler,
    merge_claude_agent_options,
)

harness = Harness.from_env()
built = harness.prompts().build("support-summary", variables={"topic": "returns"})
system = next((m["content"] for m in built.messages if m["role"] == "system"), "")
user = next((m["content"] for m in built.messages if m["role"] == "user"), "")
handler = HarnessClaudeAgentHandler(harness, session_id="sess-abc", agent_id="support-bot")


async def run() -> None:
    from claude_agent_sdk import ClaudeAgentOptions

    options = merge_claude_agent_options(
        handler,
        ClaudeAgentOptions(system_prompt=system or None),
    )
    await handler.run_query(user, options=options)


asyncio.run(run())
```

---

## Session ingest

**Base helper:** `Harness.ingest()` / framework helpers above.

### Example 1 — Framework helper (Agno)

```python
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from exemplar_harness import Harness
from exemplar_harness.integrations.agno import harness_agno_post_hook

harness = Harness.from_env()
agent = Agent(
    name="support-bot",
    model=OpenAIChat(id="gpt-4o"),
    post_hooks=[harness_agno_post_hook(harness, session_id="sess-abc", agent_id="support-bot")],
)
agent.run("What is the return window?")
```

### Example 2 — Direct ingest (no framework)

```python
from exemplar_harness import Harness

harness = Harness.from_env()
harness.ingest(
    "generic",
    session_id="sess-abc",
    event="turns",
    data={
        "turns": [
            {
                "input": "What is harness eval?",
                "output": "Automated judge over agent sessions.",
                "model": "gpt-4o",
            }
        ]
    },
    agent_id="my-agent",
    source_app="my-app",
)
```

| Parameter | Purpose |
|-----------|---------|
| `session_id` | Groups turns into one eval session |
| `agent_id` | Agent id on the ingest body (`agentId`); set on `Harness(..., agent_id=...)` to also send `X-Harness-Agent-Id` on MCP tool calls |
| `source_app` | Your application name |
| `auto_judge_run` | Run harness judge after ingest (`True` / `False` / omit) |

---

## Memory

**Base helper:** `harness.memory(...)`.

Search accepts `search_mode` (`hybrid` | `semantic` | `keyword`) and `format`
(`json` | `json_compact` | `markdown` | `toon`). `recall` defaults to `markdown`
and prefers the server-rendered `body` when present.

### Example 1 — Add and recall

```python
from exemplar_harness import Harness

harness = Harness.from_env()
memory = harness.memory(user_id="user-123", session_id="chat-abc", app_id="my-app")

memory.add("User prefers bullet-point answers.", memory_type="preference")
context = memory.recall("how should I format answers?", format="markdown")
```

### Example 2 — Search modes and formats

```python
results = memory.search(
    "formatting preferences",
    top_k=5,
    search_mode="hybrid",
    format="json",
)
envelope = memory.search_envelope("theme", format="toon", search_mode="hybrid")
# envelope.body → prompt injection; envelope.records → structured hits

listed = memory.list(limit=20)
record = memory.get(listed[0].memory_id)
memory.update(record.memory_id, content="User prefers numbered lists.")
memory.delete(record.memory_id)
```

---

## Skills

**Base helper:** `harness.skills()`.

### Example 1 — Create and list

```python
from exemplar_harness import Harness

harness = Harness.from_env()
skills = harness.skills()

record = skills.create(
    name="refund-policy",
    instructions="# Refund policy\n\nReturns within 30 days.\n\nSee [references/policy.md](references/policy.md).",
    description="Refund workflow",
    tags=["support"],
    files={"references/policy.md": "# Policy\n\n30-day returns.\n"},
)
items = skills.list(limit=20)
```

### Example 2 — Get, search, and install

```python
fetched = skills.get("refund-policy")  # by ID or unique name
hits = skills.search("refund", top_k=5)

# Materialize SKILL.md + supporting files for agent runtimes
skills.install(".agents/skills", names=["refund-policy"])
```

---

## Prompts

**Base helper:** `harness.prompts()`.

### Example 1 — Create and run

```python
from exemplar_harness import Harness

harness = Harness.from_env()
prompts = harness.prompts()

record = prompts.create(
    name="support-summary",
    title="Support summary",
    messages=[
        {"role": "system", "content": "Be concise."},
        {"role": "user", "content": "Summarize topic: {{topic}}."},
    ],
    variables=["topic"],
)
result = prompts.run("support-summary", variables={"topic": "returns"})
print(result["content"])
```

### Example 2 — Build for your own agent framework

Fetch a stored prompt, substitute `{{variables}}` locally, and hand messages to LangGraph, OpenAI, Haystack, etc. Does not call Exemplar’s model runner.

```python
built = prompts.build("support-summary", variables={"topic": "returns"})
# built.messages -> [{"role": "system", ...}, {"role": "user", ...}]
# built.model    -> default model from the registry (if set)
# pass built.messages into your framework / LLM client
```

Or reuse an already-fetched record (no second network call):

```python
record = prompts.get("support-summary")
built = record.build(variables={"topic": "returns"})
```

### Example 3 — List and get

```python
listed = prompts.list(limit=20)
fetched = prompts.get("support-summary")  # by ID or unique name
hits = prompts.search("support", top_k=5)
```

---

## HITL (human-in-the-loop)

**Base helper:** `harness.hitl()`. Lets an agent pause mid-workflow and wait for a human decision. Requests appear in the console under Agent Harness → HITL Approvals, where a reviewer approves, rejects, answers, or picks an option. Blocking wait uses polling; there are no webhooks.

### Example 1 — Approval gate (create + wait in one call)

```python
from exemplar_harness import Harness, HITLTimeoutError

harness = Harness.from_env(agent_id="deploy-agent")
hitl = harness.hitl()

try:
    result = hitl.ask(
        "Deploy build 1234 to production?",
        description="All checks green. Blast radius: payments service.",
        payload={"build": "1234", "service": "payments"},
        ttl_seconds=1800,   # server-side expiry
        timeout=1800,       # local wait budget (seconds)
        poll_interval=5,
    )
except HITLTimeoutError:
    result = None

if result and result.approved:
    deploy()
elif result and result.state == "rejected":
    print(f"Rejected: {result.response.comment}")
```

### Example 2 — Free-text input and single choice

```python
answer = hitl.ask("Which rollback strategy?", request_type="input")
print(answer.response.text)

choice = hitl.ask(
    "Pick a deployment region",
    request_type="select",
    options=["us-east-1", "eu-west-1"],
)
print(choice.response.option)
```

### Example 3 — Non-blocking create, check later, cancel

```python
request = hitl.request_approval("Rotate production credentials?")

status = hitl.get(request.request_id)   # non-blocking status check
if status.is_pending:
    hitl.cancel(request.request_id)     # e.g. the run was aborted
```

### Example 4 — As a tool in any agent framework

Plain sync methods — wrap them in a tool function for LangChain, Agno, CrewAI, OpenAI, etc.

```python
from langchain_core.tools import tool

@tool
def ask_human_approval(question: str) -> str:
    """Ask a human operator to approve or reject an action."""
    result = harness.hitl().ask(question, timeout=900)
    if result.approved:
        return "approved"
    return f"denied ({result.state}): {result.response.comment or 'no comment'}"
```

---

## Relay policy

<a id="relay-policy"></a>

**Base helper:** `harness.relay(surface=...)`. Runtime policy for agent tool calls — decide before execute, observe after. Surfaces: `claude_sdk` | `adk` | `agno` | `openai_agents` | `langchain` | `langgraph` | `pydantic_ai` | `crewai` | `semantic_kernel`.

Index and run instructions: [`examples/relay/README.md`](examples/relay/README.md).

```bash
HARNESS_EXAMPLES_SIMULATED=1 python -m examples.relay.run_all
HARNESS_EXAMPLES_SIMULATED=1 python -m examples.relay.run_all --list
EXEMPLAR_API_KEY=... python -m examples.relay.evaluate
```

<a id="relay-per-sdk-examples"></a>

### Per-SDK examples

| Surface | Adapter | Example |
|---------|---------|---------|
| Agno | `relay.agno_tool_hook` | [`examples/relay/agno.py`](examples/relay/agno.py) |
| Claude Agent SDK | `relay.claude_hooks` | [`examples/relay/claude_sdk.py`](examples/relay/claude_sdk.py) |
| Google ADK | `relay.adk_before_tool` / `adk_after_tool` | [`examples/relay/adk.py`](examples/relay/adk.py) |
| OpenAI Agents | `relay.openai_tool_input` / `openai_tool_output` | [`examples/relay/openai_agents.py`](examples/relay/openai_agents.py) |
| LangChain | `relay.langchain_middleware` | [`examples/relay/langchain.py`](examples/relay/langchain.py) |
| LangGraph | `relay.langgraph_middleware` | [`examples/relay/langgraph.py`](examples/relay/langgraph.py) |
| Pydantic AI | `relay.pydantic_ai_hooks` | [`examples/relay/pydantic_ai.py`](examples/relay/pydantic_ai.py) |
| CrewAI | `relay.crewai_hooks` / `crewai_register` | [`examples/relay/crewai.py`](examples/relay/crewai.py) |
| Semantic Kernel | `relay.semantic_kernel_filter` / `semantic_kernel_register` | [`examples/relay/semantic_kernel.py`](examples/relay/semantic_kernel.py) |

<a id="relay-hook-free-evaluate"></a>

### Hook-free evaluate

CI / preflight / custom runtimes — same Control + Enforcement stack, no SDK hooks:

```python
from exemplar_harness import Harness
from exemplar_harness.relay import RelayDecision

harness = Harness.from_env(agent_id="support-bot")
relay = harness.relay(surface="agno", source_app="my-app")

verdict = relay.evaluate(
    tool_name="shell",
    arguments={"command": "rm -rf /tmp/x"},
    user_id="user_123",
)
if verdict.decision is RelayDecision.DENY:
    raise RuntimeError(verdict.reason)

relay.evaluate_path(path=".env")
relay.evaluate_bash(command="ls -la")
```

Runnable example: [`examples/relay/evaluate.py`](examples/relay/evaluate.py)

---

## CLI (separate package)

Skills, prompts, and memory are also available as a standalone terminal package — not bundled with this SDK:

| | |
|--|--|
| PyPI | [`exemplar-cli`](https://pypi.org/project/exemplar-cli/) |
| Docs | [`exemplar-cli/README.md`](exemplar-cli/README.md) |
| Examples | [`examples/cli/README.md`](examples/cli/README.md) |

```bash
pip install exemplar-cli
export EXEMPLAR_API_KEY="eis_your_org_api_key"
exemplar doctor
```

---

## Advanced usage

One advanced pattern per offering. For full framework wiring, see [`examples/live/`](examples/live/).

### Session ingest — Session helper with auto judge

```python
from exemplar_harness import Harness

harness = Harness.from_env()
session = harness.session(
    "sess-abc",
    agent_id="support-bot",
    source_app="my-app",
    auto_judge_run=True,  # run judge after each ingest through this session
)
session.ingest(
    "generic",
    event="turns",
    data={"turns": [{"input": "Hello", "output": "Hi!", "model": "gpt-4o"}]},
)
```

### Memory — Generic hook in any agent loop

```python
from exemplar_harness import Harness
from exemplar_harness.integrations.memory import MemoryHook

harness = Harness.from_env()
memory = harness.memory(user_id="user-123", session_id="chat-abc", app_id="my-app")
hook = MemoryHook(memory, recall_top_k=5, auto_add=False)

user_input = "What do you know about me?"
recall = hook.before_turn(user_input)  # prepend to system prompt
# ... run LLM ...
hook.after_turn(user_input, assistant_output)  # no-op unless auto_add=True
```

### Skills — Install folders for agent runtimes

A skill is a directory rooted at `SKILL.md` (plus optional `references/`, `scripts/`, `assets/`).
Install the whole folder for Cursor, Claude Code, ADK, Agno, Deep Agents, AG2, CrewAI, and similar agents —
do not treat `record.instructions` as the full skill (that field is the markdown body only,
useful for quick editor/MCP use).

Live skills demos also cover framework-native loaders:

- Deep Agents: [`examples/live/deepagents_skills_demo.py`](examples/live/deepagents_skills_demo.py) (`create_deep_agent(..., skills=[...])`)
- AG2: [`examples/live/ag2_skills_demo.py`](examples/live/ag2_skills_demo.py) (`SkillPlugin`)
- CrewAI: [`examples/live/crewai_skills_demo.py`](examples/live/crewai_skills_demo.py) (`Agent(..., skills=[...])`)
- Agno / OpenAI / Google ADK / Claude Agent: see Quick start sections above

```python
from exemplar_harness import Harness

skills = Harness.from_env().skills()

# Materialize all active skills: dest/<name>/SKILL.md + supporting files
paths = skills.install(".agents/skills")

# Or one skill into Cursor project skills
skills.install(".cursor/skills", names=["refund-policy"])
```

CLI equivalent: `exemplar skills install --all --dest .agents/skills` (alias of `pull`).

### Prompts — Publish, build for frameworks, or run inline

```python
from exemplar_harness import Harness

prompts = Harness.from_env().prompts()

prompts.publish_version(
    "support-summary",
    messages=[
        {"role": "system", "content": "Be concise."},
        {"role": "user", "content": "Bullet summary for: {{topic}}."},
    ],
    change_notes="Use bullets",
)

# Hand off to your agent framework (local {{var}} substitution)
built = prompts.build("support-summary", variables={"topic": "returns"})

# Or execute via Exemplar without a stored prompt
inline = prompts.run_inline(
    messages=[
        {"role": "system", "content": "Be concise."},
        {"role": "user", "content": "Say hello."},
    ],
    model="openai/gpt-4o-mini",
)
```

---

## Framework extras

| Framework | Extra | Wire this |
|-----------|-------|-----------|
| LangChain | `langchain` | `make_langchain_callback_handler` → `callbacks=[handler]` |
| LangGraph | `langgraph` | `HarnessLangGraphHandler.make_graph_callback_handler()` |
| Deep Agents | `deepagents` | `HarnessDeepAgentsHandler.make_graph_callback_handler()` / `record_run` |
| LiteLLM | `litellm` | `register_litellm_handler` + `metadata={"session_id": ...}` |
| OpenAI SDK | `openai` | `HarnessOpenAICallback.on_completion` |
| Anthropic SDK | `anthropic` | `make_anthropic_middleware` |
| Claude Agent SDK | `claude-agent` | `HarnessClaudeAgentHandler.run_query` |
| Portkey | `portkey` | `HarnessPortkeyCallback.on_completion` |
| Agno | `agno` | `harness_agno_post_hook` → `Agent.post_hooks` |
| Haystack | `haystack` | `HarnessHaystackHandler.run_and_record` |
| LlamaIndex | `llamaindex` | `register_llamaindex_handler` |
| AutoGen | `autogen` | `HarnessAutoGenHandler.on_agent_run_complete` |
| AG2 | `ag2` | `HarnessAG2Handler.record_ask` / `record_chat_turn` |
| CrewAI | `crewai` | `make_crewai_listener` |
| Google GenAI SDK | `google-genai` | `instrument_google_genai_client` |
| Google ADK | `google-adk` | `ingest_adk_session` |
| Pydantic AI | `pydantic-ai` | `instrument_pydantic_ai_agent` |
| Semantic Kernel | `semantic-kernel` | `register_semantic_kernel_filter` |
| smolagents | `smolagents` | `harness_smolagents_step_callback` |

Matching memory helpers live under `exemplar_harness.integrations.memory.*`.
Relay policy adapters: [`examples/relay/`](examples/relay/README.md) (`python -m examples.relay.run_all`).

```bash
# List / run live demos
python -m examples.live.run_all --list
python -m examples.live.run_memory_demos --list
python -m examples.live.run_platform_demos --list
HARNESS_EXAMPLES_SIMULATED=1 python -m examples.live.run_memory_demos --only memory_agno
HARNESS_EXAMPLES_SIMULATED=1 python -m examples.live.run_platform_demos --only skills_openai
HARNESS_EXAMPLES_SIMULATED=1 python -m examples.relay.run_all
```

---

## Reference

### Optional install extras

| Extra | Module |
|-------|--------|
| `langchain` | `exemplar_harness.integrations.langchain` |
| `langgraph` | `exemplar_harness.integrations.langgraph` |
| `deepagents` | `exemplar_harness.integrations.deepagents` |
| `litellm` | `exemplar_harness.integrations.litellm` |
| `openai` | `exemplar_harness.integrations.openai` |
| `anthropic` | `exemplar_harness.integrations.anthropic` |
| `claude-agent` | `exemplar_harness.integrations.claude_agent` |
| `portkey` | `exemplar_harness.integrations.portkey` |
| `agno` | `exemplar_harness.integrations.agno` |
| `haystack` | `exemplar_harness.integrations.haystack` |
| `llamaindex` | `exemplar_harness.integrations.llamaindex` |
| `autogen` | `exemplar_harness.integrations.autogen` |
| `ag2` | `exemplar_harness.integrations.ag2` |
| `crewai` | `exemplar_harness.integrations.crewai` |
| `google-adk` | `exemplar_harness.integrations.google_adk` |
| `google-genai` | `exemplar_harness.integrations.google_genai` |
| `pydantic-ai` | `exemplar_harness.integrations.pydantic_ai` |
| `semantic-kernel` | `exemplar_harness.integrations.semantic_kernel` |
| `smolagents` | `exemplar_harness.integrations.smolagents` |

Vercel AI SDK is **not** included in the Python package (TypeScript SDK path).

### Envelope v1

Each ingest POSTs `schemaVersion: 1` to `POST /api/harness-ingest/v1/sessions` with a `sourceType` and framework-native `data` payload. The Exemplar platform API maps envelopes to eval session turns. Harness judge runs use `POST /api/harness-judge/v1/runs`; per-turn session eval uses `POST /api/harness-session-eval/v1/sessions/{sessionId}`.

