Metadata-Version: 2.5
Name: agent-learning-kit
Version: 0.1.0
Summary: Unified Future AGI SDK for agent learning workflows.
Project-URL: Homepage, https://futureagi.com
Project-URL: Documentation, https://docs.futureagi.com
Project-URL: Repository, https://github.com/future-agi/agent-learning-kit
Project-URL: Issues, https://github.com/future-agi/agent-learning-kit/issues
Project-URL: Changelog, https://github.com/future-agi/agent-learning-kit/blob/main/CHANGELOG.md
Author-email: Future AGI <hello@futureagi.io>
License: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: agent-evaluation,agent-optimization,agent-simulation,agent-testing,ai-agents,future-agi,red-teaming
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: claude-agent-sdk>=0.2.139
Requires-Dist: fi-instrumentation-otel>=0.1.16
Requires-Dist: gepa>=0.0.17
Requires-Dist: google-adk<2.8,>=2.7.1
Requires-Dist: httpx>=0.24.0
Requires-Dist: jsonschema<5,>=4.25.1
Requires-Dist: levenshtein>=0.25.0
Requires-Dist: litellm<2,>=1.80.0
Requires-Dist: nltk>=3.9.0
Requires-Dist: numpy>=1.26.4
Requires-Dist: openai<3,>=1.109.1
Requires-Dist: opentelemetry-api<2,>=1.39.1
Requires-Dist: opentelemetry-exporter-otlp<2,>=1.39.1
Requires-Dist: opentelemetry-sdk<2,>=1.39.1
Requires-Dist: optuna>=3.6.1
Requires-Dist: pandas>=2.0.0
Requires-Dist: pydantic<3,>=2.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: requests-futures>=1.0.0
Requires-Dist: requests<3,>=2.32.5
Requires-Dist: retell-sdk<6,>=5.64
Requires-Dist: rich>=13.0.0
Requires-Dist: rouge-score>=0.1.2
Requires-Dist: typer<1.0.0,>=0.9.0
Provides-Extra: a2a
Requires-Dist: a2a-sdk[http-server]>=1.1.0; extra == 'a2a'
Provides-Extra: all
Requires-Dist: aiohttp>=3.10; extra == 'all'
Requires-Dist: audioop-lts>=0.2.1; (python_version >= '3.13') and extra == 'all'
Requires-Dist: chromadb>=0.4.0; extra == 'all'
Requires-Dist: livekit-agents[cartesia,deepgram,google,openai,silero]>=1.2; extra == 'all'
Requires-Dist: livekit-plugins-elevenlabs>=1.2; extra == 'all'
Requires-Dist: sentence-transformers<6,>=5.2.3; extra == 'all'
Requires-Dist: torch<3,>=2.10.0; extra == 'all'
Requires-Dist: transformers<6,>=5.2.0; extra == 'all'
Provides-Extra: embeddings
Requires-Dist: sentence-transformers<6,>=5.2.3; extra == 'embeddings'
Provides-Extra: evaluation
Provides-Extra: feedback
Requires-Dist: chromadb>=0.4.0; extra == 'feedback'
Provides-Extra: harness-stores
Requires-Dist: psycopg[binary]>=3.2; extra == 'harness-stores'
Provides-Extra: harness-ui
Requires-Dist: fastapi<1,>=0.115; extra == 'harness-ui'
Requires-Dist: uvicorn<1,>=0.30; extra == 'harness-ui'
Provides-Extra: langchain
Requires-Dist: langchain-core<2,>=1.4.6; extra == 'langchain'
Requires-Dist: langgraph-checkpoint-sqlite>=3.1.0; extra == 'langchain'
Requires-Dist: langgraph<2,>=1.2.4; extra == 'langchain'
Provides-Extra: livekit
Requires-Dist: aiohttp>=3.10; extra == 'livekit'
Requires-Dist: audioop-lts>=0.2.1; (python_version >= '3.13') and extra == 'livekit'
Requires-Dist: livekit-agents[cartesia,deepgram,google,openai,silero]>=1.2; extra == 'livekit'
Requires-Dist: livekit-plugins-elevenlabs>=1.2; extra == 'livekit'
Provides-Extra: mcp
Requires-Dist: mcp<2,>=1.27; extra == 'mcp'
Provides-Extra: nli
Requires-Dist: torch<3,>=2.10.0; extra == 'nli'
Requires-Dist: transformers<6,>=5.2.0; extra == 'nli'
Provides-Extra: notebook
Requires-Dist: ipykernel>=6; extra == 'notebook'
Requires-Dist: nbformat>=5; extra == 'notebook'
Provides-Extra: optimize
Provides-Extra: pipecat
Requires-Dist: pipecat-ai>=0.0.108; extra == 'pipecat'
Provides-Extra: simulate
Provides-Extra: trinity
Requires-Dist: aiohttp>=3.10; extra == 'trinity'
Requires-Dist: audioop-lts>=0.2.1; (python_version >= '3.13') and extra == 'trinity'
Requires-Dist: livekit-agents[cartesia,deepgram,google,openai,silero]>=1.2; extra == 'trinity'
Requires-Dist: livekit-plugins-elevenlabs>=1.2; extra == 'trinity'
Description-Content-Type: text/markdown

<p align="center">
  <img src="https://raw.githubusercontent.com/future-agi/agent-learning-kit/main/docs/assets/futureagi-mark-email.png" alt="Future AGI" width="72" />
</p>

<h1 align="center">Agent Learning Kit</h1>

<p align="center">
  Local-first testing, simulation, red teaming, and optimization for AI agents.
</p>

<p align="center">
  <a href="https://github.com/future-agi/agent-learning-kit/blob/main/LICENSE">Apache-2.0</a>
  ·
  <a href="https://github.com/future-agi/agent-learning-kit/blob/main/docs/index.md">Docs</a>
  ·
  <a href="https://github.com/future-agi/agent-learning-kit/blob/main/CONTRIBUTING.md">Contributing</a>
  ·
  <a href="https://github.com/future-agi/agent-learning-kit/blob/main/SECURITY.md">Security</a>
  ·
  <a href="https://github.com/future-agi/agent-learning-kit/blob/main/ROADMAP.md">V1 roadmap</a>
  ·
  <a href="https://github.com/future-agi/agent-learning-kit/blob/main/LIBRARIES.md">Library inventory</a>
</p>

![Agent Learning lifecycle blueprint](https://raw.githubusercontent.com/future-agi/agent-learning-kit/main/docs/assets/hero-agent-blueprint.jpg)

Agent Learning Kit is the local-first SDK and CLI for testing, simulating,
red-teaming, and optimizing AI agents.

It brings the three core Future AGI engines into one public developer surface —
three engines, four workflows: red-teaming rides on the `simulate` and `evals`
engines rather than being a fourth engine:

- `simulate`: run local worlds, tasks, framework-shaped adapters, replays, and
  regression artifacts.
- `evals`: evaluate prompts, task outputs, runtime contracts, traces, memory,
  retrieval, safety, and robustness evidence.
- `optimize`: search over prompts, agents, framework adapters, worlds,
  multi-agent interactions, memory layers, workflows, and red-team scenarios.

Use it when you want one reproducible loop:

1. Simulate an agent or framework workflow.
2. Evaluate the behavior and runtime evidence.
3. Optimize the weak layer.
4. Promote the result into a replayable artifact.
5. Prove release readiness with local gates.

### The harness: point it at an agent and talk to it

`src/fi/alk/harness/` builds all of the above **for** an agent instead of asking you to write it.
Point it at an agent's source and it reads what that agent verifiably is, builds a real world its
tools act on, and writes test scenarios that are each proved before they are kept. It is driven
as a conversation, in a terminal or on a web page.

- **[Start here](https://github.com/future-agi/agent-learning-kit/blob/main/src/fi/alk/harness/README.md)**: setup from nothing, then how to use it
- **[The web page](https://github.com/future-agi/agent-learning-kit/blob/main/harness-ui/README.md)**: the same harness as a chat, on `localhost:8777`
- **[How it works](https://github.com/future-agi/agent-learning-kit/blob/main/src/fi/alk/harness/HOW-IT-WORKS.md)** and
  **[why it is shaped this way](https://github.com/future-agi/agent-learning-kit/blob/main/src/fi/alk/harness/DESIGN.md)**

OpenEnv/Gymnasium shapes are compatibility inputs, not the product center.
Agent Learning Kit is the primary runtime and release contract, and the bar is
the executable `environment_10x_robustness` release gate.
OpenEnv/Gymnasium-shaped traces remain compatibility evidence inside that bar.

## Install

Install from PyPI:

```bash
pip install agent-learning-kit
```

To develop against source (contributors):

```bash
git clone https://github.com/future-agi/agent-learning-kit
cd agent-learning-kit
uv sync          # or: pip install -e .
```

(npm publishing of the TypeScript SDK lands at the v1 launch.)

Optional Python extras:

```bash
pip install "agent-learning-kit[livekit]"
pip install "agent-learning-kit[nli]"
pip install "agent-learning-kit[all]"
```

TypeScript evaluation package (npm at launch; today build from
[`typescript/agent-learning-kit`](https://github.com/future-agi/agent-learning-kit/blob/main/typescript/agent-learning-kit)):

```bash
pnpm add @future-agi/agent-learning-kit
```

## Quickstart

Everything below runs fully offline — no API key, no network. Start with the
local doctor:

```bash
agent-learn doctor
```

Then run the golden path against the bundled example manifests. The
`AGENT_LEARNING_*_EXAMPLE_KEY` prefixes satisfy each manifest's
`required_env` list — that list is CI wiring metadata, not a provider
credential, so any placeholder value works.

> Prefer the SDK spine over the CLI?
> [Spec + Runner](https://github.com/future-agi/agent-learning-kit/blob/main/docs/simulate/spec-and-runner.md) runs the same simulation as
> one `SimulationSpec` fed to one `SimulationRunner` — the plug-and-play surface
> behind every simulation.

Evaluate a suite:

```bash
agent-learn eval examples/eval_suite.json \
  --output artifacts/eval.json
```

Simulate a run manifest:

```bash
AGENT_LEARNING_RUN_EXAMPLE_KEY=offline-demo-key \
  agent-learn run examples/run_manifest.json \
  --no-eval \
  --output artifacts/run.json
```

Optimize an agent workflow:

```bash
AGENT_LEARNING_OPTIMIZE_EXAMPLE_KEY=offline-demo-key \
  agent-learn optimize examples/optimization_manifest.json \
  --output artifacts/optimization.json
```

Run a red-team campaign:

```bash
AGENT_LEARNING_REDTEAM_EXAMPLE_KEY=offline-demo-key \
  agent-learn redteam examples/redteam_manifest.json \
  --output artifacts/redteam.json
```

Each command prints a `wrote <path>` line; relative `--output` paths resolve
against your current working directory.

Optional platform mode: to use Future AGI platform-backed evaluation, set
`AGENT_LEARNING_API_KEY` (it takes precedence over the `FUTURE_AGI_API_KEY`
and `FI_API_KEY` aliases), or call `configure(api_key="...")` from
`fi.alk`. See
[docs/reference/configure.md](https://github.com/future-agi/agent-learning-kit/blob/main/docs/reference/configure.md).

Cut local release proof:

```bash
agent-learn release-check --project-root .
agent-learn release-proof \
  --project-root . \
  --output /tmp/agent-learning-release-proof.json \
  --quiet
```

## TypeScript

```typescript
import { Evaluator } from "@future-agi/agent-learning-kit";
import { LocalEvaluator } from "@future-agi/agent-learning-kit/evals/local";
```

## What You Can Build

- Prompt and response evaluations.
- Local task and world simulations.
- Framework adapter probes (probe-promoted coverage) for LangChain, LangGraph,
  LlamaIndex, AutoGen, CrewAI, LiveKit, Pipecat, Browser Use, MCP, A2A, and
  custom orchestration objects.
- Runtime-simulated coverage for PydanticAI (multi-framework runtime
  simulation) and OpenAI Agents (handoff-transcript promotion).
- Runtime-contract and trace-quality checks.
- Multi-agent coordination and handoff tests.
- Retrieval and memory quality checks.
- Voice, realtime, browser/CUA, workflow, lifecycle, and protocol traces.
- Red-team corpus, campaign, adaptive-loop, and persistent-state checks.
- Optimizer governance, candidate lineage, rollback, and release proof.

## Why It Exists

Most agent stacks split testing, simulation, optimization, and safety review
across separate tools. Agent Learning Kit keeps those steps in one artifact
model so a developer can inspect what happened, score it, improve it, and replay
it in CI.

The public SDK is `agent-learning-kit`, the Python namespace is
`fi.alk`, the CLI is `agent-learn`, and the TypeScript package is
`@future-agi/agent-learning-kit`.

The active `ai-evaluation` code is included here under `src/fi/evals`, with its
TypeScript SDK source under `typescript/agent-learning-kit/src`. The
`simulate-sdk` and `agent-opt` engine code is included under `src/fi/simulate`
and `src/fi/opt`. See [LIBRARIES.md](https://github.com/future-agi/agent-learning-kit/blob/main/LIBRARIES.md) for the complete source map.
The ai-evaluation source inventory used by `agent-learn release-check` lives at
the ai-evaluation source inventory (maintained in the internal-docs repo).

## Repository Map

- [`examples/`](https://github.com/future-agi/agent-learning-kit/blob/main/examples): runnable cookbooks and manifests.
- [`src/fi/alk`](https://github.com/future-agi/agent-learning-kit/blob/main/src/fi/alk): public Python SDK facade and CLI.
- [`src/fi/evals`](https://github.com/future-agi/agent-learning-kit/blob/main/src/fi/evals): active `ai-evaluation` engine code.
- [`src/fi/simulate`](https://github.com/future-agi/agent-learning-kit/blob/main/src/fi/simulate): migrated `simulate-sdk` engine code.
- [`src/fi/opt`](https://github.com/future-agi/agent-learning-kit/blob/main/src/fi/opt): migrated `agent-opt` engine code.
- [`typescript/agent-learning-kit`](https://github.com/future-agi/agent-learning-kit/blob/main/typescript/agent-learning-kit): public
  TypeScript package, including the active evaluation SDK source.
- [`docs/index.md`](https://github.com/future-agi/agent-learning-kit/blob/main/docs/index.md): full documentation index.
- [`ROADMAP.md`](https://github.com/future-agi/agent-learning-kit/blob/main/ROADMAP.md): public v1 roadmap and post-v1 extensions.
- [`LIBRARIES.md`](https://github.com/future-agi/agent-learning-kit/blob/main/LIBRARIES.md): source map for the consolidated engines.
- [`CONTRIBUTING.md`](https://github.com/future-agi/agent-learning-kit/blob/main/CONTRIBUTING.md): local development and PR workflow.
- [`SECURITY.md`](https://github.com/future-agi/agent-learning-kit/blob/main/SECURITY.md): vulnerability reporting policy.
- [`LICENSE`](https://github.com/future-agi/agent-learning-kit/blob/main/LICENSE): Apache-2.0 license.
- [`NOTICE`](https://github.com/future-agi/agent-learning-kit/blob/main/NOTICE): Apache notice metadata.

## Development

New public SDK development belongs here. See [DEVELOPMENT.md](https://github.com/future-agi/agent-learning-kit/blob/main/DEVELOPMENT.md)
for the boundary between this package and the backing engine repos.

```bash
uv sync
uv run ruff check .
uv run pytest -q
uv run python -m build
pnpm --dir typescript --filter @future-agi/agent-learning-kit build
pnpm --dir typescript --filter @future-agi/agent-learning-kit test -- --runInBand
```

For the heavier release cut, run `agent-learn release-proof --project-root .`.
It emits `agent-learning.release-proof.v1` with command evidence for the full
local proof stack.

Before a release:

```bash
uv run python -m fi.alk.cli release-proof \
  --project-root . \
  --output /tmp/agent-learning-release-proof.json \
  --quiet
```

`release-proof` includes release-check, full-repo ruff, pytest, Python package
build, TypeScript package build/test, and `git diff --check`. Use
`--only <check>` for partial proof during development or `--dry-run` to emit the
exact command plan without executing commands.

## Project Status

The v1 release gate is local-first and executable. It covers SDK consolidation,
promptfoo-style CLI usage, native optimizer evidence, docs/examples, schema
kinds, packaging metadata, red-team corpus/campaign coverage, Future AGI
UI/action/report artifacts, framework/provider compatibility, environment
robustness, regression replay, and release proof.

All v1 gates are green on the proved release commit (see the release-proof
artifact). Roadmap milestones marked "mostly complete" or "in progress" are
extend-only: the v1 contract those gates assert is frozen and proved; the named
extensions land post-v1 without weakening any gate.

## Community

- Contributions: [CONTRIBUTING.md](https://github.com/future-agi/agent-learning-kit/blob/main/CONTRIBUTING.md)
- Code of conduct: [CODE_OF_CONDUCT.md](https://github.com/future-agi/agent-learning-kit/blob/main/CODE_OF_CONDUCT.md)
- Security reports: [SECURITY.md](https://github.com/future-agi/agent-learning-kit/blob/main/SECURITY.md)
- License: [Apache-2.0](https://github.com/future-agi/agent-learning-kit/blob/main/LICENSE)

## Deep Dive

The full documentation set — quickstarts, per-track guides, framework pages,
and reference material — starts at [docs/index.md](https://github.com/future-agi/agent-learning-kit/blob/main/docs/index.md).
