Metadata-Version: 2.5
Name: benchflow
Version: 0.7.4
Summary: Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.
Project-URL: Homepage, https://github.com/benchflow-ai/benchflow
Project-URL: Repository, https://github.com/benchflow-ai/benchflow
Project-URL: Issues, https://github.com/benchflow-ai/benchflow/issues
Project-URL: Discord, https://discord.gg/mZ9Rc8q8W3
Project-URL: Changelog, https://github.com/benchflow-ai/benchflow/blob/main/CHANGELOG.md
Author-email: Xiangyi Li <xiangyi@benchflow.ai>, Kyoung Whan Choe <choe.kyoung@gmail.com>
Maintainer-email: Xiangyi Li <xiangyi@benchflow.ai>, Kyoung Whan Choe <choe.kyoung@gmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: acp,agent-evaluation,benchmark,llm-agents,multi-turn,skillsbench
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.12
Requires-Dist: agent-client-protocol>=0.10
Requires-Dist: anyio>=4.0
Requires-Dist: google-cloud-aiplatform<2.0,>=1.133.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: litellm[proxy]==1.91.0
Requires-Dist: packaging>=24
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: tomli-w>=1.0
Requires-Dist: typer>=0.9
Provides-Extra: azure
Requires-Dist: azure-identity>=1.16; extra == 'azure'
Requires-Dist: azure-storage-blob>=12.19; extra == 'azure'
Provides-Extra: azure-service
Requires-Dist: azure-data-tables>=12.5; extra == 'azure-service'
Requires-Dist: azure-identity>=1.16; extra == 'azure-service'
Requires-Dist: azure-storage-blob>=12.19; extra == 'azure-service'
Requires-Dist: azure-storage-queue>=12.9; extra == 'azure-service'
Requires-Dist: uvicorn>=0.30; extra == 'azure-service'
Provides-Extra: bedrock
Requires-Dist: boto3>=1.40; extra == 'bedrock'
Provides-Extra: deepagents
Requires-Dist: deepagents>=0.6; extra == 'deepagents'
Requires-Dist: langchain-openai>=0.2; extra == 'deepagents'
Provides-Extra: dev
Requires-Dist: pathspec>=0.12; extra == 'dev'
Requires-Dist: pre-commit>=3.7; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24.0; extra == 'dev'
Requires-Dist: pytest>=9.0.3; extra == 'dev'
Requires-Dist: ruff>=0.7.0; extra == 'dev'
Requires-Dist: ty>=0.0.1a1; extra == 'dev'
Provides-Extra: judge
Requires-Dist: anthropic>=0.40; extra == 'judge'
Requires-Dist: google-genai>=1.0; extra == 'judge'
Requires-Dist: openai>=1.40; extra == 'judge'
Provides-Extra: sandbox-agentcore
Requires-Dist: bedrock-agentcore>=1.18; extra == 'sandbox-agentcore'
Requires-Dist: boto3>=1.43.31; extra == 'sandbox-agentcore'
Requires-Dist: cryptography>=44; extra == 'sandbox-agentcore'
Requires-Dist: pathspec>=0.12; extra == 'sandbox-agentcore'
Provides-Extra: sandbox-daytona
Requires-Dist: daytona>=0.184.0; extra == 'sandbox-daytona'
Requires-Dist: tenacity>=8.0; extra == 'sandbox-daytona'
Provides-Extra: sandbox-modal
Requires-Dist: modal>=0.73; extra == 'sandbox-modal'
Requires-Dist: tenacity>=8.0; extra == 'sandbox-modal'
Provides-Extra: train
Requires-Dist: transformers==5.6.2; extra == 'train'
Provides-Extra: trl
Requires-Dist: datasets>=2.19.0; extra == 'trl'
Requires-Dist: trl<2,>=1.8.0; extra == 'trl'
Description-Content-Type: text/markdown

<div align="center">
  <h1>BenchFlow</h1>
  <p>The universal environment framework — a benchmark is just a frozen environment.</p>
  <a href="https://pypi.org/project/benchflow/" target="_blank">
    <img src="https://img.shields.io/badge/PyPI-benchflow-3775A9?style=for-the-badge&logo=pypi&logoColor=white" alt="PyPI package">
  </a>
  <a href="https://discord.gg/mZ9Rc8q8W3" target="_blank">
    <img src="https://img.shields.io/badge/Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="Discord">
  </a>
</div>

## What

BenchFlow is a universal environment framework: it runs AI agents against task environments and scores them through one hardened contract. **A benchmark is just a frozen environment** — point BenchFlow at any of them, drive it with *any* ACP agent, and run single-agent, multi-agent, or multi-round patterns over the same Scene-based lifecycle.

## Quick start: 1. Submit a trajectory

**Send these to your coding agent.** Copy
the block below and paste it as a message to Claude Code, Codex, OpenCode, or
Cursor. That is the $2,000 eval prize path. No BenchFlow account, API key, or
Azure login.

> Submit my relevant local Claude Code, Codex, OpenCode, or Cursor session from the re:Agent e2e agentic science / ai4bio hackathon (last 72 hours, on this laptop) to the BenchFlow eval prize.
>
> 1. First make sure the latest benchflow CLI is installed: uv tool install --python 3.12 --upgrade --force benchflow
>
> 2. Then read https://raw.githubusercontent.com/benchflow-ai/benchflow/main/.agents/skills/benchflow-traj-upload/SKILL.md and follow it: find a session, open the viewer, and upload only after I (the human) review it.

The agent finds sessions on this machine, opens the viewer, and uploads after
you like what you see.

The prompt has the agent install the latest BenchFlow first (`uv tool
install --python 3.12 --upgrade --force benchflow`). The `bench traj` commands
also print a one-line upgrade hint when a newer release is available.

Prefer the terminal instead? The guided upload inspects before anything leaves
your machine — it renders a redacted trajectory report (step counts, masked
secrets, preview) and asks for confirmation:

```bash
uv tool install --python 3.12 --upgrade benchflow
bench traj upload
```

Detected secret values are replaced locally with
`<XXX-benchflow-key-values-XXX>` before upload, and the full redacted report is
retained in the uploaded `manifest.json`. See the
[upload skill](./.agents/skills/benchflow-traj-upload/SKILL.md) or the
[trajectory upload guide](./docs/traj-upload.md).

Optional — set the skill up once, then keep talking to the agent:

```bash
npx skills add benchflow-ai/benchflow --skill benchflow-traj-upload
# or, if BenchFlow is already installed:
bench traj setup
```

`npx skills add` asks which agents to install for. `bench traj setup` copies
the skill into this project and prints the same agent prompt. See the
[upload skill](./.agents/skills/benchflow-traj-upload/SKILL.md).

## Quick start: 2. Run with a ChatGPT or Claude subscription

No OpenAI or Anthropic API key is required. Start Docker, install BenchFlow,
then run **one** of these options. BenchFlow detects the saved host login and
makes it available to the agent inside the sandbox.

```bash
uv tool install --python 3.12 --upgrade benchflow
docker info >/dev/null  # Docker must be running
```

### ChatGPT subscription via Codex

Install the [Codex CLI](https://github.com/openai/codex), then:

```bash
codex login
unset OPENAI_API_KEY CODEX_API_KEY  # ensure subscription auth is used

bench eval run \
  --source-repo benchflow-ai/skillsbench \
  --source-path tasks/citation-check \
  --agent codex \
  --model gpt-5.5 \
  --sandbox docker
```

### Claude subscription via Claude Code

Install [Claude Code](https://code.claude.com/docs/en/quickstart), then:

```bash
claude auth login
unset ANTHROPIC_API_KEY ANTHROPIC_AUTH_TOKEN  # ensure subscription auth is used

bench eval run \
  --source-repo benchflow-ai/skillsbench \
  --source-path tasks/citation-check \
  --agent claude \
  --model claude-sonnet-4-6 \
  --sandbox docker
```

The agent may pass or fail the benchmark task; either result means the
evaluation completed. Each run writes rewards, token usage, and the full
trajectory under `jobs/`. See [Getting started](./docs/getting-started.md) for
other agents, models, and sandboxes.

## Install

Install or upgrade to the latest stable release from PyPI with `uv`:

```bash
uv tool install --python 3.12 --upgrade benchflow
```

- Confirm with `bench --version`.
- BenchFlow CLI releases require Python 3.12 or newer. Keep `--python 3.12`
  in the install command so `uv` does not resolve an older Python-compatible
  package that lacks the CLI entrypoints.
- If you see `Executables already exist: bench, benchflow`, re-run with `uv tool install --python 3.12 --upgrade --force benchflow` to replace stale entrypoints from an older install.
- For Daytona, Modal, or AgentCore extras, install the relevant optional package, for example `uv tool install --python 3.12 --upgrade 'benchflow[sandbox-daytona]'`.

Internal users wanting the newest preview from `main` install the [internal preview channel](./docs/release.md) (`uv tool install --python 3.12 --prerelease allow --upgrade benchflow`).

**Requirements & auth.** Install [uv](https://docs.astral.sh/uv/); the
`--python 3.12` flag lets it provision a compatible interpreter for the tool
install. Set `DAYTONA_API_KEY` for Daytona or configure Modal auth for Modal;
export an agent API key (`GEMINI_API_KEY`, `ANTHROPIC_API_KEY`, …) or use
subscription auth (`claude auth login` / `codex login`). Provider-prefixed models
may need provider-specific credentials; Azure Foundry uses `AZURE_API_KEY` +
`AZURE_API_ENDPOINT`.

## Documentation

Start with [Getting started](./docs/getting-started.md), then [Concepts](./docs/concepts.md) for the mental model. Prefer to have an AI coding agent run the whole quickstart for you? Paste the [agent quickstart prompt](./docs/agent-quickstart.md) into Claude Code, Codex CLI, or Gemini CLI. Then by goal:

| If you want to… | Read |
|------------------|------|
| Run an eval on an existing task | [Getting started](./docs/getting-started.md) |
| Understand how BenchFlow runs *any* benchmark (the three-layer model) | [Run any benchmark](./docs/running-any-benchmark.md) |
| Have an AI agent install + run the quickstart end to end | [Agent quickstart prompt](./docs/agent-quickstart.md) |
| Run an agent from the public agents repo (goose, qwen-code, prime-agent, …) | [Running external agents](./docs/external-agents.md) |
| Understand Rollout / Scene / Role / Verifier | [Concepts](./docs/concepts.md) |
| Author a new task | [Task authoring](./docs/task-authoring.md) |
| Author a task in the native `task.md` format | [Native task.md authoring](./docs/task-authoring-task-md.md) |
| Run a hosted PrimeIntellect / Verifiers environment | [CLI reference](./docs/reference/cli.md) |
| Multi-agent: coder + reviewer, simulated user, BYOS, stateful envs | [Use cases](./docs/use-cases.md) |
| Multi-round single-agent (progressive disclosure, oracle access) | [Progressive disclosure](./docs/progressive-disclosure.md) |
| Skill evaluation (when the artifact is a skill, not a workspace) | [Skill eval](./docs/skill-eval.md) |
| Contribute a trajectory capture | [Trajectory upload](./docs/traj-upload.md) |
| Understand the security model | [Sandbox hardening](./docs/sandbox-hardening.md) |
| Use public vs internal preview SDK releases | [Release channels](./docs/release.md) |
| CLI flags + commands | [CLI reference](./docs/reference/cli.md) |
| Python API surface | [Python API reference](./docs/reference/python-api.md) |

Notebooks and runnable example scripts live under [`docs/examples/`](./docs/examples/) so examples stay versioned with the docs that explain them.

> **`bench agent` vs `bench eval adopt`.** `bench agent list` / `bench agent show`
> inspect **registered AI agents** (the solver programs like Claude Code or
> Gemini CLI). Onboarding a third-party benchmark into `benchmarks/<name>/` is a
> separate workflow — `bench eval adopt <source>` scaffolds and drives the
> conversion, and `bench eval adopt <name> --verify` parity-gates it. (The legacy
> `bench agent create|run|verify` commands still work as deprecated aliases.)
> See the [CLI reference](./docs/reference/cli.md#bench-eval-adopt) for details.

## Benchmark task sources

Benchmark datasets live in external Git repos and are referenced with two fields:

```yaml
# benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml
source:
  repo: benchflow-ai/benchmarks    # GitHub org/repo
  path: datasets/harvey-lab/tasks  # optional subpath within repo
  ref: main                         # optional branch/tag
agent: gemini
model: gemini/gemini-3.1-flash-lite-preview
```

Run any benchmark via the CLI:

```bash
# From a YAML config (shipped with the repo)
bench eval run --config benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml

# Inline — mirrors the YAML source fields
bench eval run \
    --source-repo benchflow-ai/skillsbench --source-path tasks \
    --agent gemini --model gemini-3.1-flash-lite-preview --sandbox daytona --concurrency 64
```

Repos are cloned and cached locally under `.cache/datasets/` on first use.

Hosted environments are another source type. Instead of a repo, pass
`--source-env` with the environment's pinned source version to run an external
PrimeIntellect / Verifiers environment on its own native harness — BenchFlow
preserves the hosted identity (`env_uid`, `hub_url`) and still writes the shared
rollout output contract. See the [CLI reference](./docs/reference/cli.md) for
the full hosted-environment command shape.

Downstream projects should depend on the public PyPI release by default. For
internal validation before the next public release, install or lock the internal
preview channel with prereleases enabled; see [Release channels](./docs/release.md).

## Authoring tasks

A task is one `task.md` (YAML frontmatter for config + a markdown prompt body)
plus `environment/` and `verifier/` sidecars. The `bench tasks` commands cover
the authoring lifecycle:

```bash
bench tasks init my-task                 # scaffold a task.md package under tasks/
bench tasks check tasks/my-task          # validate (default --level structural)
bench tasks migrate legacy-task/ --remove-legacy  # convert old split packages to task.md
bench tasks export tasks/my-task out/             # write a compatibility export + loss report
```

See [Native task.md authoring](./docs/task-authoring-task-md.md) and the
[task standard](./docs/task-standard.md).

## Featured

- **Progressive disclosure on SWE-bench Pro** — the `BaseUser` abstraction drives a multi-round rollout: terse round-0 prompt → failing-test hints → full spec. 5/5 oracle on Daytona, runnable demo at [`docs/examples/swebench_pro_progressive_disclosure.ipynb`](./docs/examples/swebench_pro_progressive_disclosure.ipynb). See [Progressive disclosure](./docs/progressive-disclosure.md).

## Audience

- **Eval researchers / paper writers** → [Getting started](./docs/getting-started.md) → [Concepts](./docs/concepts.md) → [Use cases](./docs/use-cases.md)
- **Task authors** → [Task authoring](./docs/task-authoring.md) → [Sandbox hardening](./docs/sandbox-hardening.md)
- **Agent builders integrating with benchflow** → [Concepts](./docs/concepts.md) → [Python API reference](./docs/reference/python-api.md) → [`benchflow.agents.registry`](./src/benchflow/agents/registry.py)
- **External benchmark adapters** → [Task authoring](./docs/task-authoring.md) → [Progressive disclosure](./docs/progressive-disclosure.md#comparison-with-multi-agent-simulated-user)

## Contributing

PRs welcome. Open against `main`. CI runs ruff + tests on every PR; please run `ruff check .` and `pytest tests/` locally first.

Release channels are documented in [Release channels](./docs/release.md). In
short: merges to `main` publish an internal preview after CI passes, while a
matching release tag publishes the public release.

## License

Apache-2.0.
