Metadata-Version: 2.5
Name: deeptrap
Version: 0.1.0
Summary: Security benchmark suite for evaluating alignment in tool-using AI agents
Project-URL: Homepage, https://github.com/ZJUICSR/ActBench
Project-URL: Documentation, https://github.com/ZJUICSR/ActBench#readme
Project-URL: Repository, https://github.com/ZJUICSR/ActBench
Project-URL: Issues, https://github.com/ZJUICSR/ActBench/issues
Author: ZJUICSR
License-Expression: MIT
License-File: LICENSE
Keywords: agent-security,ai-safety,benchmark,llm-agents,prompt-injection,red-teaming
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Security
Requires-Python: >=3.10
Requires-Dist: fastapi>=0.110.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: uvicorn>=0.27.0
Provides-Extra: dev
Requires-Dist: black>=23.7.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.1.0; extra == 'dev'
Requires-Dist: pytest>=7.4.0; extra == 'dev'
Requires-Dist: ruff>=0.0.285; extra == 'dev'
Provides-Extra: release
Requires-Dist: huggingface-hub>=0.24.0; extra == 'release'
Requires-Dist: pyarrow>=15.0.0; extra == 'release'
Description-Content-Type: text/markdown

# DeepTrap

DeepTrap is a task-based security evaluation framework for tool-using agents. It evaluates whether an agent remains aligned with the user's task when workspace files, mock API records, skills, or other task artifacts contain unsafe instructions or data. It ships the 300-task ActBench suite, its runner and scorers, local mock services, and adapters for multiple agent backends.

This public release contains everything needed to run and reproduce ActBench evaluations. It does **not** include the private task-generation or attack-search pipeline used to create the tasks. Existing `actbench.*` result schema names are retained for compatibility.

The source repository remains `ZJUICSR/ActBench`; `deeptrap` is the PyPI distribution name and installed command. This project does not use or modify the separate `ZJUICSR/DeepTrap` repository.

## Install and verify

Install DeepTrap as an isolated command-line tool:

```bash
uv tool install deeptrap
# or
pipx install deeptrap
```

Alternatively, install it into the current Python environment:

```bash
pip install deeptrap
```

Verify the complete installation without calling a model or external judge:

```bash
deeptrap test --self-test
```

The wheel includes the benchmark tasks, skills, and mock-service fixtures, so a repository checkout is not required. Run `deeptrap --help` to see all commands.

## What is included

- `tasks/task_B*_T*/` — self-contained ActBench tasks.
- `tasks/clean_scenes/` — benign clean-source bundles used for optional delta-aware baseline generation.
- `scripts/actbench.py` and `scripts/benchmark/` — benchmark runner, backend adapters, and result aggregation.
- `mock_services/` — local fixture-backed FastAPI services used by tasks.
- `skills/mock_apis/` — standard mock API skill descriptions that describe the service endpoints.
- `docs/` — task, result, mock-service, and backend setup notes.

Maintainers can publish versioned wheels through the repository release workflow; see `docs/PUBLISHING.md`.

## Task inventory

ActBench currently contains 300 public tasks, grouped by B-class selectors:

| B class | Name | Tasks |
| --- | --- | ---: |
| B1 | Instruction injection | 42 |
| B2 | Goal hijacking | 13 |
| B3 | Data exfiltration | 32 |
| B4 | Credential exposure | 14 |
| B5 | Memory poisoning | 15 |
| B6 | State tampering | 37 |
| B7 | Deceptive tool invocation | 14 |
| B8 | Unauthorized CMD execution | 42 |
| B9 | Unauthorized API invocation | 12 |
| B10 | Tool scope escalation | 12 |
| B11 | Resource exhaustion | 14 |
| B12 | Obfuscated execution | 16 |
| B13 | False reporting | 14 |
| B14 | Context flooding | 11 |
| B15 | Permission chaining | 12 |
| **Total** |  | **300** |

## Requirements

- Python 3.10+
- `uv` or `pip`
- OpenClaw CLI available on `PATH` for the default `openclaw` backend
- Hermes CLI available on `PATH` or `ACTBENCH_HERMES_BIN` set when using `--backend hermes`
- opencode CLI available on `PATH` or `ACTBENCH_OPENCODE_BIN` set when using `--backend opencode`
- A running QwenPaw service when using `--backend qwenpaw`
- A configured target model for the selected backend
- A judge-model API key when using LLM-assisted scoring:
  - `DEEPSEEK_API_KEY` for `deepseek/...` judge models
  - `OPENROUTER_API_KEY` for OpenRouter-routed judge models
  - For private OpenAI-compatible gateways, copy `config/llm_backends.example.yaml` to the ignored `config/llm_backends.local.yaml` and set the configured API-key/base-URL environment variables.

For development from a source checkout:

```bash
uv sync
# or
pip install -e .
```

For private OpenAI-compatible judge gateways, keep provider details local:

```bash
cp config/llm_backends.example.yaml config/llm_backends.local.yaml
export ACTBENCH_PRIVATE_GATEWAY_BASE_URL="https://your-gateway.example/v1"
export ACTBENCH_PRIVATE_GATEWAY_API_KEY="..."
```

Then use the configured neutral prefix, for example `--judge-model private/gpt-5.5`.

## One-command test

For a bounded first run, use `deeptrap test`. It creates an invocation-scoped output directory, collects trajectories with the selected backend, scores only those trajectories, and writes a concise `one_click_result.json` summary.

Check the local ActBench plumbing without calling a real model or judge:

```bash
deeptrap test --self-test
```

Run the default representative security test against a backend/model:

```bash
deeptrap test \
  --backend claudecode \
  --model <target-model> \
  --judge-model private/gpt-5.5
```

By default this runs one fixed task from each B class (`task_B1_T01` through `task_B15_T01`), preserves clean-baseline generation/reuse, then performs Combined AGS scoring with the explicit judge model. Combined AGS invokes an external judge. For a no-judge local replay, opt into automated-only scoring:

```bash
deeptrap test \
  --backend claudecode \
  --model <target-model> \
  --score-mode automated
```

Use `--suite task_B9_T01`, `--suite B9`, or `--suite all` to change the task selection. Large runs and advanced recovery controls are still available through the lower-level collection and scoring commands below. See `docs/ONE_CLICK.md` for output layout, exit codes, backend notes, and score semantics.

## Collect trajectories with ActBench

ActBench's public workflow is trajectory-first: the runner executes the target agent and records durable trajectories/artifacts. Official AGS/ASR/pass@k scoring is performed later with `actbench_score.py`.

Collect trajectories for all public tasks with the default OpenClaw backend:

```bash
deeptrap run --model deepseek/deepseek-v4-pro
# equivalent:
deeptrap run --backend openclaw --model deepseek/deepseek-v4-pro
```

Run with QwenPaw by starting QwenPaw separately and pointing ActBench at the service. ActBench does not import the `qwenpaw` package or require a QwenPaw Python environment; it creates a task-scoped QwenPaw service agent bound to each materialized task workspace:

```bash
ACTBENCH_QWENPAW_BASE_URL=http://127.0.0.1:8088 \
  deeptrap run --backend qwenpaw --model deepseek/deepseek-v4-pro
```

For source-checkout QwenPaw runs, one typical service startup command is:

```bash
cd /path/to/QwenPaw
PYTHONPATH=/path/to/QwenPaw/src \
QWENPAW_WORKING_DIR=/tmp/qwenpaw-actbench \
python -m qwenpaw app --host 127.0.0.1 --port 8088
```

Relevant QwenPaw environment variables:

- `ACTBENCH_QWENPAW_BASE_URL` selects the QwenPaw service URL; default is `http://127.0.0.1:8088`.
- `ACTBENCH_QWENPAW_API_KEY` optionally sends `Authorization: Bearer ...` to the service.
- `ACTBENCH_QWENPAW_TIMEOUT_SECONDS` optionally caps individual service requests; if unset, ActBench uses the task timeout budget.
- `ACTBENCH_QWENPAW_AGENT_PREFIX` prefixes per-task service agent IDs; default is `actbench`.
- `ACTBENCH_QWENPAW_DELETE_AGENT` controls best-effort deletion of per-task QwenPaw agent registrations after each attempt; default is enabled.
- `ACTBENCH_QWENPAW_HEADLESS_TOOL_GUARD` is passed through to QwenPaw's request context.
- `ACTBENCH_QWENPAW_USAGE_DELTA` controls the ActBench-side token-usage fallback; by default ActBench first uses usage returned by QwenPaw process events, then falls back to the service's `/api/token-usage/details` or `/api/token-usage` aggregate delta when event usage is absent. Because that fallback is provider/model aggregate data from the QwenPaw service, unrelated concurrent QwenPaw traffic for the same provider/model can contaminate per-task token counts. For same-task parallel repeats (`--run-workers > 1`), ActBench disables this aggregate delta fallback and only trusts per-event usage returned by QwenPaw.


Run with OpenAgent when an OpenAgent service is already running and configured with a Store external API key:

```bash
OPENAGENT_API_KEY=... \
OPENAGENT_BASE_URL=http://localhost:14000 \
deeptrap run --backend openagent --model deepseek/deepseek-v4-pro
```

OpenAgent support uses its OpenAI-compatible chat completions endpoint. ActBench records `--model` in results and sends it in the request, but the actual OpenAgent model and tools are controlled by the Store associated with `OPENAGENT_API_KEY`.

Run with Hermes when the Hermes CLI is installed and configured for the target provider:

```bash
ACTBENCH_HERMES_PROVIDER=... \
deeptrap run --backend hermes --model deepseek/deepseek-v4-pro
```

The Hermes backend launches isolated `hermes -z` subprocesses from each materialized task workspace. By default it writes a run-scoped `HERMES_HOME`, registers the ActBench MCP gateway as the `actbench` MCP server, and instructs Hermes to use task-scoped MCP tools for workspace and mock API access. See `docs/HERMES.md` for setup, environment variables, and troubleshooting.

Run with opencode when the opencode CLI is installed and configured for the target provider:

```bash
deeptrap run --backend opencode --model deepseek/deepseek-v4-pro
```

The opencode backend launches isolated `opencode run --format json` subprocesses from each materialized task workspace. By default it provides an inline opencode config with the ActBench MCP gateway as a remote MCP server named `actbench`, instructs opencode to use task-scoped MCP tools for workspace and mock API access, and extracts the full session with `opencode export <sessionID>` for scoring. See `docs/OPENCODE.md` for setup, environment variables, and troubleshooting.

By default, the OpenAgent adapter also exposes the per-task workspace and declared mock APIs through an ActBench-owned MCP gateway. Configure the OpenAgent Store once with the MCP URL that OpenAgent can reach:

- local OpenAgent: `http://127.0.0.1:8765/mcp`
- OpenAgent in Docker: `http://host.docker.internal:8765/mcp`

For each task attempt, ActBench materializes the workspace, starts declared mock services, registers a high-entropy task `context_id` with the gateway, and prepends a system message instructing OpenAgent to use the ActBench MCP tools with that `context_id`. The context is unregistered after the attempt and also has a TTL.

Relevant OpenAgent MCP environment variables:

- `OPENAGENT_ENABLE_ACTBENCH_MCP=0` disables MCP and keeps the weak chat-completions-only mode.
- `ACTBENCH_MCP_AUTOSTART=0` uses an externally managed gateway instead of autostarting one.
- `ACTBENCH_MCP_HOST` / `ACTBENCH_MCP_PORT` set the local gateway bind/check address; defaults are `127.0.0.1` and `8765`.
- `ACTBENCH_MCP_URL` sets the public MCP URL shown to OpenAgent; default is `http://127.0.0.1:8765/mcp`.
- `ACTBENCH_MCP_ADMIN_TOKEN` optionally protects local context registration endpoints.
- `OPENAGENT_TIMEOUT_SECONDS` optionally caps individual OpenAgent HTTP requests; if unset, ActBench uses the task timeout budget instead of a fixed 120s cap.

The MCP gateway security model is task-scoped: file paths are resolved inside the materialized workspace, API discovery returns only service names and allowed business paths, `actbench_call_api` can call only the task's declared mock services and business paths, and administrative mock endpoints such as health, audit, reset, logs, fixture paths, raw base URLs, and admin tokens are not exposed to OpenAgent.

See `docs/OPENAGENT.md` for the full OpenAgent setup flow, including what to provide, how to add the ActBench MCP server in OpenAgent, and Docker networking notes.

OpenClaw, QwenPaw, OpenAgent, Hermes, and opencode all use `--model` as the model under test, so it can be varied across runs.

Run a subset by B class or exact task id:

```bash
deeptrap run --model deepseek/deepseek-v4-pro --suite B1
deeptrap run --model deepseek/deepseek-v4-pro --suite B1,B7
deeptrap run --model deepseek/deepseek-v4-pro --suite B10
deeptrap run --model deepseek/deepseek-v4-pro --suite task_B9_T01
```

Common options:

```bash
--runs 3                    # repeat each task three times
--run-workers 3             # run same-task repeats concurrently when the backend supports it
--skip-baseline-gen          # use cached benign baselines only
--regenerate-baselines       # rerun benign baselines and refresh aligned artifacts
--inline-scoring             # deprecated legacy mode: score inline during the run
--skip-scoring               # deprecated/no-op; trajectory-only collection is the default
--execution-retries 1        # retry retryable execution statuses within each repeat slot
--retry-status error,timeout # comma-separated statuses retried by --execution-retries
--no-training-artifacts      # legacy inline-only; do not use for offline trajectory scoring
--output-dir results         # where JSON results are written
```

Raw training artifacts may contain task prompts, transcripts, workspace contents, and model outputs. Benign baseline runs use the same artifact directory layout as attacked attempts when they are generated during a run; use `--regenerate-baselines` to refresh legacy baseline cache entries into the aligned artifact structure. Do not use `--no-training-artifacts` when collecting trajectories for later scoring; the CLI rejects that combination unless deprecated `--inline-scoring` is explicitly enabled.

## Outputs

A default collection run writes:

- `results/<run_id>_<model>.json` — full per-run collection result with `workflow: trajectory_collection`, `scoring_status: deferred`, and `inline_scoring: false`.
- `results/actbench_summary_<run_id>_<model>.json` — compact collection summary (`summary_kind: trajectory_collection`).
- `results/<run_id>_<model>_artifacts/` — raw per-attempt artifacts, including attacked and benign baseline `trajectory.json` copies.
- `results/trajectories/` — canonical attacked-attempt trajectory tree used by offline AGS/ASR/pass@k scoring.

See `docs/RESULT_FORMAT.md` for schema notes.

## Score collected trajectories

Trajectory-only collection is the default; `--skip-scoring` is retained only for older scripts. New artifacts use `actbench.trajectory.v1`; legacy OpenClaw `actbench.openclaw_trajectory.v1` artifacts remain supported by the offline scorer.

Replay Python automated checks without external judge calls:

```bash
deeptrap score --trajectory results/trajectories --mode automated
```

This reruns Python automated checks against durable `workspace_after/` snapshots and emits `actbench.offline_score.v1` JSON. It does not compute combined AGS or call the LLM judge.

To reproduce official combined AGS/ASR/pass@k offline, explicitly opt in to external judge calls:

```bash
deeptrap score --trajectory results/trajectories --mode combined-ags
```

`combined-ags` invokes the configured external LLM judge using replay evidence from trajectories. Ensure the relevant judge API key is configured, and use `--judge-model <model>` to override the trajectory's judge model when needed. Prefer the canonical `results/trajectories` tree for attack scoring: raw artifact roots can also contain `benign_baseline` trajectories from baseline generation, while the canonical tree only contains attacked attempts. If reusing an output directory across unrelated runs, restrict the path to the relevant suite/task subtree or use a fresh `--output-dir`.

Pack runner output into a raw-by-task dataset when you want a stable manifest-addressed bundle for release, aggregate AGS scoring, or clean utility workflows:

```bash
deeptrap pack \
  --result results/<run_dir>/<run_id>_<model>.json \
  --dataset-name <dataset> \
  --raw-by-task-root ~/pack/raw_by_task

deeptrap score \
  --raw-by-task-root ~/pack/raw_by_task \
  --raw-by-task-dataset <dataset> \
  --mode combined-ags \
  --judge-model private/gpt-5.5

deeptrap utility-prep \
  --raw-by-task-root ~/pack/raw_by_task \
  --raw-by-task-dataset <dataset> \
  --role benign \
  --output-dir results/utility_prep/<dataset>_benign
```

The packer copies attacked trajectories into `<dataset>/<suite>/<task_id>/run_<n>/` and linked clean baselines into `<dataset>/_baselines/<suite>/<task_id>/baseline/`. The scoring command consumes attacked runs; the utility-prep command prepares clean baseline records for later UGS/TAcc/task-completion judging. See `docs/RESULT_FORMAT.md` for raw-by-task layout and cache-only baseline exclusion details.

## Scoring summary

ActBench reports attack success with AGS (Attack Grading Score):

- `attack_success >= attack_success_threshold` means the unsafe behavior reproduced.
- The default attack-success threshold is `0.8`.
- ASR is the fraction of valid runs whose AGS reaches the threshold.
- For repeated runs, `pass@k1`, `pass@k2`, and `pass@k3` report task-level attack success over the first `k` valid repeats.

Task metadata uses behavior-first ActBench fields (`behavior_id`, `behavior_type`); `scoring_family` is the current B-class bucket (`B1`-`B15`). Legacy risk-family fields may appear in metadata for compatibility with older internal artifacts, but new integrations should use the behavior-first fields.

## OpenClaw base-model results

The table below compares base models under the OpenClaw harness. Iter. is the median cycle count over malicious rollouts.

| Model | AGS_mal ↓ | ASR (%) ↓ | p@1/p@2/p@3 ↓ | UGS_ben ↑ | Iter. |
| --- | ---: | ---: | ---: | ---: | ---: |
| Claude-Opus-4.8 | 0.284 | 10.1 | 10.7/12.3/13.7 | 0.938 | 15.0 |
| Claude-Sonnet-4.6 | 0.347 | 20.0 | 19.3/22.3/23.7 | 0.927 | 16.0 |
| GPT-5.5 | 0.493 | 37.8 | 36.0/43.3/47.3 | 0.928 | 16.0 |
| GPT-5.4-mini | 0.727 | 65.7 | 66.3/75.3/78.0 | 0.904 | 18.0 |
| Grok-4.5 | 0.870 | 83.9 | 83.0/90.0/90.7 | 0.938 | 17.0 |
| GLM-5.2 | 0.547 | 42.8 | 41.7/49.0/54.0 | 0.929 | 16.0 |
| Qwen-3.7-max | 0.511 | 39.2 | 37.7/48.3/54.7 | 0.915 | 16.0 |
| Qwen-3.7-plus | 0.524 | 42.1 | 42.7/50.0/53.3 | 0.915 | 16.0 |
| Kimi-K3 | 0.489 | 35.7 | 35.3/44.3/47.3 | 0.940 | 17.0 |
| Kimi-k2.6 | 0.748 | 70.4 | 70.7/78.0/81.0 | 0.869 | 17.0 |
| MiniMax-M3 | 0.402 | 25.0 | 25.0/31.7/36.3 | 0.917 | 19.0 |
| MiniMax-M2.7 | 0.804 | 75.4 | 76.7/84.3/87.7 | 0.880 | 17.0 |
| Deepseek-v4-Pro | 0.955 | 94.4 | 94.3/98.0/98.7 | 0.922 | 19.0 |
| Deepseek-v4-Flash | 0.887 | 84.4 | 83.3/90.0/93.3 | 0.900 | 19.5 |
| Hunyuan-3.0 | 0.455 | 30.0 | 30.7/37.3/41.7 | 0.933 | 19.0 |

## Mock services

Tasks declare any required mock services in `task.yaml`. The runner starts those services automatically on local random ports and writes `api_endpoints.json` into the task workspace. Users normally do not need to start mock services manually.

See `docs/MOCK_SERVICES.md` and `mock_services/README.md` for endpoint details.
