Metadata-Version: 2.4
Name: retrybench
Version: 0.3.0
Summary: A conformance benchmark for safe recovery from ambiguous tool outcomes in LLM agents
Author: RetryBench contributors
License-Expression: PolyForm-Noncommercial-1.0.0
Project-URL: Homepage, https://github.com/LIUYOU2015/retrybench
Project-URL: Documentation, https://github.com/LIUYOU2015/retrybench/blob/main/docs/cross-model-evaluation.md
Project-URL: Issues, https://github.com/LIUYOU2015/retrybench/issues
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: inspect
Requires-Dist: inspect-ai==0.3.281; extra == "inspect"
Requires-Dist: openai<4,>=3.4.0; extra == "inspect"
Requires-Dist: anthropic<3,>=1.0.0; extra == "inspect"
Provides-Extra: langgraph
Requires-Dist: langgraph==0.6.11; extra == "langgraph"
Provides-Extra: langchain
Requires-Dist: langchain-core==0.3.86; extra == "langchain"
Provides-Extra: figures
Requires-Dist: matplotlib==3.9.4; extra == "figures"
Requires-Dist: numpy==2.0.2; extra == "figures"
Provides-Extra: paper
Requires-Dist: cairosvg==2.8.2; extra == "paper"
Requires-Dist: pypandoc-binary==1.17; extra == "paper"
Requires-Dist: PyMuPDF==1.26.5; extra == "paper"
Requires-Dist: matplotlib==3.9.4; extra == "paper"
Requires-Dist: numpy==2.0.2; extra == "paper"
Dynamic: license-file

# RetryBench

Evaluate whether an LLM's first recovery decision after a tool timeout is
permitted, and whether replaying that decision achieves the intended state.

## Install and run

Python 3.11+ for model inference:

```bash
pip install "retrybench[inspect]"
retrybench run --model mockllm/retrybench-defer
```

The local mock runs all 240 cases without credentials or inference charges. It
always defers: unsupported retries **0/240**, permitted progress **0/80**, and
unresolved outcomes **480/480**. These are synthetic fixture results.

For a real model, set its provider's API credentials and pass a provider/model
identifier supported by [Inspect AI](https://inspect.aisi.org.uk/models.html):

```bash
retrybench run --model "$MODEL_ID"
```

The `inspect` extra includes the OpenAI and Anthropic clients. Other providers
may require their own SDK; follow Inspect's provider setup instructions.

This makes up to 240 billable inference calls with a 256-token output limit per
call and no runner retries. `--dry-run` shows the call count without inference;
`--limit 4` runs a diagnostic subset and reports the remaining cases as missing.
Temperature and reasoning effort use provider defaults unless supplied.
Each run prints its result directory and writes predictions, per-case scores,
and a report. Existing runs are never overwritten. Partial runs exit with code 2.

## Use any inference framework

The core scorer supports Python 3.9+ with no third-party runtime dependencies:

```bash
pip install retrybench
retrybench export --output cases.jsonl
# Generate one prediction per visible case using your model or framework.
retrybench score --predictions predictions.jsonl --model my-model
```

Predictions are JSONL records such as:

```json
{"case_id":"<exact exported case_id>","action":"unresolved","arguments":{"idempotency_key":null}}
```

Use `retrybench run` without `--model` for the deterministic policy baselines.
The earlier `retrybench eval --model ...` command remains supported.

## Data and interpretation

`retry-permission-v1` freezes 240 visible decisions and 480 correlated replayed
outcomes across four reused software tasks. Reports separate unsupported retry
permission, permitted progress, protocol validity, duplicates, omissions, and
unresolved recovery, with exact denominators and missing/invalid accounting.
The labeled suite is public and is not a held-out generalization test. Each model
receives only visible prompts and output schemas; keep scoring labels outside
inference. No real side-effecting tool is executed.

The wheel includes the complete, hash-checked suite. A clone, dataset download,
and the paper's internal validation history are unnecessary for evaluation.
The separate research archive supports reanalysis of historical paper studies.

[Source](https://github.com/LIUYOU2015/retrybench) ·
[Evaluation guide](https://github.com/LIUYOU2015/retrybench/blob/main/docs/cross-model-evaluation.md) ·
[Benchmark card](https://github.com/LIUYOU2015/retrybench/blob/main/BENCHMARK_CARD.md) ·
[Citation](https://github.com/LIUYOU2015/retrybench/blob/main/CITATION.cff)

Code and suite are distributed under
[PolyForm Noncommercial 1.0.0](https://github.com/LIUYOU2015/retrybench/blob/main/LICENSE).

Version archive DOI: [10.5281/zenodo.24044371](https://doi.org/10.5281/zenodo.24044371).
