Metadata-Version: 2.4
Name: llm-code-benchmark
Version: 0.2.2
Summary: OpenRouter LLM benchmark for repair-agent and validator-agent model selection.
Author: if-uri
Author-email: Tom Sapletta <tom@sapletta.com>
License-Expression: Apache-2.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.8
Requires-Dist: PyYAML>=6.0.2
Provides-Extra: test
Requires-Dist: build>=1.2; extra == "test"
Requires-Dist: mypy>=1.11; extra == "test"
Requires-Dist: pytest>=8; extra == "test"
Requires-Dist: pytest-cov>=5; extra == "test"
Requires-Dist: ruff>=0.6; extra == "test"
Requires-Dist: types-PyYAML>=6.0; extra == "test"
Dynamic: license-file

# llm-code-benchmark


## AI Cost Tracking

![PyPI](https://img.shields.io/badge/pypi-costs-blue) ![Version](https://img.shields.io/badge/version-0.2.2-blue) ![Python](https://img.shields.io/badge/python-3.9+-blue) ![License](https://img.shields.io/badge/license-Apache--2.0-green)
![AI Cost](https://img.shields.io/badge/AI%20Cost-$2.78-orange) ![Human Time](https://img.shields.io/badge/Human%20Time-10.3h-blue) ![Model](https://img.shields.io/badge/Model-openrouter%2Fqwen%2Fqwen3--coder--next-lightgrey)

- 🤖 **LLM usage:** $2.7790 (45 commits)
- 👤 **Human dev:** ~$1030 (10.3h @ $100/h, 30min dedup)

Generated on 2026-07-19 using [openrouter/qwen/qwen3-coder-next](https://openrouter.ai/qwen/qwen3-coder-next)

---

Private benchmark repository for selecting OpenRouter models for `repair-agent` and `validator-agent`.

The benchmark never stores API keys. Paid runs require `OPENROUTER_API_KEY` from the environment or GitHub Actions Secrets. Standard CI and `catalog-smoke` do not make paid model requests.

```bash
python -m pip install -e ".[test]"
python -m pytest tests -q
llm-code-benchmark catalog-smoke --max-models 2
```

Paid smoke benchmark, after explicit approval and budget setup:

```bash
llm-code-benchmark live-smoke --max-models 2 --max-tasks 2 --repetitions 1 --budget-usd 0.25
```

Recommended next smoke uses explicit models instead of router auto-selection:

```bash
llm-code-benchmark live-smoke --model meta-llama/llama-4-scout,deepseek/deepseek-v4-flash --max-models 2 --max-tasks 2 --repetitions 1 --budget-usd 0.25 --dry-run
```

Remove `--dry-run` only after explicit paid-run approval.

Reports are written to `reports/latest/` and immutable snapshots under `reports/history/`. Repair schema validation is intentionally separate from semantic patch validation, so path mismatches and invalid diffs are reported as repair task statuses rather than JSON schema failures.

## Repair Output Modes

Repair benchmark requests now prefer `file_edits`: the model returns repository-relative paths and complete final file contents, and the benchmark generates a deterministic unified diff locally before validation and `git apply --check`. Native `patch` output remains supported and is scored separately through `native_patch_success_rate`; structured edits are tracked through `structured_edit_success_rate`. `search_replace`, `create`, and `delete` are not enabled in this version.


## License

Licensed under Apache-2.0.
