Metadata-Version: 2.5
Name: pytest-skill-engineering
Version: 1.0.1
Summary: Reliable Copilot execution and evidence for coding-agent-led testing of MCP servers, CLI tools, skills, and custom agents.
Project-URL: Homepage, https://github.com/sbroenne/pytest-skill-engineering
Project-URL: Repository, https://github.com/sbroenne/pytest-skill-engineering
Author: Stefan Brunner
License: MIT
License-File: LICENSE
Keywords: agents,ai,custom-agents,llm,mcp,pytest,skill-engineering,skills,testing
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: github-copilot-sdk>=1.0.16
Requires-Dist: httpx>=0.28.1
Requires-Dist: mcp>=2.2.0
Requires-Dist: pydantic>=2.13.5
Requires-Dist: pytest-asyncio>=1.4.0
Requires-Dist: pytest>=9.1.1
Requires-Dist: python-frontmatter>=1.3.0
Requires-Dist: tomlkit>=0.15.1
Requires-Dist: trio>=0.34.0
Requires-Dist: websockets>=17.1
Provides-Extra: dev
Requires-Dist: pre-commit>=4.6.2; extra == 'dev'
Requires-Dist: pyright>=1.1.414; extra == 'dev'
Requires-Dist: pytest-cov>=7.1.0; extra == 'dev'
Requires-Dist: python-dotenv>=1.2.3; extra == 'dev'
Requires-Dist: ruff>=0.16.9; extra == 'dev'
Requires-Dist: typeguard>=4.6.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: cairosvg>=2.9.1; extra == 'docs'
Requires-Dist: mkdocs-material<9.8,>=9.7.7; extra == 'docs'
Requires-Dist: mkdocs>=1.6.1; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=1.0.6; extra == 'docs'
Requires-Dist: pillow>=12.3.0; extra == 'docs'
Provides-Extra: test
Requires-Dist: syrupy<7,>=6.1.1; extra == 'test'
Description-Content-Type: text/markdown

# pytest-skill-engineering

[![PyPI version](https://img.shields.io/pypi/v/pytest-skill-engineering)](https://pypi.org/project/pytest-skill-engineering/)
[![CI](https://github.com/sbroenne/pytest-skill-engineering/actions/workflows/ci.yml/badge.svg)](https://github.com/sbroenne/pytest-skill-engineering/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

**A test runner for people working with coding agents.**

Test whether real GitHub Copilot sessions can use your MCP servers, CLI tools,
skills, system prompts, and custom agents. Write a user task, execute it through
pytest, and verify the actual output with ordinary assertions.

The framework runs the task and records what happened. Your existing coding agent
investigates the evidence, reads the source, and helps you improve the interface.
**Version 1.0 removes AI judging, report dashboards, rankings, and skill refinement.**
See the [migration guide](docs/migration.md) for this breaking change.

## Why this exists

Your tool can pass its own tests while Copilot chooses the wrong operation,
supplies the wrong arguments, or misunderstands its descriptions. That is a
different question from whether the tool implementation works.

Use real task execution to test the AI-facing interface, and independent output
checks to establish correctness. A completed session or a tool call by itself
does not prove that the requested task succeeded.

## Quick start

You need Python 3.11+, [uv](https://docs.astral.sh/uv/), and a GitHub account with
Copilot access.

```powershell
uv add pytest-skill-engineering
gh auth login --hostname github.com
uv run pytest-skill-engineering init
uv run pytest-skill-engineering doctor
uv run python -m pytest "tests\test_copilot_eval.py" -v
```

`init` creates a Todo MCP starter test, explicit JSON evidence settings, and
explicit starter-model rates in `pricing.toml`. It refuses conflicting settings
or an existing starter. Live execution can consume Copilot premium requests.

pytest shows ordinary test results and assertion failures. Give your coding
agent `aitest-reports\results.json` and the relevant source to investigate what
happened. No dashboard or second AI judgment is generated.

## What we learned by testing our own idea

We evaluated a companion skill for coding agents using this framework and chose
not to ship it. The comparison did not demonstrate a benefit; adding general
test-writing advice was not evidence that users needed another product layer.
Use the documentation, examples, and native evidence directly.

The [case study](docs/use-cases/companion-skill.md) records the actual results,
failures, limitations, and removal decision. Its
[historical experiment](examples/skill-dogfood/) remains reproducible with a
frozen test fixture, not an installable companion skill. Default sample checks
are offline; live execution is explicit. The [quickstart](examples/quickstart/)
is the smaller first example.

Testing your own domain skills remains a supported framework capability.

## A test that checks the result

```python
from __future__ import annotations

import subprocess
import sys

from pytest_skill_engineering.copilot import CopilotEval


async def test_addition(copilot_eval, tmp_path):
    agent = CopilotEval(
        name="addition",
        model="gpt-5.6-luna",
        instructions="Write Python code and save the requested file.",
        working_directory=str(tmp_path),
    )
    result = await copilot_eval(agent, "Create calc.py with add(a, b) returning a + b.")
    assert result.success, result.error
    assert (tmp_path / "calc.py").is_file()
    checked = subprocess.run(
        [
            sys.executable,
            "-c",
            "from calc import add; assert add(2, 3) == 5; assert add(-2, 2) == 0",
        ],
        cwd=tmp_path,
        capture_output=True,
        text=True,
        timeout=10,
    )
    assert checked.returncode == 0, checked.stderr
```

Use your approved isolation when executing generated code. A temporary directory
is not a security sandbox.

## What stays in the framework

| Capability | What it provides |
| --- | --- |
| `CopilotEval` and `copilot_eval` | Real SDK sessions with explicit configuration |
| MCP, CLI, skills, plugins, custom agents | The interfaces and definitions under test |
| Execution controls | Time, usage, request, tool, and permission controls; one attempt per execution |
| Captured evidence | Configuration, calls, arguments, outputs, completion flags, errors, and usage |
| Ordinary pytest checks | Consumer-owned verification and recorded properties |
| `ab_run` and repetitions | Isolated working directories and repeated observations |
| Structured JSON | Captured execution and ordinary pytest outcomes, without rankings or advice |
| Pricing | Explicit USD estimates and recorded Copilot premium requests |

A/B entries share one pytest outcome. Record side-specific checks; do not read
that shared outcome as independent per-side success or causal improvement.
Missing prices and incomplete evidence are recorded explicitly.

## Let your coding agent investigate

Ask your existing coding agent to inspect a failed test, its saved JSON evidence,
and the source. It can explain the supported cause and help fix it, without
changing the test's success criteria after seeing the failure.

Reading current schema-4.0 evidence needs no authentication or paid call.
The framework records results; the coding agent interprets them. Subjective
review is advice, not an automatic pytest verdict.

Read the [full documentation](https://sbroenne.github.io/pytest-skill-engineering/).

## License

MIT. Inspired by [agent-benchmark](https://github.com/mykhaliev/agent-benchmark).
