Metadata-Version: 2.4
Name: agent-skill-evals
Version: 0.1.0rc4
Summary: Versioned evaluation, coverage, and release governance for Agent Skills
Author: Andrew Brookins
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: jinja2>=3.1
Requires-Dist: jsonschema>=4.23
Requires-Dist: pydantic>=2.10
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.9
Requires-Dist: typer>=0.15
Description-Content-Type: text/markdown

# Agent Skill Evals

`skill-eval` tests an Agent Skill as a versioned artifact. It runs the same cases with and
without the skill, records what the agent did, grades the evidence, measures skill lift and
constraint coverage, runs adversarial probes, and applies release gates.

The system is offline-first. Its schemas, validation, local and custom runners, deterministic
graders, coverage analysis, reports, gates, SQLite registry, and portfolio summary do not
depend on a hosted service. [Microsoft Waza](https://github.com/microsoft/waza) is the default
agent runner when it is installed; local and custom command adapters are also available.

## Install for development

You need Python 3.11 or newer and [uv](https://docs.astral.sh/uv/).

```console
uv sync --locked --all-groups
uv run skill-eval --version
```

All examples below use `uv run skill-eval`. After installing the project as a tool, you can
use `skill-eval` directly.

To exercise an unpublished checkout from another repository, use `uvx` without installing it
globally:

```console
uvx --from /path/to/agent-skill-evals skill-eval --version
```

After a release is published, pin the package rather than relying on a moving checkout:

```console
uvx --from 'agent-skill-evals==0.1.0rc4' skill-eval --version
```

The release candidate is configured as the MIT-licensed `agent-skill-evals` distribution, authored
by Andrew Brookins and intended for publication from the `abrookins` PyPI account. The Python import
remains `skill_eval`, and the installed command remains `skill-eval`.

To execute the default runner path, install Waza `v0.38.2` and make `waza` available on
`PATH`. The adapter checks the exact version before every run. See the
[pinned integration contract](docs/waza-contract.md) for the verified distributions,
capabilities, exit codes, and known gaps. Waza and any model credentials are separate from
this package.

## The evaluation loop

An eval suite lives beside its skill:

```text
my-skill/
├── SKILL.md
├── skill.yaml                  # optional structured skill metadata
└── evals/
    ├── eval.yaml               # canonical cases, conditions, and graders
    ├── gates.yaml              # release thresholds and waivers
    ├── runner.yaml             # waza, local, or custom execution
    ├── synthesis.yaml          # draft-generation policy
    ├── fixtures/               # inputs copied into isolated workspaces
    ├── graders/                # schemas and grader programs
    ├── probes/                 # adversarial probe catalog
    ├── generated/              # extracted constraints and reviewable drafts
    ├── runs/                   # immutable normalized run evidence
    └── reports/                # JSON, HTML, Markdown, and JUnit output
```

The core loop is:

```text
SKILL.md -> analyze -> author or synthesize -> validate -> run -> grade
         -> coverage -> report -> gate -> registry and portfolio
```

Cases can run under four counterfactual conditions:

- `no_skill`: the target skill is unavailable.
- `with_skill`: the target skill can be retrieved normally.
- `forced_skill`: the harness requires the target skill and checks invocation evidence.
- `with_distractors`: the target appears beside unrelated skills.

Comparing the same case and trial across conditions shows whether the skill helps, has no
effect, or makes the result worse. A result is not release-ready merely because it passes with
the skill: gates can also require positive lift, correct invocation, broad behavior coverage,
stable trials, bounded cost and latency, and no critical security failure.

Conditions that need another skill tree take explicit, domain-neutral source options. Repeat
`--distractor`, `--malicious-distractor`, or `--library-skill` for multiple sources; use
`--previous-skill` and `--stale-skill` for their respective version conditions. For example:

```console
uv run skill-eval run ../my-skill --condition with_distractors \
  --distractor ../unrelated-skill --distractor ../another-skill
```

## Add evals to a skill in under 15 minutes

Start with a skill that already has `SKILL.md`:

```console
uv run skill-eval init ../my-skill
uv run skill-eval analyze ../my-skill
```

`init` is additive and never overwrites existing files. `analyze` writes editable constraints,
trigger examples, and a security-boundary inventory under `evals/generated/` while preserving
source locations.

Next, add one high-value case to `evals/eval.yaml`. Use the generated constraints to give the
case a stable ID, an observable outcome, explicit conditions, and deterministic graders. Copy
needed input files into `evals/fixtures/` and refer to them with relative paths. A good first
suite has at least:

- one normal positive case;
- one edge or malformed-input case;
- one negative trigger that must not invoke the skill;
- one case for each critical security boundary;
- `no_skill` and `with_skill` conditions on utility cases.

The complete sample at
[`examples/skills/csv-summary-skill`](examples/skills/csv-summary-skill) includes normal,
missing-value, injection, distractor, and prose-negative cases with paired conditions, fixtures,
JSON Schema grading, tool restrictions, and release gates.

Validate before spending model time:

```console
uv run skill-eval validate ../my-skill
uv run skill-eval security-probes ../my-skill
```

Then run the suite and use the printed run and grade IDs in the remaining commands:

```console
uv run skill-eval run ../my-skill --conditions no_skill,with_skill --trials 3
uv run skill-eval grade ../my-skill --run-id RUN_ID
uv run skill-eval coverage ../my-skill --run-id RUN_ID --grade-id GRADE_ID
uv run skill-eval report ../my-skill --run-id RUN_ID --grade-id GRADE_ID
uv run skill-eval gate ../my-skill --run-id RUN_ID
```

Open `evals/reports/RUN_ID/report-REPORT_ID/index.html` to review prompts, outputs, artifacts,
formal grades, evidence, previous-run comparisons, and human feedback. The `report` and `gate`
commands print the exact versioned path. Keep the JSON and JUnit files as CI artifacts.

## Optional synthesis and description optimization

The deterministic analyzer can turn extracted constraints into draft case plans. Drafts remain
separate from `eval.yaml` until a person accepts and promotes them:

```console
uv run skill-eval synthesize ../my-skill
uv run skill-eval review ../my-skill --reviewer "$USER"
uv run skill-eval promote ../my-skill --by "$USER"
```

`--seed FILE` imports a canonical seed file and `--waza-task FILE` imports an existing Waza task;
both options may be repeated. Review decisions and promotions are audited. Regenerating analysis
or synthesis never overwrites reviewed artifacts.

Description optimization runs repeated held-out trigger trials through Waza. It reports a
candidate but cannot change `SKILL.md` until a reviewer approves the exact evaluated text:

```console
uv run skill-eval optimize-description ../my-skill --model MODEL --trials 5
uv run skill-eval approve-description ../my-skill CANDIDATE_ID --reviewer "$USER"
uv run skill-eval promote-description ../my-skill CANDIDATE_ID --by "$USER"
```

## Runners

`evals/runner.yaml` selects one runner. `skill-eval run --runner TYPE` and `--model MODEL` can
override the file for one run.

### Waza: the default

New suites use this configuration:

```yaml
schema_version: 1
runner:
  type: waza
  command: waza
  model: null
  env: {}
  timeout_seconds: 600
  max_tokens: 200000
  capture:
    stdout: true
    stderr: true
    filesystem_diff: true
    tool_calls: true
    artifacts: true
    trace: true
```

Set `model` in this file or pass `--model`. The adapter translates representable cases to Waza
schema `1.2`, invokes the exact pinned binary without a shell, and normalizes Waza results into
the canonical trace format. It uses Waza's native no-skill baseline and delegates compatible
graders. Forced-skill checks, distractor semantics, project-specific security boundaries,
coverage, waivers, and multidimensional gates remain canonical.

### Local command

Use `type: local` for an agent command that reads the prompt from standard input and writes its
final response to standard output:

```yaml
schema_version: 1
runner:
  type: local
  command: my-agent --non-interactive
  model: local-model
  env: {}
  timeout_seconds: 600
  max_tokens: 200000
  capture:
    stdout: true
    stderr: true
    filesystem_diff: true
    tool_calls: false
    artifacts: true
    trace: true
```

The command runs directly, not through a shell. It receives `AGENT_MODE`, `AGENT_RUN_ID`,
`AGENT_CASE_ID`, `AGENT_SKILL_ID`, `AGENT_MODEL`, `AGENT_WORKSPACE`, `AGENT_SKILLS_DIR`,
`AGENT_SKILL_LIBRARY_DIRS`, `AGENT_INSTRUCTION_FILES`, `AGENT_FORCE_SKILL`, and
`AGENT_RETRIEVAL_TARGET`. The built-in local adapter cannot capture tool calls, token use, or
cost, so disable gates that require unavailable evidence or use a custom adapter.

### Custom command

Use `type: custom` when a command can emit normalized trace JSON. The built-in
`json_stdout` parser accepts one JSON object containing `final_output` plus optional turn, tool,
cost, token, skill-event, and case-failure fields:

```yaml
schema_version: 1
runner:
  type: custom
  command: my-agent-adapter
  model: custom-model
  env: {}
  timeout_seconds: 600
  max_tokens: 200000
  parser:
    name: json_stdout
    options: {}
```

Custom commands receive the same standard input, workspace, and environment contract as local
commands. An unknown parser or malformed output is a runner error, never a case failure.

### Evidence-specific gates

Release checks are enabled by default. When a runner cannot produce meaningful evidence for a
specific metric, disable only that check in `evals/gates.yaml`; the gate report retains it as an
auditable `waived` check rather than silently dropping it:

```yaml
release_gates:
  utility:
    check_distractor_degradation: false
  invocation:
    check_positive_invocation_rate: false
    check_false_trigger_rate: false
  efficiency:
    check_cost_delta: false
    check_latency_delta: true
```

Keep checks enabled when the runner supplies real evidence. These switches describe a stable
runner capability; use expiring, owner-attributed waivers for temporary exceptions to an
available check.

## Security runs

Security probes have their own run lineage and do not inflate utility scores:

```console
uv run skill-eval security-probes ../my-skill
uv run skill-eval security-run ../my-skill --condition with_skill --trials 3
uv run skill-eval gate ../my-skill --run-id RUN_ID --security-run-id SECURITY_RUN_ID
```

Use `--probe PROBE-1,PROBE-2` to select probes. Critical boundary failures, forbidden tools,
secret exfiltration, and configured probe thresholds can block release.

## CI and exit codes

Run these checks before merging changes to an eval suite:

```console
uv run skill-eval validate ../my-skill
uv run skill-eval run ../my-skill --conditions no_skill,with_skill
uv run skill-eval grade ../my-skill --run-id RUN_ID
uv run skill-eval coverage ../my-skill --run-id RUN_ID
uv run skill-eval report ../my-skill --run-id RUN_ID
uv run skill-eval gate ../my-skill --run-id RUN_ID
```

The CLI uses stable process exit codes:

| Code | Meaning |
| ---: | --- |
| `0` | Pass |
| `1` | Eval or release-gate failure |
| `2` | Warning-only gate outcome |
| `3` | Invalid suite or configuration |
| `4` | Runner or environment error |

Do not retry code `1` as an infrastructure failure. Code `4` means the evidence is incomplete
because execution did not finish reliably. Gate waivers must name an owner and reason, have a
bounded validity window, and match the intended run or status. Expired, revoked, mismatched,
or missing waivers do not suppress failures and remain visible in JSON, HTML, and JUnit output.

This repository's [CI workflow](.github/workflows/ci.yml) runs linting, formatting, strict type
checking, tests with coverage, packaging, and the full end-to-end path with network access
disabled. It publishes JUnit and report artifacts even when a gate fails.

## Offline behavior and optional dependencies

The following operations are deterministic and can run without network access once dependencies
and the selected local executable are installed: initialization, analysis, synthesis planning,
review and promotion, suite validation, local/custom execution, canonical grading, coverage,
reporting, gates, registry migrations, and portfolio summaries.

These operations need an external capability and fail explicitly when it is unavailable:

- Waza execution needs the pinned Waza binary; Copilot-backed execution needs its model access.
- Description optimization currently needs `runner.type: waza` and a model.
- A structured LLM rubric needs a configured provider or compatible Waza prompt grader.
- Waza installation, model downloads, and dependency installation are not offline operations.

There is no silent hosted fallback. A missing external runner produces exit code `4`; invalid or
unavailable optional configuration produces an actionable error.

## Command reference

Every command takes a required `SKILL` directory unless noted. Comma-separated selectors reject
blank values. Run `uv run skill-eval COMMAND --help` for generated usage text.

| Command | Purpose | Command-specific arguments and options |
| --- | --- | --- |
| `init` | Add an eval-suite scaffold without overwriting files. | No options. |
| `analyze` | Extract constraints, trigger prompts, and security boundaries. | `--regenerate` writes a content-addressed candidate set while preserving reviewed files. |
| `synthesize` | Create reviewable draft cases without changing `eval.yaml`. | Repeatable `--seed PATH`; repeatable `--waza-task PATH`. |
| `review` | Interactively accept, edit, reject, skip, inspect, or regenerate drafts. | Required `--reviewer TEXT`; optional `--case ID`. |
| `promote` | Add accepted drafts to the canonical suite. | Required `--by TEXT`; optional `--case ID`. |
| `optimize-description` | Compare descriptions on repeated held-out trigger trials. | `--trials N` (at least 2, default 3); `--model TEXT`; `--executor TEXT` (default `copilot-sdk`); repeatable `--candidate TEXT`. |
| `approve-description` | Approve one exact evaluated description. | Required `CANDIDATE_ID` argument and `--reviewer TEXT`. |
| `promote-description` | Write an approved description to `SKILL.md`. | Required `CANDIDATE_ID` argument and `--by TEXT`. |
| `security-probes` | Generate or validate the deterministic probe catalog. | No options. |
| `security-run` | Execute probes in isolated security lineage. | `--probe ID,ID`; `--conditions NAME,NAME` or `--condition NAME`; `--trials N` (default 1); `--runner TEXT`; `--model TEXT`; `--run-id TEXT`; the same condition-source options as `run`. |
| `validate` | Validate schemas, IDs, selectors, graders, fixtures, constraints, and gates. | No options. |
| `run` | Execute selected cases, conditions, and trials. | `--conditions NAME,NAME` or `--condition NAME`; `--case ID,ID`; `--tag TAG,TAG`; `--trials N`; `--runner TEXT`; `--model TEXT`; `--output-dir PATH`; `--previous-skill PATH`; `--stale-skill PATH`; repeatable `--distractor PATH`, `--malicious-distractor PATH`, and `--library-skill PATH`. |
| `grade` | Grade a persisted run. | Required `--run-id TEXT`; `--canonical`; `--evidence-dir PATH`. |
| `coverage` | Compute observed behavior-constraint coverage. | Required `--run-id TEXT`; `--grade-id TEXT`; `--evidence-dir PATH`. |
| `report` | Write versioned JSON, self-contained HTML, and JUnit. | Required `--run-id TEXT`; `--grade-id TEXT`; `--evidence-dir PATH`; `--previous-run-id TEXT`; `--feedback-file PATH`. |
| `gate` | Evaluate release thresholds and return a stable exit code. | `--run-id TEXT` or `--report PATH`; `--security-run-id TEXT`. |

Global options are `--version`, `--install-completion`, `--show-completion`, and `--help`.

## Project quality checks

```console
uv run ruff check .
uv run ruff format --check .
uv run mypy src tests
uv run pytest --cov=skill_eval --cov-report=term-missing
uv build
uvx --from mfcqi mfcqi-py analyze --skip-llm src
```

The [CSV Summary demo](examples/skills/csv-summary-skill) is the smallest complete example. The
automated end-to-end test exercises analysis, synthesis, human approval, description
optimization, a contract-faithful Waza run, paired lift, grading, coverage, reporting, gate
failure, bounded waivers, registry persistence, and portfolio classification.