Metadata-Version: 2.5
Name: skill-harness
Version: 0.2.3
Summary: I wanted to know whether a skill was actually any good. This runs the same task with the skill and without it, and reports what can honestly be said about the difference - often "not enough to call it". A missing figure is a typed refusal, never an invented score. First-class Claude Code support.
Project-URL: Homepage, https://github.com/MrBinnacle/skill-harness
Project-URL: Repository, https://github.com/MrBinnacle/skill-harness
Project-URL: Issues, https://github.com/MrBinnacle/skill-harness/issues
Project-URL: Changelog, https://github.com/MrBinnacle/skill-harness/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/MrBinnacle/skill-harness/tree/main/docs
Author: MrBinnacle
License: MIT
License-File: LICENSE
Keywords: ab-testing,agent-skills,ai-agents,claude-code,eval-harness,evals,llm-evaluation,prompt-evaluation,skill-evaluation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.12
Requires-Dist: anthropic>=0.87
Requires-Dist: click>=8.1
Requires-Dist: openai<3,>=2.41
Requires-Dist: pydantic>=2.6
Requires-Dist: rich>=13.7
Requires-Dist: scipy>=1.11
Requires-Dist: statsmodels>=0.14
Requires-Dist: tiktoken<1.0,>=0.7
Provides-Extra: dev
Requires-Dist: hypothesis>=6.100; extra == 'dev'
Requires-Dist: jsonschema>=4.26.0; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: numpy>=2.0; extra == 'dev'
Requires-Dist: pre-commit>=4; extra == 'dev'
Requires-Dist: pytest-cov>=4.1; extra == 'dev'
Requires-Dist: pytest-randomly>=4.1; extra == 'dev'
Requires-Dist: pytest-socket>=0.7; extra == 'dev'
Requires-Dist: pytest-xdist>=3.5; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: scipy>=1.11; extra == 'dev'
Requires-Dist: statsmodels>=0.14; extra == 'dev'
Provides-Extra: inspect
Requires-Dist: inspect-ai>=0.3.245; extra == 'inspect'
Requires-Dist: inspect-swe>=0.2.65; extra == 'inspect'
Description-Content-Type: text/markdown

<p>
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/MrBinnacle/skill-harness/main/assets/banner-dark.svg">
    <img alt="skill-harness — the skill eval that refuses to invent a score" src="https://raw.githubusercontent.com/MrBinnacle/skill-harness/main/assets/banner-light.svg" width="680">
  </picture>
</p>

# skill-harness

[![CI](https://github.com/MrBinnacle/skill-harness/actions/workflows/ci.yml/badge.svg)](https://github.com/MrBinnacle/skill-harness/actions/workflows/ci.yml)
[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/MrBinnacle/skill-harness/blob/main/LICENSE)
[![PyPI](https://img.shields.io/pypi/v/skill-harness.svg)](https://pypi.org/project/skill-harness/)

## Why this exists

I wanted to know if you could tell if a skill was any good.

Not "is this file well written" — whether it changes anything. Search for a skill that makes
AI writing sound less like AI writing and you get a dozen of them. Each one costs you context
in every conversation you have. So how do you tell one from another in a way that actually
means something? I assumed plenty of people had already asked that and that an answer was
sitting somewhere. I went looking for it, didn't find one I trusted, and ended up building
this instead.

"This" is two public repositories: [MrBinnacle/skills](https://github.com/MrBinnacle/skills)
holds the skills, and this repository is the instrument built to answer the question about them.
Measured on 2026-08-15, that stands at **71 commits of collection against 323 commits of machinery
built to find out whether the collection is worth anything**:

```bash
git clone https://github.com/MrBinnacle/skills.git        && git -C skills        rev-list --count HEAD
git clone https://github.com/MrBinnacle/skill-harness.git && git -C skill-harness rev-list --count HEAD
```

The basis is a fresh clone at `HEAD` — what a plain `git clone` gets you — so those two commands
are the whole claim, and you can land on the same figures yourself, give or take what has merged
since.

skill-harness runs the same task with a skill and without it, and reports what can honestly be
said about the difference. Often what can honestly be said is "not enough to call it." That
turned out to be the useful part, and it took me a while to accept it.

It has first-class support for Claude Code skills and is built to extend to other agent
ecosystems.

## What does this skill cost you, and which parts of it are worth that cost?

That is the ratified wording, and the phrasing is load-bearing. "Is this skill good" hides two
different questions inside one word: a skill has a **price** you pay in every conversation
whether or not it fires, and a **benefit** that may or may not show up when it does. The price
is arithmetic on text and can be reported for free. The benefit needs a paid comparison, and
most of the time the honest answer about it is that there isn't enough evidence to say.

Those two are measured differently, refused differently, and reported separately everywhere in
the output. Collapsing them into a single score is the thing this tool exists not to do.

## What it has found so far

Plainly, because this is the part a README usually hides:

**Zero production-skill KEEPs.** Not one. The full keep lane has fired end to end
exactly once, on 27 July 2026, and that run was a *declared synthetic positive control* — a
skill I built to carry an invented fact, so the effect was real by construction. It returned
KEEP at 8/8 with the skill against 0/8 without, posterior probability of a win 0.99. That
tells you the instrument fires when a real effect is there. It does not tell you a single
real skill is worth its slot, and I am not going to let it be read that way.

The most common honest result, by a wide margin, is that the model already does the task fine
without any skill at all. On two deliberately hardened tasks a frontier agent passed 14 out of
14 runs with no skill present — there was nothing left for a skill to improve, so there was
nothing to measure. That is a real finding about the task, not a failure of the tool, and it
is written up in full: [the double-ceiling case
study](https://github.com/MrBinnacle/skill-harness/blob/main/docs/case-studies/double-ceiling-structurally-unmeasured.md).

One paired run before that, in July 2026, cost about $6.17 and returned a pre-registered NO-GO
— an apparatus check, not a measurement of benefit. I published it as such.

None of that is a scheduling accident. A sized benefit run launches only on the first task
whose no-skill screen returns a pass rate below 1, and so far every production skill I have
screened ceilings at 1: the model passes every attempt without it. You cannot measure help
that isn't needed.

If you came here for a number that says your skill is good, this tool will disappoint you on
purpose.

## Why it refuses

Comparing a skill against nothing is noisier than it looks. In a 60-trial arc on agentic tasks
I measured run-to-run swings of ±17.6% with everything held constant. An effect smaller than
that is invisible at three runs a side, which is roughly what most published skill comparisons
do. Hand-picked tasks tilt the result before anything runs. Pass/fail test banks price what a
skill *costs* and quietly skip what it *does*. The write-up, with the evidence grade attached
to each finding, is [here](https://github.com/MrBinnacle/skill-harness/blob/main/docs/findings/why-naive-skill-benchmarks-mislead.md).

So the design rule is the one I'd want from anyone reporting a number to me: **a figure that
isn't there is stated as a typed refusal, never filled in.** There is no third option — no
placeholder zero, no free-typed excuse, no estimate standing in for a measurement.

Three things follow from that.

**One.** Every paid comparison has a control arm. With and without, never a score in a vacuum,
because a score in a vacuum cannot tell you the model didn't need your skill.

**Two.** When the evidence won't carry a call, the answer is `UNMEASURED` with a reason
attached from a fixed list of eight — `no_data`, `inadmissible`, `underpowered`,
`falsifying_case_missing`, `budget_exhausted`, `falsifying_case_stale`,
`fdr_correction_failed`, `mechanical_vacuous`. "I don't know" is information; which flavour of
not-knowing is more information still. Definitions:
[`docs/concepts/why-unmeasured.md`](https://github.com/MrBinnacle/skill-harness/blob/main/docs/concepts/why-unmeasured.md).

**Three.** Evidence is checked against a gate before it can enter an aggregate, and the check
is snapshotted at write time in an append-only store. Data that fails the
evidence-admissibility gate is kept — never deleted — and never counted. Judge-graded results
count only where that judge has been calibrated against that specific axis first — swapping
answer order to cancel position bias, controlling for length, defending against injection, and
measuring agreement with a human before any judged verdict is allowed to count.

The same rule points inward. Two of the instrument's own weak points are measured rather than
asserted, and both numbers stay on the front page whether they flatter it or not:

**Extraction repeat-variance:** MEASURED for one skill — three repeat extractions of the same
`SKILL.md` returned 29/33/34 clauses, so clause counts are **not stable** run to run and
nothing downstream is allowed to key on clause position.
[#152](https://github.com/MrBinnacle/skill-harness/issues/152).

**Vacuity-flag precision:** MEASURED at 0.972 by blind cross-family adjudication over 102
estimation rows — but that is **flag-level only**. When the adjudicators also had to agree on
*which kind* of vacuity, kind-precision 0.835: `not_a_directive` matched 77/77, while
`weak_directive` matched 4/20.
[#153](https://github.com/MrBinnacle/skill-harness/issues/153).

## Try the free offline skill audit

`skill audit` is fully offline. No API key, no database, no network.

```bash
pip install skill-harness
skill-harness skill audit path/to/your/SKILL.md
```

It reports three things: the **cost triple** (what the skill costs you standing, when it
fires, and in its side docs — plain arithmetic on text, not a claim about effect), a set of
**structural checks** against [Anthropic's authoring
spec](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices), and
an **evaluability preflight** telling you what a paid run could and could not measure about
this skill today.

```text
OFFLINE AUDIT — no API calls, no cost
  skill:  caveman
  body:   41 lines / 233 words

  PASS  name            name 'caveman' meets spec
  PASS  body-length     body 41 lines (budget 500)
  INFO  description-unparsed-block-scalar
        description uses a multi-line YAML block scalar, which this audit's
        minimal frontmatter parser cannot read — checks skipped
        (UNMEASURED, not passed)

  Standing cost (mechanical): raw … tokens · calibrated … tokens
  Fired cost (mechanical):    raw … tokens · calibrated … tokens
  Aux cost (mechanical):      raw … tokens · calibrated … tokens

Summary: 2 pass · 0 warn — UNMEASURED is a verdict, not a failure.
```

Note what it does in the middle there. It couldn't parse the description, so it says so and
skips the check rather than passing something it never read. That behaviour is the whole
design, repeated at every layer.

`--strict` exits 1 on warnings, for CI. On Windows terminals, set `PYTHONUTF8=1` first.

## Measuring for real

```bash
skill-harness skill init path/to/SKILL.md --execute   # extract testable claims
skill-harness run ablation <skill_id> --execute       # the with/without comparison
skill-harness run evaluate-skill <skill_id>           # aggregate to a verdict
```

Either `ANTHROPIC_API_KEY` or `OPENROUTER_API_KEY` works. Every command that can spend money is
dry-run by default; `--execute` is required to spend, and per-run and daily caps are enforced
on top of that. Reproduction scripts:
[`examples/`](https://github.com/MrBinnacle/skill-harness/tree/main/examples/).

## What it measures, and what it refuses to

The answer comes back as one of three verdicts: **KEEP**, **CUT**, or **CAN'T-TELL-YET** — and
which of those you are even eligible for depends on the skill's registered value class, not on
the numbers alone. A CUT says why: `subsumed` (the model was already doing it), `no_lift` (you
needed the help and the skill didn't deliver it), or `harmful`.

There's a guard on that, and it is the **value-class guard**. Some skills exist to stop one
specific wrong move, and a model that passes without the skill hasn't proved the skill is
useless — it has proved the trap didn't come up. So `subsumed` is a CUT only for skills
registered as `TRANSFORMATIVE_LIFT`, the class whose entire claim is lift above the bar.
Everything else reclassifies to CAN'T-TELL-YET, on the grounds that this is the wrong
instrument for that kind of skill rather than a verdict on it.

Two of my own skills moved that way when the guard landed — `append-only-evidence-design`
(calibration) and a hardened `git-pull-rebase-trap` (trap-discipline). Under the pre-guard rule
both returned **CUT (subsumed)**, each at a no-skill pass rate of 1.00; the value-class guard
reclassified both to CAN'T-TELL-YET. Those pre-guard CUTs are preserved as dated historical
output rather than quietly edited into agreement.

What has still never fired is the other half: a paired run measuring how much a skill actually
helps, once you know the model needs help. By design, a sized benefit run launches only when a
screen returns a sub-1 pass rate, and none has yet.

## The reporting vocabulary is a published standard

Everything above — verdicts, refusal reasons, the cost triple, the evidence-admissibility
statuses, and the model pin and prompt fingerprint that stamp *which generation* produced a
figure — is fixed by the **Skill Efficacy Reporting Standard (SERS)**, a JSON Schema plus a
prose companion: [`docs/sers/`](https://github.com/MrBinnacle/skill-harness/tree/main/docs/sers/).

It's separate from this tool's internals on purpose. If you build your own harness, you can
emit conforming reports without adopting anything of mine. CI checks that this repo's own
receipts validate against it, that the schema's enums match the code's, and that deliberately
poisoned receipts get rejected — a guard that can't fail isn't guarding anything.

Models change underneath all of this, which means every figure has a shelf life. That's why
instrument identity is a required field and not a nicety: two numbers from two generations are
visibly non-comparable rather than silently averaged.

## What this isn't

It is not the most featureful skill benchmarker available, and I'd rather say so than let you
find out. If you want the most *featureful* skill benchmarking today, [adewale's
skill-eval-harness](https://github.com/adewale/skill-eval-harness) is the closest neighbour
and is further along on several axes; some of its disciplines are on my adoption list, with
attribution. If you're comparing prompts and configurations rather than skills,
[promptfoo](https://github.com/promptfoo/promptfoo) is the mature choice. If you're evaluating
models and agents, [Inspect](https://github.com/UKGovernmentBEIS/inspect_ai) is the
institutional one.

Reach for this one when your question is whether the number deserves to exist at all.

I make no first-mover claims anywhere in this repo. I checked twelve of them against primary
sources before writing any positioning, and enough of them were wrong that I stopped making
them. The two claims carrying the most weight — the pre-spend eligibility gate and the rule
that thresholds are ratified from enumerated tables rather than authored by hand — are
labelled as scheduled for external review until that review has actually happened. They'll be
upgraded or downgraded by a dated amendment, never silently.

Citations belong in the methods paper I'm writing, not on a front page.

## The other half

Verdicts that nobody acts on aren't worth producing, so there's a second repo where they land:
[MrBinnacle/skills](https://github.com/MrBinnacle/skills), a small collection where each skill
carries its own dated evidence record and controlled results are read from that skill's record,
not a front-page roll-up. Skills are re-screened when a major model ships and publicly retired —
with the record intact — once the model no longer needs them or a platform change meets a
pre-registered trigger. Each retirement is made against its stated criterion.

The two repos run on one rule, pointed at two different things. This one won't state a number
the evidence doesn't support. That one won't keep a skill the evidence no longer supports.
Same refusal, different end of the pipe.

## Dig deeper

- [Measurement receipts index](https://github.com/MrBinnacle/skill-harness/blob/main/docs/receipts-index.md) — every case study, finding,
  observation, assurance report, ratification, SERS instance, and the
  `skill audit --extraction` join surface: what each claims and what each
  refuses to claim.
- [Why this exists](https://github.com/MrBinnacle/skill-harness/blob/main/docs/why-this-exists.md) — how a non-specialist ends up building a
  measurement instrument, and the loop that made it possible.
- [The double-ceiling case study](https://github.com/MrBinnacle/skill-harness/blob/main/docs/case-studies/double-ceiling-structurally-unmeasured.md)
  — the run where there was nothing left to measure. Ask any skill benchmark what its
  without-the-skill pass rate was before you believe the rest of it.
- [The ablation that caught its own author](https://github.com/MrBinnacle/skill-harness/blob/main/docs/case-studies/ai-slop-sentinel-under-ablation.md)
  — three times, before a contaminated result could ship. The chain of refusals is the
  deliverable.
- [When ablation measures the wrong layer](https://github.com/MrBinnacle/skill-harness/blob/main/docs/case-studies/displaced-enforcement-skill-ablation-blind-spot.md)
  — if a discipline really fires in a hook, ablating the skill text tells you nothing about
  the discipline.
- [`docs/PRD.md`](https://github.com/MrBinnacle/skill-harness/blob/main/docs/PRD.md) — the full specification: evidence model, oracle tiers, gate
  rules, CLI surface.
- [The observation ledger](https://github.com/MrBinnacle/skill-harness/blob/main/docs/observations/README.md) — per-record screen history, annotated
  rather than rewritten.

Status: v0.2.3 on PyPI. Some older screen records are not yet in the evidence store. The
observation ledger shows the evidence behind each record.

MIT licensed. Issues and PRs welcome —
[`CONTRIBUTING.md`](https://github.com/MrBinnacle/skill-harness/blob/main/CONTRIBUTING.md).
