Metadata-Version: 2.4
Name: punchmark
Version: 0.1.0
Summary: Point it at the response archive a benchmark number was computed on and get a producer-identity verdict -- SAME-PRODUCER / SUBSTITUTED / UNDETERMINED at a declared false-alarm rate -- with a one-line certificate attachable to the published score.
Project-URL: Repository, https://github.com/KurathSec/Punchmark
Project-URL: Issues, https://github.com/KurathSec/Punchmark/issues
Project-URL: Changelog, https://github.com/KurathSec/Punchmark/blob/main/CHANGELOG.md
Author: Yuxiang Ji
License-Expression: MIT
License-File: LICENSE
Keywords: auditing,evaluation,fingerprinting,hosted-models,model-substitution,reproducibility
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.12
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9; extra == 'docs'
Requires-Dist: mkdocs>=1.6; extra == 'docs'
Description-Content-Type: text/markdown

# punchmark

**No practitioner can currently re-certify an already-published benchmark number backwards
from the responses it was computed on.** A hosted route name persists while whatever serves
it moves. Every runnable producer-identity tool needs fresh probes issued live to an
endpoint that has since changed. punchmark is the retrospective half: point it at the
response archive you already have and get a producer-identity verdict for the number you
already published.

```
pip install punchmark
punchmark fit    dev/*.jsonl.gz --candidates routeA,routeB --out model.pmk-model.json
punchmark score  archive.jsonl.gz --model model.pmk-model.json --far 0.01
punchmark certify --ruling pmk-r-...
```

Four outputs:

1. **A fitted whole-set producer identifier** with its printed
   miss-rate-versus-false-alarm curve over a named candidate set.
2. **A per-(route, window) ruling**: `SAME-PRODUCER`, `SUBSTITUTED` or `UNDETERMINED`,
   issued at an operating point you declared yourself (as opposed to one you inherited).
   `UNDETERMINED` is a verdict in its own right: it is what a passing statistic means
   when your archive could not have resolved the substitution you asked about.
3. **A one-line certificate** attachable to the published score:
   *"producer identity of route R over window W HOLDS at false-alarm rate 0.01 against
   candidate set C."*
4. **The minimum substituted fraction your k and item count could have resolved**, so a
   null result reads as a power limit and not as reassurance.

## What a verdict means, and what it does not

A ruling certifies the **route label as served** within a **closed candidate set**. It is
never a statement about model weights: a public route name does not denote a fixed
configuration, and punchmark does not repair that. A `SAME-PRODUCER` ruling means exactly
"a substitution of the declared fraction by any candidate in the set would have been
flagged with the calibrated power, and none was", and nothing more. See `docs/honesty.md`
for the full list of things a ruling does not show.

## How it works

- **Inputs**: gzipped-JSONL response archives, one row per item with k draws of response
  **text only**: no logprobs, no headers, no timing. The route name comes from the
  archive filename (fixed by you at collection time); the collection window comes from a
  sidecar you write. Neither is ever inferred from content.
- **Oracle**: by construction. Cross-route labels score identification. Same-route
  subsets inside one window calibrate the false-alarm rate. Seeded substitutions (spliced
  from another route's rows over the identical item set) measure power.
- **Discipline**: every threshold is an empirical null quantile, cross-fitted so
  calibration optimism cannot silently overshoot the declared false-alarm rate. Every
  committed artifact is byte-stable and content-addressed. Rulings are append-only: a
  ruling can be superseded but is never edited. The CI gate fails any calibration move
  that arrives without a declared version bump.

## Status

Pre-release (0.1.0 in preparation). The calibrated chargram detector, the reference
corpus manifest, the shipped default model and the held-out validation study
(`validation/angle_a/FINDING.md`) are all committed; the numbers in `docs/validation.md`
trace to committed derived artifacts. Nothing in this repository states or implies a
finding about the capabilities of any model.

## Data provenance

The reference calibration corpus is defined over the committed response archives of the
public [Spaghetti Architect](https://github.com/KurathSec/Spaghetti-Architect) benchmark,
pinned by commit and per-file hash in `calibration/spaghetti/MANIFEST.json`. No
completion text is re-published here; the command
`punchmark corpus rebuild --corpus calibration/spaghetti --source <checkout>`
verifies a local checkout byte-for-byte. Results in this repository are statements
about producer identity of those archives as re-analysed here. They are not claims
about, or corrections to, any capability table published from that benchmark
elsewhere.

## License

MIT.
