# Pinned corpus for the Tier 1 retrieval-quality gate (scripts/eval_retrieval.py).
#
# Format: `owner/repo <40-char commit SHA> <entry count> [distinct count]`,
# one per line. The fourth field is how many of those entries are distinct
# *content* — `catalog.distinct_content`, the digest over name + description +
# body that `catalog._content_digest` stamps at scan time. It is optional so a
# list written before the field existed still parses; `--relock` fills it in,
# and tests/unit/test_eval_corpus.py requires it on every row of THIS list.
#
# Materialise it with
#   bash scripts/ensure_eval_corpus.sh
# which taps each repo, checks it out at the pinned commit, and verifies it
# scans into the recorded number of entries (scripts/eval_corpus.py). Re-measure
# the counts with `python3 scripts/eval_corpus.py --relock`; check the
# distribution without a network with `--audit`.
#
# --- corpus size: written from the rows below by eval_corpus.py ---
# total:   10,731 entries in 20 repos
# sixrepo: 921 entries in the 6 repos above the scale
#          divider -- a historical baseline, not a subset that scores
# largest: sickn33/antigravity-awesome-skills, 6,634 entries (61.8%)
# next:    affaan-m/ECC, 1,621 entries (15.1%)
# content: 5,938 distinct items; sickn33/antigravity-awesome-skills holds 35.7%
#          of them, and the rows overstate the corpus 1.81x
# scores:  tests/eval/baseline.json, re-baselined with every move
# --- end corpus size ---
#
# Two of those lines measure two different things, and the gap between them is
# the point. `largest:` is a share of the ROWS, which is what BM25 scores: the
# index is built from catalog.all_entries() with no de-duplication, so a
# registry that vendors the same skill under sixty-eight names really does
# occupy sixty-eight of the documents a query is ranked against. `content:` is
# a share of the distinct MATERIAL, which is what "the corpus is mostly one
# publisher's house style" actually means. sickn33 is 61.8% of the rows and
# 35.7% of the content; a ceiling on rows alone reads its vendoring as
# editorial dominance, and a ceiling on content alone would miss it inflating
# the index. eval_corpus.py floors both (MAX_ROW_SHARE, MAX_CONTENT_SHARE).
#
# The block above is the only place this header states the corpus's size.
# `--relock` and `--refresh` rewrite it with the rows, so it cannot disagree
# with them; a sentence typed here did, the first time the monthly refresh
# moved the pins. The sizes and twenty-repo scores quoted elsewhere (CLAUDE.md,
# the Makefile, docs/eval.html) are checked against these rows and
# baseline.json by tests/unit/test_corpus_prose.py.
#
# The golden set (golden.jsonl) grades real catalog items by the BODY each row
# pins, so the gate needs the repos that actually contain those bodies tapped
# before it can score. The first six repos are the minimal set the name-graded
# floors were measured on, and that is now a historical fact rather than a
# trimming recipe: both shipped sets pin exemplars into repos BELOW the scale
# divider as well (golden.jsonl into two, golden-natural.jsonl into five), so
# no proper subset of this file can score either set. See "the divider is not
# a target list" below. The rest exist to make the corpus a realistic size,
# which is the whole point: when #372 grew the list from six
# repos to twenty, BM25 went from recall@10 1.000 / hit@1 0.791 to 0.863 /
# 0.473. A user with twenty taps gets the second number, so flooring against
# the first was measuring the corpus rather than the retrieval.
#
# WHY THE SHAs. This list pinned repo *names* only, and tapping shallow-clones
# whatever the default branch points at, so the corpus was reproducible only for
# as long as twenty third parties happened not to push. Nothing had drifted when
# this was written, but the size was worth stating, because it was not what
# anyone assumed: most of the gate's corpus is ONE third-party repository,
# sickn33/antigravity-awesome-skills (the block above has its share today).
#
# The margin that protects was thin. When #410 pinned the list, BM25 scored
# recall@10 0.863 here and CI floored it at 0.85 — a buffer of 1.15 queries out
# of 91: one golden query could drop out of the top 10 and the gate still pass,
# two could not. A single upstream push to the repo holding most of the corpus
# could therefore turn a required check red on a pull request that touched
# nothing to do with retrieval. Pinning removes that coupling, and makes every
# number in this file reproducible rather than "true on the day it was
# measured".
#
# WHY THE COUNTS. A SHA fixes the tree; it does not fix what this project makes
# of that tree, and it does nothing when the tree cannot be fetched at all. Both
# gaps end in the gate scoring a corpus its floors were never set on, and the
# direction of the error is the surprise. Measured over the 91-query required
# set, one repo removed and everything else identical:
#
#   at the #410 pins        entries  recall@10 / hit@1 / MRR / nDCG@10
#   all 20 repos             10,152  0.852 / 0.473 / 0.605 / 0.657
#   minus sickn33 (62%)       3,843  0.885 / 0.593 / 0.711 / 0.746
#   minus that and ECC        2,227  0.967 / 0.659 / 0.769 / 0.814
#   minus LessUp (targets)    9,682  0.676 / 0.374 / 0.483 / 0.523
#   floors                           0.780 / 0.400 / 0.520 / 0.580
#
# That table is HISTORICAL -- it was measured when every row was graded by name
# and any subset of this file could still be scored. Today a missing repo that
# holds a pin is a SystemExit out of prepare_row before a single query runs, so
# the "minus sickn33" and "minus that and ECC" rows are no longer reproducible
# with the shipped golden.jsonl. What the table still shows is the direction,
# and it is the reason the counts exist: losing a *scale* repo that holds no
# pin makes the gate EASIER -- every metric rises and the check goes green
# having measured a third of the intended corpus. Losing a repo that holds
# golden targets used to fail all four floors, indistinguishably from "this
# pull request broke retrieval"; it now fails loudly by name instead, which is
# the one improvement pinning bought here. So skipping whichever repos are
# missing today is unsafe in both directions — the counts are what make the
# first case detectable before anything is scored, and an unreachable repo
# now exits 75 (EX_TEMPFAIL) so a third party's outage does not read as a
# regression here.
#
# The same table is why the concentration is not fixed by trimming: dropping
# the big repo raises every number. Diluting it means adding breadth. The
# ceilings in eval_corpus.py (MAX_ROW_SHARE, MAX_CONTENT_SHARE) are
# ratchets against making it worse.
#
# NOTE: `boost tap --defaults` is NOT sufficient — it taps only skill repos and
# scores 0.000 on every rule and workflow query (overall recall ~0.33). That is
# why this explicit list exists.
#
# If you edit this list — including moving a pin — re-measure and re-baseline:
#   BOOST_HOME=/tmp/evalhome python3 scripts/eval_corpus.py --relock
#   BOOST_HOME=/tmp/evalhome python3 scripts/eval_retrieval.py --build --save-baseline
#
# Cost: tapping is ~1 s/repo (measured 6 in 6.3 s, 14 more in 20.4 s), so this
# adds roughly 20 s to a cold CI run and nothing to a warm local one (the
# sentinel in ensure_eval_corpus.sh skips re-tapping, and is keyed on this
# file's digest so editing it can never leave a stale corpus behind). Pinning
# adds at most one `git fetch --depth 1 <sha>` per repo, and none when the
# shallow clone already carries the pinned commit. CI caches the materialised
# corpus on the same digest, so a repo that disappears upstream stops being able
# to redden every open pull request at once.
anthropics/skills                       3b3fad96af16a10759d930941b4520ba0c40edae    20    20
0xfurai/claude-code-subagents           9236d10702cdbba37eaa34515f1e1dbff8452506   138   138
NeoLabHQ/context-engineering-kit        23e2428e809d77717f8acc9659c374a3a1fcb93e   268    90
Aaronontheweb/dotnet-cursor-rules       9be2d201482bdafca8c687359f0ef0a71895d7af    11    11
ChrisWiles/claude-code-showcase         a95518f0cb67e86230119da40429169bc4c35a6f    14    14
LessUp/awesome-cursorrules-zh           ce39351450c4720a11c7edeadaef26d60c5c02e7   470   470

# --- scale: outside the six the floors were first measured on. NOT a
#     "no golden targets" line -- golden.jsonl pins two repos below it and
#     golden-natural.jsonl pins five, so this file has no tail that can be dropped ---
aliarghyani/vue-cursor-rules            92609798e192312ce16edb78aa5424a8caaa415b    14    14
affaan-m/ECC                            ca185ef5f7667078a1e70a763bd3a9c71c48acf0  1621  1588
BagelHole/DevOps-Security-Agent-Skills  0365f57a079b1332f95cf26e31dd2d5332a8399f   163   163
composio-community/awesome-codex-skills 0930e1373789d2eda449039f7ac154b33031de89   880   880
first-fluke/oh-my-agent                 75b3fbce93258f37843d6db8a7e32ab52a6d9aa3   205   140
iktakahiro/python-fastapi-ddd-skill     aa498d4c81acae475e05528520009c275cbe554f     6     6
langchain-ai/skills-benchmarks          9195f8c296f7076152cf5c698d6abd4f84853eb9    21    21
minio/skills                            22961d30013f4aadc863c41d14b28e58cb64c063     4     4
OneWave-AI/claude-skills                82859c0ebaff803889be6ca2efa0834ba8787773   205   205
quarkusio/quarkusdev-skills             b7c025b996bb4b9c3054740b242d1f368ae0eadd     5     5
sickn33/antigravity-awesome-skills      782f68446a14547133838e57bb677ebbddfc0cec  6634  2117
tuanhaviet22/magento-skills             32d21015bd8eb46b8351fc87a8988ddca44de0f7     5     5
anthropics/claude-agent-sdk-python      9597fc956a2a18ff8a6dc5675d9823b457e8264f     6     6
eclosion-labs/terraform-cursor-plugin   bcaad7033598c7bbb4efd73483b49962bd3398d9    41    41
