# Pinned corpus for the Tier 1 retrieval-quality gate (scripts/eval_retrieval.py).
#
# Format: `owner/repo <40-char commit SHA> <entry count>`, one per line.
# Materialise it with
#   bash scripts/ensure_eval_corpus.sh
# which taps each repo, checks it out at the pinned commit, and verifies it
# scans into the recorded number of entries (scripts/eval_corpus.py). Re-measure
# the counts with `python3 scripts/eval_corpus.py --relock`; check the
# distribution without a network with `--audit`.
#
# --- corpus size: written from the rows below by eval_corpus.py ---
# total:   10,731 entries in 20 repos
# targets: 921 entries in the 6 repos above the scale divider
# largest: sickn33/antigravity-awesome-skills, 6,634 entries (61.8%)
# next:    affaan-m/ECC, 1,621 entries (15.1%)
# scores:  tests/eval/baseline.json, re-baselined with every move
# --- end corpus size ---
#
# The block above is the only place this header states the corpus's size.
# `--relock` and `--refresh` rewrite it with the rows, so it cannot disagree
# with them; a sentence typed here did, the first time the monthly refresh
# moved the pins. The sizes and twenty-repo scores quoted elsewhere (CLAUDE.md,
# the Makefile, docs/eval.html) are checked against these rows and
# baseline.json by tests/unit/test_corpus_prose.py.
#
# The golden set (golden.jsonl) grades real catalog items by NAME, so the gate
# needs the repos that actually contain those items tapped before it can score.
# The first six repos are the minimal set covering all 77 golden targets across
# the skill / rule / workflow kinds. The rest exist to make the corpus a
# realistic size, which is the whole point: when #372 grew the list from six
# repos to twenty, BM25 went from recall@10 1.000 / hit@1 0.791 to 0.863 /
# 0.473. A user with twenty taps gets the second number, so flooring against
# the first was measuring the corpus rather than the retrieval.
#
# WHY THE SHAs. This list pinned repo *names* only, and tapping shallow-clones
# whatever the default branch points at, so the corpus was reproducible only for
# as long as twenty third parties happened not to push. Nothing had drifted when
# this was written, but the size was worth stating, because it was not what
# anyone assumed: most of the gate's corpus is ONE third-party repository,
# sickn33/antigravity-awesome-skills (the block above has its share today).
#
# The margin that protects was thin. When #410 pinned the list, BM25 scored
# recall@10 0.863 here and CI floored it at 0.85 — a buffer of 1.15 queries out
# of 91: one golden query could drop out of the top 10 and the gate still pass,
# two could not. A single upstream push to the repo holding most of the corpus
# could therefore turn a required check red on a pull request that touched
# nothing to do with retrieval. Pinning removes that coupling, and makes every
# number in this file reproducible rather than "true on the day it was
# measured".
#
# WHY THE COUNTS. A SHA fixes the tree; it does not fix what this project makes
# of that tree, and it does nothing when the tree cannot be fetched at all. Both
# gaps end in the gate scoring a corpus its floors were never set on, and the
# direction of the error is the surprise. Measured over the 91-query required
# set, one repo removed and everything else identical:
#
#   at the #410 pins        entries  recall@10 / hit@1 / MRR / nDCG@10
#   all 20 repos             10,152  0.852 / 0.473 / 0.605 / 0.657
#   minus sickn33 (62%)       3,843  0.885 / 0.593 / 0.711 / 0.746
#   minus that and ECC        2,227  0.967 / 0.659 / 0.769 / 0.814
#   minus LessUp (targets)    9,682  0.676 / 0.374 / 0.483 / 0.523
#   floors                           0.780 / 0.400 / 0.520 / 0.580
#
# Losing a *scale* repo makes the gate EASIER: every metric rises and the check
# goes green having measured a third of the intended corpus. Losing a repo that
# holds golden targets fails all four floors, indistinguishably from "this pull
# request broke retrieval". So skipping whichever repos are missing today is
# unsafe in both directions — the counts are what make the first case detectable
# before anything is scored, and an unreachable repo now exits 75 (EX_TEMPFAIL)
# so a third party's outage does not read as a regression here.
#
# The same table is why the concentration is not fixed by trimming: dropping
# the big repo raises every number. Diluting it means adding breadth. The
# ceiling in eval_corpus.py (MAX_SHARE) is a ratchet against making it worse.
#
# NOTE: `boost tap --defaults` is NOT sufficient — it taps only skill repos and
# scores 0.000 on every rule and workflow query (overall recall ~0.33). That is
# why this explicit list exists.
#
# If you edit this list — including moving a pin — re-measure and re-baseline:
#   BOOST_HOME=/tmp/evalhome python3 scripts/eval_corpus.py --relock
#   BOOST_HOME=/tmp/evalhome python3 scripts/eval_retrieval.py --build --save-baseline
#
# Cost: tapping is ~1 s/repo (measured 6 in 6.3 s, 14 more in 20.4 s), so this
# adds roughly 20 s to a cold CI run and nothing to a warm local one (the
# sentinel in ensure_eval_corpus.sh skips re-tapping, and is keyed on this
# file's digest so editing it can never leave a stale corpus behind). Pinning
# adds at most one `git fetch --depth 1 <sha>` per repo, and none when the
# shallow clone already carries the pinned commit. CI caches the materialised
# corpus on the same digest, so a repo that disappears upstream stops being able
# to redden every open pull request at once.
anthropics/skills                       3b3fad96af16a10759d930941b4520ba0c40edae    20
0xfurai/claude-code-subagents           9236d10702cdbba37eaa34515f1e1dbff8452506   138
NeoLabHQ/context-engineering-kit        23e2428e809d77717f8acc9659c374a3a1fcb93e   268
Aaronontheweb/dotnet-cursor-rules       9be2d201482bdafca8c687359f0ef0a71895d7af    11
ChrisWiles/claude-code-showcase         a95518f0cb67e86230119da40429169bc4c35a6f    14
LessUp/awesome-cursorrules-zh           ce39351450c4720a11c7edeadaef26d60c5c02e7   470

# --- scale: not golden targets, present so the corpus is a realistic size ---
aliarghyani/vue-cursor-rules            92609798e192312ce16edb78aa5424a8caaa415b    14
affaan-m/ECC                            ca185ef5f7667078a1e70a763bd3a9c71c48acf0  1621
BagelHole/DevOps-Security-Agent-Skills  0365f57a079b1332f95cf26e31dd2d5332a8399f   163
composio-community/awesome-codex-skills 0930e1373789d2eda449039f7ac154b33031de89   880
first-fluke/oh-my-agent                 75b3fbce93258f37843d6db8a7e32ab52a6d9aa3   205
iktakahiro/python-fastapi-ddd-skill     aa498d4c81acae475e05528520009c275cbe554f     6
langchain-ai/skills-benchmarks          9195f8c296f7076152cf5c698d6abd4f84853eb9    21
minio/skills                            22961d30013f4aadc863c41d14b28e58cb64c063     4
OneWave-AI/claude-skills                82859c0ebaff803889be6ca2efa0834ba8787773   205
quarkusio/quarkusdev-skills             b7c025b996bb4b9c3054740b242d1f368ae0eadd     5
sickn33/antigravity-awesome-skills      782f68446a14547133838e57bb677ebbddfc0cec  6634
tuanhaviet22/magento-skills             32d21015bd8eb46b8351fc87a8988ddca44de0f7     5
anthropics/claude-agent-sdk-python      9597fc956a2a18ff8a6dc5675d9823b457e8264f     6
eclosion-labs/terraform-cursor-plugin   bcaad7033598c7bbb4efd73483b49962bd3398d9    41
