Metadata-Version: 2.4
Name: pyterrier-tar
Version: 0.2.1
Summary: Replay-only technology-assisted-review stopping extensions for PyTerrier
Author-email: Aaron Fletcher <afletcher53@gmail.com>
License-Expression: MPL-2.0
Project-URL: Upstream, https://github.com/terrier-org/pyterrier
Keywords: technology-assisted review,systematic review,stopping rules,high-recall retrieval,pyterrier
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE.txt
License-File: THIRD_PARTY_NOTICES.md
Requires-Dist: ir-measures>=0.4.1
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: pyterrier<2,>=1.1.2
Requires-Dist: scipy
Provides-Extra: ms2
Requires-Dist: pomdp-py==1.3.5.1; extra == "ms2"
Provides-Extra: ms2-notebook
Requires-Dist: pomdp-py==1.3.5.1; extra == "ms2-notebook"
Requires-Dist: matplotlib>=3.6; extra == "ms2-notebook"
Requires-Dist: jupyterlab<5,>=4; extra == "ms2-notebook"
Requires-Dist: nbclient<1,>=0.10; extra == "ms2-notebook"
Requires-Dist: nbformat<6,>=5; extra == "ms2-notebook"
Requires-Dist: ipykernel<8,>=6; extra == "ms2-notebook"
Provides-Extra: scifact
Requires-Dist: pomdp-py==1.3.5.1; extra == "scifact"
Provides-Extra: scifact-notebook
Requires-Dist: pomdp-py==1.3.5.1; extra == "scifact-notebook"
Requires-Dist: matplotlib>=3.6; extra == "scifact-notebook"
Requires-Dist: jupyterlab<5,>=4; extra == "scifact-notebook"
Requires-Dist: nbclient<1,>=0.10; extra == "scifact-notebook"
Requires-Dist: nbformat<6,>=5; extra == "scifact-notebook"
Requires-Dist: ipykernel<8,>=6; extra == "scifact-notebook"
Provides-Extra: plot
Requires-Dist: matplotlib>=3.6; extra == "plot"
Provides-Extra: grl
Requires-Dist: gymnasium>=0.29; extra == "grl"
Requires-Dist: scikit-learn; extra == "grl"
Requires-Dist: stable-baselines3>=2.3; extra == "grl"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Requires-Dist: matplotlib; extra == "test"
Requires-Dist: scikit-learn; extra == "test"
Provides-Extra: tutorial
Requires-Dist: matplotlib; extra == "tutorial"
Requires-Dist: pyterrier[java]<2,>=1.1.2; extra == "tutorial"
Requires-Dist: scikit-learn; extra == "tutorial"
Dynamic: license-file

# pyterrier-tar

Replay-only technology-assisted-review (TAR) stopping extensions for [PyTerrier](https://github.com/terrier-org/pyterrier).

This is an independent package, not a PyTerrier fork.  It depends only on
documented public `pyterrier` APIs so that the dependency can be upgraded and
compatibility tested independently. It does not vendor PyTerrier code.

## Status

Alpha (0.2.1). Each stopping rule replays a recorded, labelled ranking and
reports where it would have stopped, why, and at what review cost. Rules whose
authors released code or results are checked against them, and the rest
against their published definitions; the Published Evidence section below
summarises every check and what it does not establish.

The package also registers the CLEF eHealth TAR collections, as published in a
public mirror, as `tar:clef2017`, `tar:clef2018`, and `tar:clef2019`. These
dataset adapters hash-check the mirror's archives and pinned released AutoTAR
rankings before using them for replay validation. For its CLEF 2017
development topics the mirror's abstract qrels differ from the official ones
(see `DATASETS.md`).

File names such as `BENCHMARKING.md` and paths under `examples/` refer to the
source distribution, which holds the full documents, the example scripts, and
the notebooks. The wheel that `pip install` uses holds only the package; the
Install section shows how to fetch the rest.

## Stopping rules

### Experimental MS² review-conclusion stopping

`examples/notebooks/demo_ms2_pomdp.ipynb` explores the original
MS² data and walks through reading linked abstracts, updating a three-way belief,
and stopping with h2r's native `pomdp_py.POUCT`. The complete PyTerrier pipeline
is `MS2BM25 >> MS2EvidenceStop`; install `.[ms2-notebook]` for the notebook or
`.[ms2]` for the solver. Data downloads are checksum-verified and cached locally.
The example uses real study text with explicitly identified teaching observations
and assumed probabilities. It is not a calibrated evidence-strength model.
See `MS2_EXPERIMENT.md` in the source distribution for assumptions, exact-planner checks,
the training-annotation interface and evaluation requirements.

### Experimental SciFact verdict stopping

The source distribution includes `SCIFACT_EXPERIMENT.md` and the
`examples/scifact_*.py` scripts for a separate claim-verdict stopping experiment.
The executable walkthrough is `examples/notebooks/demo_scifact_pomdp.ipynb`;
install the `scifact-notebook` extra for its plotting and Jupyter dependencies.
`pyterrier_tar.scifact` adapts the prepared SciFact toolkit to PyTerrier topics,
corpus records and complete BM25 rankings. `pyterrier_tar.decision` supplies a
paid-report replay and an exact stationary finite-depth reference planner;
`pyterrier_tar.scifact_pomdp` optionally integrates h2r's `pomdp_py` agents and
POUCT solver (install the `scifact` extra). A specialised exact planner is also
available through the library's planner interface.
This experimental API scores verdict error plus reading cost. It requires real
reader reports and grouped training/evaluation manifests; it does not interpret
unannotated abstracts as negative labels or issue recall certificates.

### Recall stopping rules

For resumable experiments, uncertainty and failure reports, ranking comparisons,
TARexp/CSV history import, and diagnostic plots, see the Benchmarking section
below.

Every rule is a PyTerrier transformer over a ranked, labelled trajectory
(`qid`, `rank`, `label`, in review order). `transform` returns the reviewed
prefix; `stop_report` gives the stop and its cost; `stop_trace` explains the
decision; `calculation_trace` shows every checkpoint the rule evaluated. A rule
sees labels only up to the checkpoint it is judging, which the test suite
enforces for every rule except two that read later labels by design: the
offline `Oracle`, and `SCAL`, which labels its whole sample before it chooses
a cutoff, as S-CAL does. Before a rule's first checkpoint, `calculation_trace`
reports the next checkpoint its configuration defines, found with every later
label read as non-relevant, which can lie beyond a short trajectory; `Kneedle`
reports none when no checkpoint fits.

A rule that aims at a recall target takes `target_recall` as its first
argument, with no default, and a rule that needs the collection size takes
`collection_size` next: `tar.IPHyperbolic(.8, 3000)`, `tar.CMHHeuristic(.8, 3000)`.
Control-set rules take their screened frame first: `tar.QBCB(control, .8)`.

Rules and metrics that need the collection size either require
`collection_size` or accept it optionally. When it is optional (`SAFE` with an explicit estimate,
`BetaBinomial`, `stopping_metrics`) and omitted, the trajectory is taken to be
the whole collection, so pass it whenever the ranking is truncated. Each topic
of a multi-topic replay has its own pool: pass a mapping from qid to size, such
as `tar.CMHHeuristic(.8, {"T1": 5000, "T2": 3200})`; one integer is
shared by every topic.
`CLEFTARDataset.complete_ranking` appends a topic's unranked documents. A
`qid` and a `docno` are matched across frames as text, so `1` and `'1'` are the
same query or document; a `docno` may appear only once per query.

### Choosing one

```python
tar.catalogue()                        # every rule: family, promise, what it needs, source
tar.catalogue(promise='certificate')   # only the rules that carry a guarantee
```

Start from what you can supply. With **labels alone**, you can run most
trajectory rules or the checkpoint rules that also take a collection size;
the TARexp variants additionally require recorded rounds. With
**calibrated probabilities** from your classifier, the estimation rules become
available. If you can **screen an independent random sample**, `QBCB`,
`TargetRecapture`, and its `TMAdapted` name give a guarantee rather than an
estimate, and they charge that sample as review cost.

### What a rule promises

`tar.catalogue()` separates heuristic rules, model-based estimators, cost-benefit
or learned policies, sampling certificates, and the offline Oracle. Estimators
and policies have **no general recall guarantee**. Their observed target
attainment and cost must be evaluated on suitable held-out data.

Certificate claims require their sampling and independence assumptions: the
control is a uniform random sample of the population whose recall is
certified, drawn independently of the ranking, and the ranking contains every
control document. A sample of the top of a truncated ranking cannot certify
recall over the rest. The probability is over control draws, conditional on
the number of positive controls, and not conditional on the rule firing.
`coverage_summary` measures **reviewed-prefix recall** over repeated independent
control draws within each fixed topic. It excludes controls outside the stopping
rank. Total screened recall and cost remain available separately. Full-review
fallbacks that reach the target are distinguished from firings.

`EVALUATION.md` explains the contracts, recall fields and limits.

### Trajectory rules — labels and recorded review rounds

Only `TARexpKnee` and `TARexpBudget` read recorded rounds. The other rules check
a fixed batch grid (`initial_documents + j x batch_size`) and ignore any `round`
column, so on an imported history their checkpoints need not be its round ends.

| Rule | Stops when | Source |
| --- | --- | --- |
| `Kneedle` | At each checkpoint from `min_documents` (1000) on, it locates the knee of the gain curve (rank 1 included), then compares the pre-knee to post-knee slope ratio against `156 - min(relevant at knee, 150)`. The default checkpoints are Yang, Lewis, and Frieder's fixed-batch mapping of the BMI schedule (sec. 3.5.1, Table 1): 1201, 1401, ..., 2601, 3001, ..., so a ranking of 1,200 or fewer documents is never evaluated. `batch_size=1` (`schedule='source'`) replays Sneyd and Stevenson's released checkpoints and, when none fires, keeps their last checkpoint with `fired=False`. `knee_distance='absolute'` (default) is Sneyd and Stevenson's `find_knee`; `'signed'` keeps only points above the chord, Satopää et al.'s sign convention without the rest of their Kneedle algorithm. | [Cormack and Grossman (2016a)](https://doi.org/10.1145/2911451.2911510); [Yang, Lewis, and Frieder (2021)](https://doi.org/10.1145/3469096.3469873) |
| `Budget` | At a fixed batch end, three quarters of the collection is reviewed, or `n >= 10N/r` with a knee slope ratio of at least 6. When the trajectory is the whole collection, its end closes a short final batch, where the 75% cap fires. `collection_size` is one integer for every topic or a qid mapping. | [Cormack and Grossman (2016a)](https://doi.org/10.1145/2911451.2911510); fixed-batch procedure of [Yang, Lewis, and Frieder (2021)](https://doi.org/10.1145/3469096.3469873), sec. 3.6 |
| `TARexpKnee`, `TARexpBudget` | Separate round-based variants matching TARexp's maximum slope ratio and released round arithmetic. Require consecutive recorded `round` values, starting with seed round 0; checkpoints are round endpoints. The maximum over round splits fires once the mean positives per counted round reach the threshold, whatever the curve's shape. Parity holds for 0/1 labels. `TARexpBudget` also requires `collection_size`, TARexp's `n_docs` including any control documents. | [TARexp source](https://github.com/eugene-yang/tarexp/tree/d23724a73175cfacbaa95dbaf160a34bc17a1ac0) |
| `Rule2399` | At a fixed batch end, reviewed documents reach `2399 + 1.2 x relevant found`; a ranking shorter than 2,399 documents never fires. | [Cormack and Grossman (2016b)](https://doi.org/10.1145/2983323.2983776); fixed-batch procedure of [Yang, Lewis, and Frieder (2021)](https://doi.org/10.1145/3469096.3469873), sec. 5.2 |
| `ReviewHalf` | The first batch end at or beyond `floor(N/2)` of the known collection; the stop can overshoot by up to `batch_size - 1`, so with the defaults a pool of 4 to 200 documents is reviewed in full. `collection_size` is one integer for every topic or a qid mapping. | [Wallace et al. (2010)](https://doi.org/10.1186/1471-2105-11-55); [TARexp source](https://github.com/eugene-yang/tarexp/tree/d23724a73175cfacbaa95dbaf160a34bc17a1ac0) |
| `FixedRound` | `max_round` review batches after the seed block are done: the stop is `initial_documents + max_round x batch_size`, as TARexp's `FixedRoundStoppingRule` (its `max_round_exec` stops one round later). | [Yang, Lewis, and Frieder (2021)](https://doi.org/10.1145/3469096.3469873), sec. 3.1; [TARexp source](https://github.com/eugene-yang/tarexp/tree/d23724a73175cfacbaa95dbaf160a34bc17a1ac0) |
| `BatchPrecision` | `patience` consecutive batches after the seed have precision at or below the cutoff, each checked when it ends. The default cutoff 5/200 is TARexp's; the paper's BatchPos-20-1 and -4 are `BatchPrecision(.1, 1)` and `(.1, 4)`. TARexp's own rule checks each batch one round later and counts the seed round. | [Yang, Lewis, and Frieder (2021)](https://doi.org/10.1145/3469096.3469873), sec. 5.3 |
| `ConsecutiveIrrelevant` | A run of `count` (default 100) non-relevant documents completes, or of `fraction` of `collection_size` rounded up (the "K% consecutive irrelevant" rule ASReview users apply; Oliveira et al. evaluate K = 7%). The rounding is this package's choice: Campos et al.'s (2024) code floors, and Oliveira et al.'s definition was not available. `initial_documents` delays the first check, but its records count in the run. | common heuristic; [Oliveira et al. (2026)](https://doi.org/10.1016/j.ijmedinf.2026.106516) |
| `SAFE` | Every supplied key paper is marked relevant, at least twice the estimated total relevant documents and `min_fraction` of the collection are reviewed, and the last `consecutive` documents are non-relevant. | [Boetje and van de Schoot (2024)](https://doi.org/10.1186/s13643-024-02502-7) |
| `Oracle` | The prefix first reaches the target recall, using complete labels: the least review depth for the target on this ranking. It reads labels beyond its stop, so it is never deployable. On a truncated history pass `relevant_total`, such as the qrels count. A topic with no relevant documents stops at 0 and counts as fired. | [Stevenson and Bin-Hezam (2023), sec. 5.1.2](https://doi.org/10.1145/3631990) |

SAFE accepts sample counts as
`tar.SAFE(initial_relevant=1, initial_reviewed=10, collection_size=1000)`,
or an explicit collection estimate as
`tar.SAFE(estimated_relevant=100, collection_size=1000)`. Both give a minimum
Phase-2 depth of 200. A bare `SAFE(1)` is rejected because the sample size is
unknown. Depth and the consecutive window count Phase 2 only, and Phase-1
cost is excluded; this is an explicit convention, not a full-procedure parity
claim. See `GAPS.md`.

### Checkpoint rules — a statistical test at fixed intervals

| Rule | Stops when | Source |
| --- | --- | --- |
| `CMHHeuristic` | The BUSCAR urn p-value (the smallest over the reviewed list's suffixes) is below `alpha` at a batch checkpoint. Each checkpoint is an unadjusted test at `alpha`, so over repeated checkpoints the chance of stopping below target exceeds `alpha` (8.6% at `alpha = 0.05` under random review order with the defaults), and the minimum over suffixes is not exactly calibrated even at one look. It assumes the ranker's picks are no less likely to be relevant than random ones. `bias` other than 1 is buscarpy's Wallenius variant; no source says how to choose it. | [Callaghan and Müller-Hansen (2020)](https://doi.org/10.1186/s13643-020-01521-4); `bias`: [buscarpy](https://github.com/mcallaghan/buscarpy) |
| `AnytimeCMH` | The same p-value below `alpha / (k(k+1))` at the k-th checkpoint of the fixed grid; over K checkpoints the levels sum to `alpha K/(K+1)`. By the union bound the chance of stopping below target is at most `alpha` only if that minimum-over-urns p-value is itself calibrated at every checkpoint; calibrated single-urn p-values do not ensure this (an exact counterexample gives 0.063 at `alpha = 0.05`), and nothing ensures it for a real ranking, so it has no recall guarantee. It never stops earlier than `CMHHeuristic`. | original to this package (A. Fletcher) |
| `PoissonPoint` (IP-P) | A power-law rate fitted to window relevance, at one-based window starts, yields a Poisson quantile of the documents still unfound, and `found >= target x (found + quantile)`; also, whatever the target, when the latter half of the windows counts nothing relevant. The released windows leave each window's last document uncounted, and the released relevant-document gate is 0 from 35% of the ranking, so a ranking with nothing relevant early stops there. | [Stevenson and Bin-Hezam (2023)](https://doi.org/10.1145/3631990) |
| `IPHyperbolic` (IP-H) | As IP-P with a hyperbolic rate, fitted at zero-based window midpoints. `tail='released'` (default) reproduces the released code's expected-remaining formula, which understates it by `(1-b)^2` and so falls to about 0 as the bounded fit approaches `b = 1`, after which the rule fires whatever the target; `tail='model'` integrates the fitted rate. | [Stevenson and Bin-Hezam (2023)](https://doi.org/10.1145/3631990) |
| `IPExponential` (IP-E) | As IP-P with Sneyd and Stevenson's exponential rate `a exp(-kx)`, fitted at zero-based window midpoints. | [Stevenson and Bin-Hezam (2023)](https://doi.org/10.1145/3631990) |
| `IPAPPrior` (IP-A) | As IP-P with the AP-prior rate. `tail='released'` (default) reproduces the released code, whose rate is `aN/x` and whose tail integral is a constant, so the rule fires at the first fit below the error threshold whatever the target; `tail='model'` fits the paper's `a log(N/x)` rate and integrates it. | [Stevenson and Bin-Hezam (2023)](https://doi.org/10.1145/3631990) |

Every point-process rule takes `process='cox'` for the paper's Cox-process
variants (CX-P, CX-H, CX-E, CX-A): the fitted parameters are perturbed along
their standard errors, the predicted total is averaged over the normal density
as the released code does (including its integration over the central 98% of
the density and, for IP-H, its swapped standard errors for `b` and `c`), and
the rule fires when `target x total <= found`. This is the release's expected
per-node quantile, not the Poisson-mixture quantile of the paper's Eq. 7. The
release's own numbers show the Poisson and Cox variants rarely differ.

Three readings depart from the released loop; none changes a stop on the
published CLEF 2017--19 inputs. A fitted or perturbed tail that is negative or
not finite gives no estimate, where the release reads it as nothing left to find
and stops (`invalid_tail='released'` replays the release; CX-H keeps the
released reading by default, because its published comparison depends on it,
so its Cox nodes can certify far below the target). A Cox checkpoint whose
parameter errors are not finite never certifies. A checkpoint at the whole
collection is a full review, not a firing. With `n_windows` other than 10 the
empty-tail test reads the latter half of the windows (the release reads
`y[5:]`), and checkpoints below 10 or `n_windows` documents are skipped.

### Control-set rules — an independently screened sample

| Rule | Stops when | Source |
| --- | --- | --- |
| `QBCB` | The reviewed prefix holds `J` of the `r` positive controls at a checkpoint, where `J` is the smallest rank with `BinomCDF(J-1; r, target) >= confidence`; every positive control must be ranked. Accepts only `target > 0.5` and `confidence >= 0.5`, where the binomial bound has no known finite-population violation; use `TargetRecapture` for a lower target. | [Lewis, Yang, and Frieder (2021)](https://doi.org/10.1145/3459637.3482415) |
| `TargetRecapture` | All `k = ceil(-ln(1-confidence)/(1-target))` sampled targets have been reviewed. | [Cormack and Grossman (2016a)](https://doi.org/10.1145/2911451.2911510) |
| `TMAdapted` | The same generalised target-size formula already implemented by `TargetRecapture`, exposed under the point-process paper's name. At 95% confidence, 80%, 90%, and 99% recall require 15, 30, and 300 positive targets. | [Stevenson and Bin-Hezam (2023), §5.1.1](https://arxiv.org/html/2311.08597v1) |
| `ControlSetRecall` | The share of positive controls inside the reviewed prefix reaches the target: the control-set recall point estimate of e-discovery practice, with no confidence statement. `QBCB` is the same evidence read as a binomial lower bound. Never fires without a positive control. | e-discovery practice; analysed by [Lewis, Yang, and Frieder (2021)](https://doi.org/10.1145/3459637.3482415) |
| `BaselineInclusionRate` | Relevant documents found, counting the pilot's, reach `ceil(target x N x pilot positives / pilot size)`, the total extrapolated from a random pilot's inclusion rate. A point estimate. It never fires when the pilot finds nothing, where the sources stop at once. | [Shemilt et al. (2014), §3.2](https://doi.org/10.1002/jrsm.1093); the countdown rule of [Lewis, Yang, and Frieder (2021), §6.2](https://doi.org/10.1145/3459637.3482415) |
| `sample_control` | Helper: draws an unlabelled worklist to screen independently. Draw it from the whole population whose recall is certified, normally the complete ranking of the collection; `collection_size=` checks each query's frame size. Keep its `sample_order` column, which `TargetRecapture` uses to choose its targets. Control screening is charged in `stop_report` and `stopping_metrics`. | — |

### Estimation rules — scores, probabilities, or sampling metadata

| Rule | Stops when | Source |
| --- | --- | --- |
| `Quant`, `QuantCI` | Estimated recall from calibrated probabilities reaches the target; `QuantCI` first subtracts `nstd` standard deviations. Pass `score_snapshots` to replay per-round model scores instead of one fixed column; the rule is then checked at the snapshot indices unless `review_checkpoints` are given. | [Yang, Lewis, and Frieder (2021)](https://doi.org/10.1145/3469096.3469873) |
| `SDTraining`, `SDSampling` | A Gaussian quantile sets a cutoff on a complete descending score ranking. `SDTraining.fit()` pools normalised relevant scores from separate training topics; `SDSampling` fits an independently screened uniform control sample and charges all sampling effort. These are model estimates, without a recall certificate. | [Li and Kanoulas (2020)](https://doi.org/10.1145/3411755), Algorithms 6 and 7, after [Hollmann and Eickhoff (2017)](https://ceur-ws.org/Vol-1866/paper_102.pdf) |
| `AutoStop` | Documents found reach the target share of a Horvitz-Thompson estimate of the relevant total over a fixed with-replacement sample (`loose`), or of that estimate plus one standard deviation (`strict_v1`, `strict_v2`; `strict_v2` needs `collection_size`). It estimates the relevant total; there is no recall guarantee. Name a `draw_column` when documents were drawn more than once, and pass `review_checkpoints` to check at recorded batch ends. Never fires before a relevant document is found. Not the adaptive procedure. Pass `collection_size` to `stopping_metrics` too. | [Li and Kanoulas (2020)](https://doi.org/10.1145/3411755) |
| `SCAL` | The prefix's Horvitz-Thompson estimate reaches `target x 1.05` (S-CAL's prevalence factor; `prevalence_factor=1` for the plain share) of the estimate over the whole recorded sample, given in the order its strata were selected. Pass `stratum_column` to consider only stratum ends, as S-CAL does. The sample past the cutoff is charged as review cost; pass `control_documents()` and `collection_size` to `stopping_metrics`. Recall and cost are then those of the whole sample at any cutoff: the unlabelled documents S-CAL also retrieves above the cutoff are not in the input. | [Cormack and Grossman (2016b)](https://doi.org/10.1145/2983323.2983776) |
| `Chao` | Chao's (1987) estimate of unseen relevant documents satisfies `found >= target x (found + unseen)`, the paper's Optimistic criterion, from each relevant document's committee capture count at the checkpoint (a static `capture_count`, or time-indexed captures through `capture_history_column`). `estimate='upper'` is its Conservative criterion, replacing the total by the upper bound of Chao's log-normal confidence interval at `confidence`. The paper's Poisson-regression (Rivest) estimator is not implemented. | [Bron et al. (2025)](https://doi.org/10.1145/3724116) |
| `SWIFTRecall` | The span-based recall estimate of SWIFT-Active Screener reaches the target: the inclusion rate is `lookback / D`, where `D` is the span back to the `lookback`-th previous relevant document, and estimated recall is `TP / (TP + rate x unscreened)`. Labels only; the paper leaves the point estimate of the rate unstated, so the maximum-likelihood rate is used. | [Howard et al. (2020)](https://doi.org/10.1016/j.envint.2020.105623) |
| `BetaBinomial` | The beta-binomial posterior that few enough relevant documents remain is at least `confidence` (exact ties fire), once a relevant document is found and while documents remain unreviewed; a review that reaches the end of the collection is a full-review fallback. Assumes the unreviewed tail is exchangeable with the reviewed prefix, so it errs late on a priority ranking. | original to this package (A. Fletcher) |
| `EVPI`, `EVPIGreedy`, `EVPISmooth`, `EVPIBatch` | A myopic surrogate, the forecast relevance probability of the next document (or window) times the expected value of perfect information under the current belief, falls to the screening cost of the documents it covers. It is not the value of information about the next document, and the greedy policy is not optimal. Windows use the decision row's belief and the later rows' label-free `score` forecasts only. `EVPISmooth` reconsiders after every document; `EVPIBatch` commits to whole windows and decides at their boundaries. The document that triggers the stop is included in the reviewed prefix. | [Fletcher and Stevenson (2026)](https://doi.org/10.1145/3805712.3809548), replayed with the differences in the docstring |

`AnytimeCMH` and `BetaBinomial` are original to this package rather than
replays of published methods. The `EVPI` family replays the Greedy, Smooth (K)
and Batch (K) policies of Fletcher and Stevenson (2026), whose rankings,
calibrators and results are unreleased. None carries a parity claim: they are
covered by formula tests and the shared leakage and trace checks only.

Score-distribution rules require a complete, static ranking with comparable
scores from the same scoring procedure; `complete_ranking()`'s placeholder tail
scores (marked `score_imputed`) are refused. They standardise scores within each
topic, which assumes comparable prevalence and pool composition across topics,
retain ties at the quantile, and keep the top document when the quantile is above
every score (Li and Kanoulas keep none). A cutoff keeping the entire ranking, as
at target 1, is a full review rather than a fired stop. `SDTraining` rejects a
replay topic whose qid was a training topic; documents shared between topics are
not detected. `SDSampling` requires every sampled document in the ranking and
retains full review without firing if the sample has fewer than two relevant
documents or relevant scores tied up to rounding; pass its `stop_report` and
sample to `stopping_metrics` (`report=`, `control=`), since `review_fraction` in
`pt.Experiment` sees only the prefix. Sample labels are fitting data; ranking
labels are used for evaluation.
The non-uniform sampling extension remains unimplemented.

### Learned rules

`RLStop` trains a separate PPO policy for each recall target. Its state contains
reviewed-window relevance rates and -1 for unseen windows, with no classifier
or recall-target input. It uses the shared `_iter_checkpoints` replay and trace
interface. `n_windows=100` is the published setting; each topic needs at least
that many records. Source: [Bin-Hezam and Stevenson (2024)](https://doi.org/10.1145/3626772.3657911).

The paper and released code differ in the positive reward: `reward='paper'`
(the default) uses `1-i/T`, while `reward='released'` uses `1-(i-1)/T`.
Both use `-(i-T)/(B-T)` after the target window. Windows here cover every
record exactly once, use their actual sizes, and advance once per continue
action. These choices avoid released window/reset bookkeeping defects; they
do not reproduce published numerical results. The state holds window
proportions (the arXiv version), not the published counts, and replay takes
the arg-max action, where the release samples actions and averages 10 runs. Saved policies include checked
configuration and file digests, and loading at another recall target is rejected.

```python
import pyterrier_tar as tar

# training and held_out are labelled ranked frames from separate topic sets.
rule = tar.RLStop(target_recall=.9)  # requires the grl extra
rule.fit(training)
rule.save("rlstop-90")
restored = tar.RLStop(target_recall=.9).load("rlstop-90")
reviewed = restored.transform(held_out)
report = restored.stop_report(held_out)
```


`GRLStop` trains a PPO policy over 100 review windows, where the state is the
relevance rate in each reviewed window, a logistic-regression prediction for
each unreviewed one, the current window, and the target recall. It needs the
`grl` extra. Source: [Bin-Hezam and Stevenson (2025)](https://doi.org/10.1145/3726302.3729879).
By default one policy is trained for the replay target only; the paper trains
one policy over several targets (the release uses eight, 0.7 to 1.0), which
`train_target_recalls` reproduces. `calculation_trace(trajectory, index)`
exposes the state and policy action at each window before the last, where the
untrained final decision is not taken; `stop_trace(trajectory)` explains the
final decision.

### Datasets and helpers

`CLEFTARDataset` backs `pt.get_dataset('tar:clef2017')` and its 2018/2019
siblings: hash-pinned public archives, one candidate pool per topic,
`complete_ranking()` to append unretrieved documents, and `label_ranking()` to
attach qrels only after a ranking exists. `get_qrels('content')` gives the
full-text (content-level) judgements, and `get_results()` the released
AutoTAR ranking or, for 2017 and 2018, any of 54 participant and baseline
runs (`result_variants()`). `SynergyDataset` backs `tar:synergy` (the 26
reviews of ASReview SYNERGY) and `tar:synergy-plus` (119 reviews): each review
is a topic with its own pool of OpenAlex records. Data is downloaded into
PyTerrier's cache, never this package.

```python
import pyterrier as pt

dataset = pt.get_dataset('tar:clef2017')

# Replay the pinned, released AutoTAR ranking of one topic.
ranked = dataset.get_results('autotar').query("qid == 'CD008081'")
trajectory = dataset.label_ranking(dataset.complete_ranking(ranked, 'CD008081'))
reviewed = pt.tar.Kneedle().transform(trajectory)
```

To rank a topic yourself, build an index for it, because each
systematic-review topic has its own candidate pool (this needs the `tutorial`
extra):

```python
from tempfile import TemporaryDirectory

import pandas as pd

topic = dataset.get_topics().query("qid == 'CD008081'")
docs = pd.DataFrame(dataset.get_corpus_iter('CD008081'))
with TemporaryDirectory() as index_dir:
    indexref = pt.index.IterDictIndexer(index_dir, meta={'docno': 64}).index(docs.to_dict('records'))
    bm25 = pt.terrier.Retriever(indexref, wmodel='BM25', num_results=len(docs))
    # The topic queries are Boolean PubMed searches; tokenise() reduces them to plain terms.
    ranked = (pt.rewrite.tokenise() >> bm25).transform(topic)
trajectory = dataset.label_ranking(dataset.complete_ranking(ranked, 'CD008081'))
```

Labels are joined only after a ranking has been created; they must never be
made available to a ranker or a live stopping decision.

`DATASETS.md` (in the source distribution) describes every dataset: sources, pools, label
levels, per-year statistics, and candidates for more stopping datasets.
`describe()` on a CLEF dataset, a `ReplayDataset` or a result gives each
topic's pool size, relevant documents and prevalence.

### Metrics

Stopping and ranking quality are measured together, since a rule's stop
depends on the ranking it reads.

`stopping_metrics` reports, per stop, recall, cost, review fraction,
reliability, oracle depth, and the thresholded CLEF TAR measures: the
`loss_e`/`loss_r`/`loss_er` components, `threshold` and `norm_threshold`, and
the 2017 (`total_cost_uniform`, `total_cost_weighted`) and 2018
(`total_cost_2018`) costs, charging any separately screened controls once.
`ranking_metrics` reports, per ranking, the full-ranking CLEF TAR measures:
`ap`, `last_rel`, `norm_last_rel`, `wss_95`, `wss_100`, `norm_area` and
`recall@k%`. Both match the official `tar_eval.py` and `tar_eval_2018.py` on
every topic of the pinned sample runs, whose documents are shown or unshown,
judged 0 or 1 and listed once; `tar_eval`'s handling of feedback rows, other
judgements and repeats is not implemented. `stopping_frontier` scores one
review against several targets, and `reliability` and `review_fraction` are
ir-measures metrics for `pt.Experiment`.

Decision metrics judge a stop by the decision it leaves, as in decision-
sufficiency and decision-theoretic stopping. Give a decision trajectory (the
estimate available after each number of reviewed documents, optionally with
confidence bounds and a threshold) and `decision_metrics` adds decision
agreement with the full-evidence decision, Type I/II errors, `should_stop`
(the full-evidence interval clears the threshold) and `stopped_appropriately`
(the rule fired exactly when it should). `decision_utility` adds the payoff of
each decision under a risk threshold `q*` or an explicit payoff matrix, and net
utility at each screening cost; `decision_summary` adds rates with Wilson
intervals and regret against the best rule; `icer` gives the extra documents
screened per point of agreement.

`reliability_summary` gives the share of topics reaching the target with a
Wilson interval, which is the check for an estimator. `coverage_summary` gives
the share of repeated control draws whose reviewed prefix reached the target,
whether or not the rule fired, which is the check for a certificate.

### Adding a rule

One file per approach, inside the folder for what the rule reads:
`rules/trajectory/` for labels only, `rules/checkpoint/` for a test at fixed
checkpoints, `rules/control/` for an independently screened sample, and
`rules/estimation/` for recorded probabilities or sampling metadata.

```python
# src/pyterrier_tar/rules/trajectory/my_rule.py
from .._base import _TrajectoryStoppingRule, _batch_positions


class MyRule(_TrajectoryStoppingRule):
    """Stops once a batch holds no relevant documents."""

    _equation = 'fire if the batch ending at n has no relevant document'

    def __init__(self, batch_size: int = 200, initial_documents: int = 1):
        if batch_size < 1 or initial_documents < 1:
            raise ValueError('batch sizes must be positive')
        self.batch_size = batch_size
        self.initial_documents = initial_documents

    def _iter_checkpoints(self, ranked, labels):
        for stop in _batch_positions(len(labels), self.batch_size, self.initial_documents):
            batch = labels[max(0, stop - self.batch_size):stop]
            yield {'index': stop, 'relevant_in_batch': int((batch > 0).sum()),
                   'fired': not (batch > 0).any()}
```

Every `_TrajectoryStoppingRule` subclass implements `_iter_checkpoints`, yielding
one dict per checkpoint with at least `index` and `fired`. The base class uses
these calculations for both the stopping decision and `calculation_trace`.
Rules needing other columns read them from `ranked`; `_stop_position(labels)`
remains an adapter for label-only callers. Add `_trace_details` for the columns
`stop_trace` should carry, and `_unreached_details` for metadata before the first
checkpoint. A source procedure that returns a cutoff without firing can override
`_fallback_stop`, with `_fired` distinguishing that cutoff from a fired checkpoint.
Override `_prepare_input` and delegate to its base implementation when a rule
needs input normalization or fitted-policy checks shared by all replay methods.

The row for checkpoint `k` must use only `labels[:k]`. Export the class from
the group's `__init__.py`, from `rules/__init__.py`, and from
`pyterrier_tar/__init__.py`; the shared tests then pick it up, including the
leakage check and the report/trace consistency check. New rules are expected to
come with a test that re-derives the stop from the rule's own definition, as
`tests/test_rule_validity.py` does for the others.

## Scope and safety boundary

`pyterrier-tar` replays recorded, labelled trajectories.  It is not a live active-learning or screening controller, and it does not certify target recall in deployment.  Labels beyond the reviewed prefix must not be exposed to a stopping rule.  Separately sampled controls must be independently screened and accounted for in review cost.

Replay is not neutral for rules built to run inside a review. `examples/live_vs_replay.py`
runs a live continuous-active-learning review on each CLEF topic and compares Quant,
AutoStop, S-CAL and GRLStop run live with the same rule replayed here on that run's
ranking (`LIVE_VS_REPLAY.md`). At target 0.9, replay overstated AutoStop's reliability
(`strict_v1`: 85.7% of topics replayed, 51.6% live), because it samples from a ranking
built from every label; Quant replayed from the final model's scores stopped earlier
(79.0% of the pool reviewed against 82.0%, and never later); S-CAL (91.2% reliable
replayed, 90.1% live) and GRLStop (51.7% and 53.3%) changed little.

## Install

```bash
python -m pip install pyterrier-tar
python -m pip install 'pyterrier-tar[grl]'
```

The `grl` extra installs `gymnasium`, `scikit-learn`, and `stable-baselines3`
for `RLStop` and `GRLStop`; `plot` adds matplotlib for the figures; `tutorial` adds
PyTerrier's Java support and matplotlib for the notebooks that build Terrier
indexes.

The documents this README names, the example scripts, the notebooks, and the
tests are in the source distribution, not the wheel:

```bash
python -m pip download pyterrier-tar --no-deps --no-binary :all:
tar -xzf pyterrier_tar-*.tar.gz
```

The unpacked directory holds `BENCHMARKING.md`, `EVALUATION.md`,
`PUBLISHED_EVIDENCE.md`, `GAPS.md`, `RESULTS.md`, `RESULTS_UNCERTAINTY.md`,
`CHANGELOG.md`, `examples/`, and `tests/`. The same archive is on the
[PyPI files page](https://pypi.org/project/pyterrier-tar/#files).

## Quick start

```python
import numpy as np
import pandas as pd
from pyterrier import tar

# A recorded review: ranked documents, with the label revealed as each is read.
# Relevance thins out down the ranking, as it does in a real screening run.
rng = np.random.default_rng(0)
size = 6000
prevalence = .7 * np.exp(-np.arange(size) / 300) + .002
trajectory = pd.DataFrame({
    'qid': 'CD008081',
    'docno': [f'd{position}' for position in range(size)],
    'rank': range(size),
    'label': (rng.random(size) < prevalence).astype(int),   # 1 for relevant
})

rule = tar.Kneedle()
reviewed = rule.transform(trajectory)          # the prefix the rule would have read
report = rule.stop_report(trajectory)          # stop, fired, review cost
print(rule.stop_trace(trajectory).iloc[0].reason)
print(rule.calculation_trace(trajectory, index=int(report.stop.iloc[0])).tail())

qrels = trajectory.loc[trajectory.label > 0, ['qid', 'docno', 'label']]
print(tar.stopping_metrics(trajectory, reviewed, qrels, target_recall=.95, report=report))
```

On this trajectory Kneedle stops after 2,001 of 6,000 documents, having found
94.7% of the relevant ones; `stop_trace` says the knee slope ratio reached its
dynamic threshold, and `calculation_trace` shows the ratio at every checkpoint
it tested.

## Benchmarking

### From a dataset and rankers

`evaluate` prepares rankings and runs stopping rules in one call. The first
version supports datasets with an explicit corpus for each topic, including
`tar:clef2017`, `tar:clef2018`, and `tar:clef2019`. Dataset IDs are resolved through
PyTerrier; the CLEF provider downloads its pinned data on first use.

```python
import pyterrier_tar as tar

result = tar.evaluate(
    dataset="tar:clef2017",
    ranker="bm25",
    rules=[tar.PoissonPoint, tar.ConsecutiveIrrelevant(50)],
    topics=["CD008081"],       # omit to evaluate every topic
    query_column="title",     # use the review title as a plain-text query
    targets=[.8, .9, .95],
)
print(result.summary)
print(result.per_topic[["qid", "rule", "target", "recall", "review_fraction"]])
```

`result.trajectory` contains the completed, labelled rankings, and
`result.report()` returns the Markdown experiment report. The shipped
`examples/evaluate_toy.py` runs the same API with small local data and no
downloads.

Compare several rankers in one call using a name-to-ranker mapping:

```python
result = tar.evaluate(
    dataset="tar:clef2017",
    rankers={"BM25": "bm25", "Random": "random"},
    rules=[tar.PoissonPoint, tar.ConsecutiveIrrelevant(50)],
    query_column="title",
    targets=[.8, .9, .95],
    output="artifacts/ranker-comparison",
)
print(result.summary)
print(result.per_topic[["ranker", "qid", "rule", "target", "recall", "review_fraction"]])
```

Each value can also be a PyTerrier transformer or a per-topic factory. All
rankers use the same topics, review pools, rules, targets and seeds, and each
topic's corpus is read once. The summary and report keep ranker names separate.
Use `result.datasets["BM25"]` for that ranker's prepared `ReplayDataset`, or
`result.trajectory` for all labelled rankings with a `ranker` column.
For a single ranker, `result.data` and `result.trajectory` retain their original
behaviour. Supply exactly one of `ranker` and `rankers`.

* **Ranking.** `"bm25"` uses this package's in-memory BM25 baseline, without
  Java or an index; it does not claim Terrier scoring parity. `"random"` uses
  `ranking_seed=0` by default. `"autotar"` replays the dataset's released
  ranking from `get_results("autotar")`, which the CLEF datasets provide, so
  `rankers={"AutoTAR": "autotar", "BM25": "bm25", "Random": "random"}` compares
  all three in one call. `"run:<name>"` replays `get_results("<name>")`, such
  as the CLEF participant run `"run:Sheffield/Test_Data_Sheffield-run-1"`, so
  one call compares stopping across many real rankings. You can also pass an
  existing PyTerrier transformer, or a per-topic factory as shown below.
* **Labels.** `qrels_variant="content"` scores CLEF stopping against
  content-level qrels and `"abstract"` SYNERGY+ against title/abstract
  inclusion; the default is each dataset's default qrels.
* **Rules.** Basic classes and configured instances are templates. Requested
  targets and each topic's actual pool size supply `target_recall` and
  `collection_size`; the other settings are preserved and the supplied
  instances are not changed. This shorthand covers `Kneedle`, `Budget`,
  `Rule2399`, `ReviewHalf`, `FixedRound`, `BatchPrecision`,
  `ConsecutiveIrrelevant`, `Oracle`, `CMHHeuristic`, `AnytimeCMH`, `PoissonPoint`,
  `IPHyperbolic`, `IPExponential`, `IPAPPrior`, `BetaBinomial`, and `SWIFTRecall`. `FixedRound` needs a configured instance,
  for example `FixedRound(3)`. Name variants with a mapping:
  `rules={"50 irrelevant": tar.ConsecutiveIrrelevant(50),
  "100 irrelevant": tar.ConsecutiveIrrelevant(100)}`. The `QBCB` class stands
  for `tar.sampled_qbcb()`: each task screens the topic's documents in a
  seeded random order, shared by compared rankers, until the fewest positives
  that certify the target at the given confidence (0.95 by default) have been
  found, charging every screened control. That minimum sample stops at its last
  positive control, where QBCB's bound holds at any target, so the sampler
  also accepts targets of 0.5 or below, which `QBCB` with a control frame of
  your own rejects. Lewis, Yang, and Frieder (2021, sections 6.3 and 8) report
  that the minimum sample usually increases total review cost against a
  larger one. Use `tar.sampled_qbcb(0.9)` for
  another confidence; a screened control frame of your own still needs an
  explicit `RuleSpec`.
* **Additional inputs.** For controls, calibrated probabilities, recorded
  rounds or sampling histories, fitted policies, and custom rules, pass an
  explicit `RuleSpec` as in the next section. Put external dataframes in
  `artifacts` and external parameters in the spec's `config`. The convenience
  layer does not train policies, infer sampling histories, or calibrate scores.
* **Pools and labels.** Each pool is the union of corpus and qrels document
  IDs for its topic. Rankers receive no evaluation labels; labels are attached
  afterwards, with unjudged documents treated as non-relevant. Missing
  documents are appended by document ID. Their scores/features remain missing,
  so rules requiring complete scores need a ranker that actually scores the
  full pool. Cross-pool hits and labels in ranker output are rejected.
* **Saving.** `output="artifacts/my-evaluation"` uses the existing resumable
  runner. One directory stores the whole comparison, including every ranker.
  Ranking is recomputed on each call; hashes must match to reuse
  completed stopping tasks. `ranker_config` records external ranking settings
  for provenance; with named rankers, use
  `ranker_configs={"My ranker": {"model": "version-1"}}`. Omitted names have
  empty configuration. These settings record provenance; they do not configure
  the supplied rankers. Changing rankers, names or settings requires a new
  output directory. `seeds` controls rule repetitions, separately from
  `ranking_seed`.
* **Parallelism.** `n_jobs=8` (or `-1` for every CPU) runs the stopping tasks
  in forked worker processes; ranking stays in the calling process. Tasks are
  independent and each draws from its own seeded generator, so the results
  and their row order are identical to a sequential run, and only the parent
  writes checkpoints. `run_experiment` takes the same option. Forking needs
  Linux or macOS; keep `n_jobs=1` when a rule or ranker holds a Java or GPU
  handle.

A dataset object can replace the dataset ID. It must provide `get_topics()`
with `qid` and the chosen query column, `get_qrels()` with `qid/docno/label`,
and `get_corpus_iter(qid=...)` yielding that topic's `docno/title/text` records.
Shared-corpus datasets need an adapter defining their review pools.

For a fresh Terrier index per topic, pass a factory. Its `RankerContext`
contains the label-free `topic`, `documents`, and pool IDs; `index_path` is a
temporary location kept alive until the returned pipeline finishes ranking.
This example requires the `tutorial` extra and a configured Java runtime:

```python
import pyterrier as pt

def bm25_for_topic(context):
    indexref = pt.index.IterDictIndexer(
        str(context.index_path), meta={"docno": 128}
    ).index(context.documents.to_dict("records"))
    return pt.rewrite.tokenise() >> pt.terrier.Retriever(
        indexref, wmodel="BM25", num_results=len(context.documents)
    )

# Use ranker=bm25_for_topic, ranker_name="Terrier BM25" in evaluate().
```

Factories can instead return a ranking DataFrame with `qid/docno/rank` and
optional score/features columns.

### Getting the results

`evaluate()` returns an `EvaluationResult`, the one place for every result of
an experiment, and `tar.load_results(directory)` rebuilds it from any
`output=` directory or saved result:

```python
result = tar.evaluate("tar:clef2017", rankers={"AutoTAR": "autotar", "BM25": "bm25"},
                      rules=[tar.Kneedle, tar.IPHyperbolic], output="artifacts/clef2017")
result.per_topic        # one row per ranker, rule, topic, target and seed: the stop and its metrics
result.rankings         # one row per ranker and topic: the quality of the ranking replayed
result.table            # both together, each stop beside the ranking it stopped
result.summary          # stopping results per rule, with intervals
result.ranking_summary  # mean ranking quality per ranker
result.describe()       # pool, relevant documents and prevalence per topic
result.report()         # Markdown: stopping, ranking quality, and decisions when given
result.save("artifacts/clef2017-report")
result = tar.load_results("artifacts/clef2017")
```

With decision trajectories (`evaluate(decisions=...)`, or
`ReplayDataset(decisions=...)`), `result.decision_metrics(threshold)` and
`result.decision_summary(threshold, costs=[...], q_star=...)` add the decision
measures. For `run_experiment`, wrap its output with
`tar.EvaluationResult.from_experiment(datasets, results)` or load its `output`
directory.

### From prepared rankings or recorded histories

`run_experiment` is the lower-level API for recorded histories, per-topic
results, uncertainty and failure reports. The input ranking is fixed before
labels are attached; this workflow does not train a ranker or select a stopping
rule.

```python
from pathlib import Path
import pyterrier_tar as tar

# trajectory has qid, docno, rank, label and covers the full collection.
data = tar.ReplayDataset("my-collection", "recorded-ranker", trajectory)
rules = [
    tar.RuleSpec("IP-P", lambda c: tar.PoissonPoint(c.target, c.collection_size)),
    tar.RuleSpec("50 irrelevant", lambda c: tar.ConsecutiveIrrelevant(50),
                 config={"count": 50}),
]
output = Path("artifacts/my-experiment")
results = tar.run_experiment([data], rules, targets=[.8, .9, .95],
                             seeds=[0], output=output)
summary = tar.experiment_summary(results)
summary.to_csv(output / "summary.csv", index=False)
(output / "REPORT.md").write_text(tar.experiment_report(results))
```

* **Rules.** A factory receives `RuleContext(trajectory, target, seed,
  collection_size)` and returns a fresh rule. Built-in rules are classified
  from the catalogue; a custom class needs `promise=` (`heuristic`,
  `estimator`, `policy`, `certificate`, or `offline bound`), which is metadata, not a
  verified guarantee. A factory that samples must draw from `context.rng()`,
  which is keyed by topic, target and seed; `default_rng(context.seed)` alone
  repeats one stream for every topic and target. The dataset name is not part
  of the key, so two datasets with a common qid share draws, and nothing checks
  that a factory uses it: global randomness gives draws that differ between
  runs and between sequential and parallel runs.
* **Repetition.** `repetitions="once"` uses the first seed and `"seeds"` uses
  every seed. Certificates default to `"seeds"` and other rules to `"once"`.
* **Resume and provenance.** Completed tasks are written atomically under
  `tasks/`, and repeating the call resumes them. `manifest.json` binds the
  parameters, input-frame hashes, package source hash and dependency versions;
  changing the manifest requires a new directory. Only one writer may use a
  directory.
* **Partial histories.** Omitting `qrels` declares the trajectory to be the
  entire collection, and the runner cannot infer unseen relevant documents.
  For a partial history, pass full-pool qrels including negatives.
* **Uncertainty and failures.** `experiment_summary` gives target attainment
  with Wilson 95% intervals, mean review fraction with bootstrap intervals,
  minimum recall, recall shortfalls, and counts of firings, full-review
  fallbacks, exhausted partial histories and zero-relevant topics.
  `worst_topics` keeps the topic, target and seed of individual failures. For a
  certificate, each row is one topic across independent control draws, and
  certification frequency is separate from reaching the target: reviewing
  everything can reach the target without producing a certificate. The
  intervals are descriptive and do not adjust for selecting among methods or
  targets.
* **Baseline rankings.** `bm25_ranking` and `random_ranking` are generated
  before qrels are joined, and `complete_rankings(pool, ranking)` appends
  unranked documents, so rankers are compared on the same pool. `bm25_ranking`
  is a specified in-memory baseline and claims no Terrier scoring parity.
* **Review histories.** `read_review_history("events.csv", scores="scores.csv")`
  reads portable CSV events (`qid,docno,rank,label,round`); `import_tarexp`
  and `read_tarexp` read TARexp ledgers. Native TARexp checkpoints are gzipped
  Python pickles and may execute code, so `read_tarexp` requires
  `trusted=True`: only load checkpoints you trust. TARexp records batch
  membership, not within-batch review order, so a document-level stop inside
  an imported batch is not a reconstructed historical decision.
* **Figures.** With the `plot` extra, `figure_recall_effort`,
  `figure_target_misses`, `figure_control_cost`, `figure_stopping_outcomes`
  and `figure_stopping_diagnostic` return figures 180 mm wide, and
  `save_figure(fig, "figures/name")` writes PDF, SVG and 600-dpi PNG.

`BENCHMARKING.md` has the rest: what the manifest records, ranking comparisons
on CLEF with `examples/benchmark_rankings.py`, the history formats, and what
each figure shows.

## Development

From the unpacked source distribution:

```bash
python -m pip install -e '.[test,grl,tutorial]'
pytest
```

`test` runs the suite, `grl` adds the RLStop and GRLStop tests, and `tutorial` the
notebooks. On a CUDA-free machine, `UV_TORCH_BACKEND=cpu` keeps `grl` from
pulling the GPU wheels.

## Results on CLEF

Every rule replays the released AutoTAR ranking of each topic in CLEF 2017,
2018, and 2019 (91 topics with a relevant document: 30 from 2017, 30 from 2018,
31 from 2019), completed with the pool's unranked documents. Review is the
share of each topic's pool screened, averaged over topics; for certificates it
includes the control sample.

**These numbers describe one ranker.** A rule that reads the ranking, which is
every rule here except the certificates, can do much worse when the ranking is
weaker. Rules are grouped by what they promise, so compare within a group, not
across groups.

Reliability is the observed share of topics reaching the target. An estimate
or nominal confidence parameter does not provide a general recall guarantee
across topics. `ControlSetRecall` reads the control samples drawn for `QBCB`
below, 20 per topic, as a point estimate; its review includes the sample, and
its recall credits positive controls beyond the stop (reviewed-prefix recall
alone reached the target in 70.7% / 79.2% / 88.8% of draws). The Oracle uses
complete labels to mark the least review depth for each target on this ranking.

| Rule | Reliability at 0.8 / 0.9 / 0.95 | Review at 0.8 / 0.9 / 0.95 |
| --- | --- | --- |
| CMHHeuristic | 100.0% / 100.0% / 100.0% | 47.6% / 60.8% / 72.1% |
| AnytimeCMH | 100.0% / 100.0% / 100.0% | 63.8% / 78.1% / 90.1% |
| PoissonPoint (IP-P) | 100.0% / 100.0% / 100.0% | 27.7% / 28.6% / 28.9% |
| IPHyperbolic (IP-H) | 93.4% / 90.1% / 89.0% | 16.1% / 16.8% / 18.1% |
| IPHyperbolic (IP-H), tail='model' | 100.0% / 97.8% / 97.8% | 17.6% / 18.5% / 20.0% |
| IPExponential (IP-E) | 98.9% / 95.6% / 94.5% | 17.1% / 18.0% / 19.4% |
| IPAPPrior (IP-A) | 100.0% / 97.8% / 94.5% | 22.5% / 22.5% / 22.5% |
| IPAPPrior (IP-A), tail='model' | 100.0% / 100.0% / 100.0% | 29.0% / 29.0% / 29.0% |
| PoissonPoint (CX-P), process='cox' | 100.0% / 100.0% / 100.0% | 27.7% / 28.4% / 28.9% |
| IPHyperbolic (CX-H), process='cox' | 93.4% / 91.2% / 87.9% | 17.7% / 18.9% / 19.9% |
| BetaBinomial | 100.0% / 100.0% / 100.0% | 90.3% / 96.5% / 99.0% |
| SWIFTRecall | 100.0% / 100.0% / 100.0% | 88.1% / 90.5% / 98.1% |
| SWIFTRecall(every) | 100.0% / 100.0% / 100.0% | 30.1% / 41.6% / 54.3% |
| ControlSetRecall | 79.8% / 87.1% / 93.6% | 46.1% / 63.7% / 81.9% |
| Oracle (offline bound) | 100.0% / 100.0% / 100.0% | 5.1% / 6.5% / 7.8% |

A heuristic takes no target and promises nothing about recall. It stops in the
same place whatever the target; the columns show how often that happened to
reach each one. A rule that never fires reviews the whole pool, which counts as
reaching every target; `RESULTS_UNCERTAINTY.md` gives the fired counts.
Kneedle's first checkpoint is 1,201 documents, so it never fires on the 24
pools of 1,200 or fewer; Rule2399 cannot fire on the 43 pools below 2,399
documents.

| Rule | Review | Fired | Review when fired | Reached 0.8 / 0.9 / 0.95 |
| --- | --- | --- | --- | --- |
| Kneedle | 54.7% | 67 of 91 | 38.5% | 100.0% / 100.0% / 100.0% |
| Budget | 43.8% | 91 of 91 | 43.8% | 100.0% / 100.0% / 100.0% |
| Rule2399 | 72.3% | 45 of 91 | 43.9% | 100.0% / 100.0% / 100.0% |
| ReviewHalf | 58.8% | 91 of 91 | 58.8% | 100.0% / 100.0% / 100.0% |
| BatchPrecision (cutoff 5/200, patience 1) | 31.5% | 79 of 91 | 21.1% | 100.0% / 98.9% / 96.7% |
| ConsecutiveIrrelevant (count 100) | 20.8% | 88 of 91 | 18.1% | 100.0% / 97.8% / 95.6% |

A certificate promises the target with 95% probability over the draw of an
independent control sample, under its sampling assumptions. Coverage measures
reviewed-prefix recall, excluding outside controls: the share of 20 simulated
control samples per topic that reached the target. A topic with too few
relevant documents cannot be certified, and the rule then reviews everything.

| Rule | Target | Coverage | Topics it could certify | Review (of which control) |
| --- | --- | --- | --- | --- |
| QBCB | 0.8 | 98.0% | 71 of 91 | 48.5% (42.3%) |
| QBCB | 0.9 | 98.2% | 60 of 91 | 65.6% (60.0%) |
| QBCB | 0.95 | 99.1% | 39 of 91 | 83.5% (79.0%) |
| TargetRecapture | 0.8 | 98.5% | 71 of 91 | 50.0% (43.8%) |
| TargetRecapture | 0.9 | 98.6% | 59 of 91 | 66.5% (60.9%) |
| TargetRecapture | 0.95 | 99.2% | 38 of 91 | 83.8% (79.4%) |

`Quant`, `QuantCI`, `AutoStop`, `SCAL`, `Chao`, the `EVPI` family, `GRLStop`,
`RLStop`, `SDTraining`, `SDSampling`, `TARexpKnee`, `TARexpBudget`, `FixedRound`,
`BaselineInclusionRate`, and `SAFE` are not in these tables: each needs an input
the CLEF release does not contain, a trained policy, or a parameter whose value
would be an arbitrary choice. `TMAdapted` shares the reported `TargetRecapture`
implementation.

These tables are copied from `RESULTS.md`, which also breaks them down by CLEF
year; `RESULTS_UNCERTAINTY.md` gives intervals and fired and fallback counts.
`examples/results_table.py` regenerates both.

## Published Evidence

A passing check does not turn a heuristic rule into a deployment recall
certificate. Each row says what was compared and how far the claim goes.

| Rule or method | Checked against | What the check establishes |
| --- | --- | --- |
| `stopping_metrics`, `ranking_metrics` | the official CLEF `tar_eval.py` and `tar_eval_2018.py` on the two pinned sample runs, all 20 topics | 14 `tar_eval.py` measures and 114 `tar_eval_2018.py` Task 2 measures (recall@1%..100% among them) match at the precision each script prints, on runs of shown and unshown documents judged 0 or 1; NCG and `tar_eval`'s other input conventions (feedback rows, judgements -1 and above 2, repeats) are not covered |
| `IPHyperbolic` (IP-H) | the authors' released rankings and results for CLEF 2017--19 | the topic means of recall, cost, reliability, `loss_er`, and relative error agree at 3 decimals in all nine settings; means do not pin each topic's stop |
| `PoissonPoint` (IP-P) | the same release, CLEF 2017 | the topic means of every IP-P and IP-H aggregate of the released `df_all_rates.csv` agree at 3 decimals (the paper prints the IP-P rows at target 0.9 only) |
| `IPExponential` (IP-E), `IPAPPrior` (IP-A), and `process='cox'` (CX-P, CX-E, CX-A) | the same release, CLEF 2017 | every released IP-E and IP-A aggregate at three targets, and the Table 1 CX-P, CX-E, and CX-A rows, agree as 3-decimal topic means; CX-H is not asserted: it differs from the printed row (cost 0.177 against 0.172) and, in an earlier audit, from the released rates in 8 of 9 CLEF settings, see the known gaps |
| `ControlSetRecall`, `Chao(estimate='upper')` | their published definitions | definition checks only: the control-set point estimate, and Chao's variance and log-normal interval against the worked examples of Bron et al. (2024, arXiv v1) and of the 2025 version of record |
| `Oracle` | the released point-process baseline table | all nine CLEF 2017--19 oracle rows (topic means) match at 3 decimals |
| `Kneedle` | Sneyd and Stevenson's source on the CLEF 2017 Waterloo run | the source's total effort (86,243) and 0.998333 mean of per-topic recalls rounded to two decimals (unrounded 0.998406), with `min_documents=1, batch_size=1`; the default schedule follows Yang, Lewis, and Frieder's Table 1 mapping and has no published run to compare with |
| `CMHHeuristic` | `buscarpy`, pinned | 216 p-values, 48 p-values of prefixes with no relevant document, and five retrospective stop positions (through `stop_report`) match; buscarpy also tests the final position, which the package's checkpoint grid does not; source-definition parity, not a paper reproduction |
| `QBCB` | Lewis, Yang, and Frieder's Table 1 | every published `r -> J` control rank; no numerical paper parity, because the paper's inputs were not released |
| `AutoStop` | the released AutoStop source's formulas, re-typed in the script, CLEF topic CD008081 | the `loose`, `strict_v1`, and `strict_v2` totals on 30 distinct draws with uniform probabilities; no released code is run, and this is not the adaptive procedure |
| `Quant`, `QuantCI`, `Rule2399`, `ReviewHalf`, `FixedRound` | TARexp's definitions | definitions follow TARexp and Quant's variance mirrors it, but TARexp's Quant does not stop before its round 2 and counts reviewed controls as known (see `QuantCI`); `Rule2399`, `ReviewHalf` and `FixedRound` match decisions recorded from TARexp's own classes, except that `Rule2399` and `FixedRound` report a stop at a short final round as unfired; no paper result is asserted |
| `BatchPrecision` | Yang, Lewis, and Frieder (2021), sec. 5.3 | follows the paper's rule; TARexp's `BatchPrecStoppingRule` counts the seed round and checks each batch one round later, so its stops differ; no paper result is asserted |
| `Budget`, `SCAL`, `Chao`, `TargetRecapture`, `BaselineInclusionRate` | their published definitions | definition checks only; `Chao` is the paper's Chao (1987) estimator on its printed equations, not the python-allib code behind the paper's tables, and not its Poisson-regression Chao (Rivest) estimator |
| `RLStop`, `SDTraining`, `SDSampling`, `TMAdapted`, `TARexpKnee`, `TARexpBudget` | pinned source and published definitions | reward/state and Gaussian-quantile tests; published target sizes; 1,310 endpoint decisions against TARexp source; a small CPU PPO save/load test. No paper-result reproduction. |
| `GRLStop` | the released GRLStop source | compared line by line; no trained policies or result tables were released, so there is nothing numerical to reproduce |
| `AnytimeCMH`, `BetaBinomial`, the `EVPI` family | nothing external | `AnytimeCMH` and `BetaBinomial` are original to this package; the `EVPI` family replays Fletcher and Stevenson (2026), whose data are unreleased: formula tests and the shared leakage and trace checks only |

Every rule that decides from a reviewed prefix is also checked for leakage:
changing the labels after its stop, or after a trace index, never changes it.
The check changes labels only: a fixed score or probability column, such as
Quant's, must come from a model fitted no later than the first checkpoint.

`PUBLISHED_EVIDENCE.md` is the full ledger, with the pinned revisions and
hashes behind each row. The examples under `examples/` are intentionally
separate from CI when they require network access, large public archives, or
locally held trajectories.

For real-data or external-source smoke checks:

```bash
python examples/published_cmh_buscar.py
python examples/published_clef_tar_eval_metrics.py
python examples/published_ip_h_clef.py
python examples/published_kneedle_clef2017.py
python examples/published_point_process_clef2017.py
python examples/published_autostop_clef2017.py
PYTHONPATH=examples python examples/published_baselines_clef.py
PYTHONPATH=examples python examples/published_clef_runs_oracle.py
```

Run them from a scratch directory: each caches its inputs in `data/` relative
to the working directory, about 1.3 GB in total.

`examples/notebooks/parity_published.ipynb` runs all of them and shows one
parity table. Every notebook is either a `demo_` (how the package behaves) or a
`parity_` (a published result reproduced):

| Notebook | Purpose |
| --- | --- |
| `demo_stopping_rules` | Every public stopping rule on a small synthetic trajectory: its decision, trace, and replay-only boundary. |
| `demo_trajectory_rules` | Why each trajectory-only rule fires, drawn: revealed labels, checkpoints, the trigger quantity, and the stop. |
| `demo_workflows` | Two worked workflows on real collections: a CLEF topic from index to stop, and control-set stopping on Vaswani with its screening cost. |
| `demo_fresh_rankers_stopping` | Build fresh BM25/TF-IDF rankings, run editable stopping methods in memory, and display publication figures. |
| `demo_multitopic_rankers_stopping` | The same across ten topics by default or a whole CLEF year, with all-topic outcomes and a paired diagnostic. |
| `demo_publication_figures` | Load and verify saved benchmark results, then display and export the publication figures. |
| `parity_published` | One table for every published claim: runs each networked parity script and asserts its result. |
| `parity_archives` | Four audits of other groups' released results: Repke et al. (2026), König et al. (2024), Bron et al. (2024), and the SYNERGY v2 Kneedle screen. Each needs archives you download, and skips when they are absent. |

`published_kneedle_clef2017.py` checks Kneedle's released 86,243 effort. The
same eight scripts run weekly in CI's `published-parity` job, with NumPy,
pandas and SciPy pinned. The notebooks that build Terrier indexes need the
`tutorial` extra, which installs PyTerrier's Java support.

Only the metrics asserted by each evidence check should be treated as validated.
For example, the CLEF metric checker asserts the 14 `tar_eval.py` and 114
`tar_eval_2018.py` measures it lists, and the IP-H CLEF checker the 3-decimal
topic means of recall, cost, reliability, loss and relative error in each of
the nine CLEF 2017--19 settings; agreement of means does not pin each topic's
stop.

## Known gaps

What this package does not implement or cannot reproduce, and why. It replays
recorded reviews, so nothing here establishes how a rule behaves when it
steers a live one.

These published results need data the authors did not release. Nothing in the
package should claim them.

| Result | Missing input |
| --- | --- |
| QBCB's RCV1 figures | the 20% subset, seeds, active-learning trajectories, and control-sample identities |
| Yang et al. (2021) heuristics on RCV1 | the saved review histories |
| Adaptive AutoStop | the per-draw sampling distributions, which a trajectory cannot reconstruct |
| Bron et al. (2024) Table 7 integer displays | the analysis and formatting code; raw aggregates do not round to the printed integers (57.566 prints as 57) |
| The `Knee` rows of the point-process baseline table | the 2018 qrels are rebuilt in that repository from a PID list it does not ship; `published_baselines_clef.py` reports the difference instead of asserting it |
| The CX-H rows of the point-process results | unresolved: `IPHyperbolic(process='cox')` gives cost 0.177 against the printed 0.172 in the CLEF 2017 Table 1 row, and an earlier audit found it differs from the released rates in 8 of the 9 CLEF 2017--19 settings. A verbatim port of the released Cox loop, not shipped here, agreed with it on every CLEF 2017 topic; that the SciPy release the loop ran with (whose bounded-fit covariance it perturbs along) explains the gap is untested. `published_point_process_clef2017.py` reports the 2017 row instead of asserting it |
| `TM` and `TM-adapted` rows of the same table | the paper (§5.1) attributes them to Li and Kanoulas's target-method implementation, whose per-topic target draws and runs were not released; the repository's own `run_target_method` seeds its draw with `random.seed(1)` and is never run in the released notebook |
| RLStop and GRLStop paper results | no trained policies or result tables were released |

These methods are named but not implemented.

| Name | State |
| --- | --- |
| Non-uniform score-distribution feedback sampling | `SDTraining` and uniform-sample `SDSampling` are implemented; the triangular, bias-corrected sampling extension is not. |
| SALτ, the SLD-adjusted quantification stopping rule of Molinari and Esuli (2024) | not implemented: its adjusted SLD step needs the previous round's classifier scores, the newly annotated batch, and a validation split, which a recorded ranking does not carry. `Quant` over recorded `score_snapshots` is the nearest replay. |
| The Poisson-regression (Rivest) Chao estimator of Bron et al. (2025) | not implemented; `Chao` offers their Chao (1987) estimator with both the Optimistic and Conservative criteria. |
| The `p̂` of Howard et al. (2020), SWIFT-Active Screener | the paper states `D ~ NegBin(lookback, p)` and `recall = TP / (TP + p̂ U)` but not how `p̂` is computed from `D`; `SWIFTRecall` uses the maximum-likelihood rate `lookback / D`, and SWIFT-Active Screener's own code is not released, so no parity claim is made. |
| The classifier-augmented counting process of Bin-Hezam and Stevenson (2023) | not implemented: it retrains a logistic-regression classifier on the judged prefix at every checkpoint and fits the rate function over its predictions for the unjudged remainder, which a recorded ranking with one score column cannot replay. `PoissonPoint` is its base rule. |
| Adaptive AutoStop, a live S-CAL controller | out of scope: this package replays recorded trajectories and does not select documents; `examples/live_vs_replay.py` runs both live outside the package, and `LIVE_VS_REPLAY.md` measures what replay changes |

And these are limits of the engineering.

| Gap | Notes |
| --- | --- |
| CLEF archives come from one mirror | `npai.science.uu.nl`, hash-pinned. The hashes protect the content, not its availability |
| Extracted CLEF archives are trusted | each archive's SHA-256 is checked before it is extracted, but the extracted files are not re-checked on later loads; delete the `clef2017` (or 2018, 2019) directory in the PyTerrier cache to extract and verify again |
| 9 of 119 SYNERGY v2 datasets will not compose from source | missing source files or server errors at the pinned commit; `tar:synergy-plus` serves the composed 3.0 tag instead |
| SYNERGY+ 3.0 text is mostly titles | OpenAlex no longer supplies most abstracts: 26% of its 494,338 records have one (97% in release 1.0); 65 of its 119 reviews record final inclusion only, so `get_qrels('abstract')` covers 54 |
| CLEF participant runs cover 2017 and 2018 only | the served runs are Bin-Hezam and Stevenson's curated set; CLEF-TAR holds more 2017–2019 run files, not yet audited or pinned |
| Parity runs weekly, archive audits by hand | the `published-parity` job downloads about 1.3 GB; the archive notebooks need artifacts you supply |

These tables are copied from `GAPS.md`. New findings about limits belong there
rather than in a commit message.

## Reporting a problem

There is no public issue tracker. Send bug reports and questions to
afletcher53@gmail.com.

## Compatibility policy

The supported PyTerrier range is declared in `pyproject.toml`.  New code must import only public `pyterrier` names; imports from private modules such as `pyterrier._ops` are prohibited.  Each supported PyTerrier release will receive a focused compatibility test before its range is widened.

## License and notices

This source is licensed under the
[Mozilla Public License 2.0](https://www.mozilla.org/MPL/2.0/). `LICENSE.txt`
and `THIRD_PARTY_NOTICES.md` are installed with the package, in the
`licenses` folder of its `dist-info` directory.

The package contains no third-party source code. It was first written in a
fork of PyTerrier and extracted into this package; it uses PyTerrier only
through PyTerrier's public API. The stopping rules implement published methods
from their papers. Where a rule reproduces a released configuration, such as
`IPHyperbolic`'s released tail formula, it is an implementation of the
published method, not a copy of the authors' source code.
