Metadata-Version: 2.5
Name: gtrex
Version: 1.1.0
Summary: Grounded text extraction you can check: verifiable quotation grounding for LLM extraction pipelines, with declarative normalization ladders, repair-and-reject cascades, and calibrated acceptance thresholds.
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: attribution,conformal-prediction,grounded-extraction,grounding,hallucination,information-extraction,llm,provenance,reproducibility
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Description-Content-Type: text/markdown

# gtrex

*Grounded text extraction you can check.*

`gtrex` checks whether a quotation extracted by a language model appears in
its source text, under a verification policy that is explicit, versioned and
fingerprinted. It certifies literal quotation provenance. It does not certify
that an extracted fact is true.

```bash
pip install gtrex        # Python >= 3.10, no runtime dependencies
```

## Verify

```python
from gtrex import Policy, Verifier

verifier = Verifier(source_text, Policy())
cert = verifier.verify(model_quote)

cert.grounded   # whether the policy accepts the quotation
cert.tier       # how much normalization it took, e.g. 'typography'
cert.start      # character offsets into the original source text
cert.span_text  # the source's own characters
```

Verification always checks the whole quotation, so a verbatim opening with an
invented tail is not grounded. If your extraction stores a source location,
`verifier.verify_at(model_quote, start, end)` checks that location only: a
match elsewhere in the document cannot validate it. The certificate then
reports `scope="span"`, the `requested_span`, and, after a failed check,
whether a contiguous match exists elsewhere.

## Policies

A grounding rate depends on choices: which normalizations to apply, what to
do with an ellipsis, whether stitched fragments count, and which tier counts
as grounded. A `Policy` records all of them. `Policy.fingerprint` digests that
configuration together with a verifier revision, which changes whenever the
code changes a verdict. Report the fingerprint, the package version and the
Unicode database version with any grounding rate; the run manifest records
all three.

The default ladder has five cumulative tiers: `exact`, `whitespace`,
`typography`, `case` and `skeleton`. Each tier declares whether it is
*sound*, an assumption that it forgives presentation, not content.
Verification reports the tightest tier that contains the whole quotation, and
a match must cover whole source characters. `skeleton` is unsound, because
removing punctuation makes different statements collide:

```
"Operating loss was ($10.0 million)."  ->  operatinglosswas100million
"Operating loss was $100 million."     ->  operatinglosswas100million
```

Counting an unsound tier as grounded requires `allow_unsound=True`.

Three policies are built in:
- `STRICT` accepts only character-identical quotations.
- `DEFAULT` accepts the sound tiers. A quotation joined from separate
  fragments with an ellipsis is reported as stitched, not grounded.
- `PERMISSIVE` accepts the skeleton tier, ignores ellipses and admits
  stitched fragments. It exists to measure how much such a policy inflates a
  rate, not as a recommendation, so it only verifies (`repair_mode="off"`).

`gtrex explain --policy strict` prints a policy with every assumption and its
fingerprint. If an upstream parser inserted text the model never saw, such as
layout markers, genuine quotations fail; declare it with `marker_pattern`.

## Repair

A `Repairer` handles near-misses:

```
verify -> [fragment salvage, opt-in] -> locate-and-substitute -> reject
```

In the default `substitute` mode it locates the best-matching source span
and stores the source's own characters, keeping the model's wording and the
certificates before and after. `annotate` keeps the model's wording and
attaches the located span; `off` only verifies. The substitution threshold
defaults to 0.65 when neither the policy nor a calibration sets one. That
value was hand-picked for an earlier pipeline and never calibrated, so a
repair at the default warns, and every result records `threshold_source`.
Calibrate one or set one where the repair rate matters. A quotation grounded
at a sound tier is stored as the source's characters (`normalized`); one
grounded only at an unsound tier goes through the threshold and guards. A substitution may store a source span shorter than the claim; the
stored text is certified by its own `after` certificate.

Two guards refuse substitutions that no similarity threshold can catch,
because the edited text shares almost every token with the source:

```
claim: "Operating loss was $100 million for the year ended ..."
span:  "Operating loss was ($10.0 million) for the year ended ..."       -> numeric_guard

claim: "We believe that we will be able to compete effectively ..."
span:  "We believe that we will not be able to compete effectively ..."  -> polarity_guard
```

The numeric guard compares figures, their order and adjacent signs. The
polarity guard compares negation, modal and direction words (`not`, `may`,
`will`, `increase`, `decrease`, `more`, `less`, `higher`, `lower` and others)
in the claim and the located span. Both are partial checks: they block common
meaning-changing edits; they do not prove that a substitution preserves
meaning.

## Command line

```bash
gtrex explain                        # the policy and every assumption
gtrex verify    claims.jsonl -o certified.jsonl --manifest run.json
gtrex repair    claims.jsonl -o repaired.jsonl --calibration calibration.json
gtrex calibrate labelled.jsonl -o calibration.json --alpha 0.05
gtrex sensitivity claims.jsonl       # the same claims under the three built-in policies
gtrex audit-export answers.csv -o audit.json
```

Input is JSON Lines. Each record needs a string `claim` and exactly one of an
inline `source` or a `source_path` (read as UTF-8, byte for byte):

```jsonl
{"id": "q1", "claim": "We depend on a limited number of suppliers", "source_path": "docs/report.txt"}
```

Other fields pass through unchanged and in input order, and results go under
one key, `gtrex`. A record that already has a `gtrex` field is refused unless
you pass `--replace-existing`. `verify --at-offsets` requires `start` and
`end` in every record and checks only that span.

`repair` takes its threshold from `--calibration`, which uses the one
`calibrate` wrote after checking that it was made for this policy's scorer and
pipeline, or from `--threshold`, or from the policy; otherwise it uses the
uncalibrated default 0.65 and says so. The manifest records the threshold and
its source (`calibration`, `threshold_flag`, `policy` or `default`). It stores the record's text in `claim` (in
`substitute` mode, the source's own text) and keeps the model's wording in
`gtrex.claim_original`, with the certificates `gtrex.before` and
`gtrex.after`. `gtrex.accepted` says the record was kept; `gtrex.grounded`
says its text verifies. An annotation is accepted and not grounded.

`calibrate` reads JSON Lines of `{"score": ..., "faithful": true, "group":
...}`, where `score` is a repair's `gtrex.alignment.score` and `faithful` is
your judgement of that repair. `sensitivity` reports, for each built-in
policy, the share of claims it accepts and the share contained as one
contiguous run.

`audit-export` checks that a CSV export kept an evidence value for every
populated answer column (by default `Q...` columns paired with
`Q..._evidence`) and exits nonzero if one is missing. It checks completeness
only, not the evidence itself.

Exit status is 0 on success and 1 on invalid input (the message names the
file and line), a failed calibration or an incomplete export. Output is
always UTF-8.

## Calibration, precisely

A repair is accepted when `score >= threshold`, and calibration uses the same
rule. `calibrate_risk` implements Learn-then-Test with fixed-sequence testing
over the observed scores (plus 1.0, or a grid you supply), using the exact
binomial tail as the p-value, and returns the least conservative certifiable
threshold among those candidates. The controlled quantity is the joint risk

    P(score >= threshold AND the repair is unfaithful)

for claims like the calibration claims that reach the alignment-score filter.
It is **not** `P(unfaithful | accepted)`, which is reported only as a point
estimate, and it does not cover claims accepted at verification or by
fragment salvage. The guarantee, `P(risk <= alpha) >= 1 - delta` over the
calibration draw, holds **only if the calibration examples are independent
draws from the claims you will repair**. Several examples from one document,
or several perturbations of one quotation, are not independent: give each
example a `group` (such as a document id), and `calibrate_risk` refuses
repeated groups. Overriding that is recorded in the calibration as
`uncertified`.

`calibrate_coverage` is split conformal on `1 - similarity`, giving
`P(a genuine repair is accepted) >= 1 - alpha` for exchangeable scores at the
same stage; the guards may still block some of those repairs afterwards.

A calibration records its target, stage, assumptions, a digest of its data,
and the identity of the scorer and pipeline its scores came from. A manifest
or `repair --calibration` refuses a calibration made for another
configuration, and the loader rejects incompatible method revisions,
acceptance rules, targets, stages and unknown fields. This validates the
file's declared format; it does not protect it against deliberate editing.
A threshold calibrated on synthetic perturbations is calibrated for those
perturbations. Use the certified value at full precision (scores and
thresholds are exported unrounded), because rounding it down admits scores
the certificate excluded. Both functions raise `CalibrationError` rather than
return an uncertifiable threshold.

## What it does not check

- **Entailment.** A verbatim quotation can be attached to the wrong entity,
  relation or direction. Grounding is necessary, not sufficient.
- **Recall.** Nothing here measures what the model failed to extract.
- **Other fields.** Verify the quotation, then validate your schema's other
  fields against it separately.

A rejected claim is not a detected hallucination. It is a claim the policy
will not vouch for, which also covers splicing, reordering, paraphrase,
parser artifacts, and a source that was not the document the model saw.

## Known limitations

- **Lexical boundaries are heuristic.** By default a match may not start or
  end inside a word, hyphenated term, signed figure, decimal, grouped number
  or time. HTML entities are decoded before the edges are checked, and
  apostrophe-suffixed names and sentence punctuation are allowed. Scripts
  that do not separate words with spaces need a language-specific tokenizer.
  `match_boundaries=False` turns the check off.
- **"Sound" is an assumption.** Case folding merges `mW` and `MW`, and
  typography folding merges a minus sign with a hyphen. Choose the ladder for
  your domain.
- **The guards are partial.** The numeric guard reads numbers by the English
  convention (`,` groups thousands, `.` marks decimals). A malformed number,
  or one written with a fraction character such as `1½`, must match
  verbatim, and a superscript is a separate figure. It ignores physical
  units, uncommon scale words, many sign conventions and spelled-out numbers.
  The polarity guard also reads negative words just before the located span.
  It misses some shifted negation scopes, it counts the month "May" as a
  modal, and it can refuse a valid repair because of a negation earlier in
  the sentence, such as in a preceding clause or table row. The guards run on
  alignment repair, not on claims that already verify as literal source
  text.
- **The fingerprint covers the configuration**, including every
  normalization step's id, version and parameters, plus the verifier
  revision. It is not a hash of the code, so report the package version with
  it.
- **Unicode.** Normalization is per character, so a decomposed accent does
  not match its precomposed form. Verdicts depend on Python's Unicode
  database, which the manifest records; the same fingerprint under another
  database is not a claim of identical verdicts.
- **Calibration is synthetic until you label your own repairs.** The
  guarantees hold for the population the calibration examples were drawn
  from, at the alignment stage only.
- **Repair.** Substitution trusts the aligner's best span above your
  threshold. Keep `gtrex.claim_original`, and review substituted records
  where it matters. Opted-in salvage keeps one verified fragment and records
  what it dropped; what was dropped is not certified.
- **Performance.** Each source is normalized once per tier, so very large
  corpora are slower than they need to be.

## Upgrading

1.1.0 is verifier revision 7, so every fingerprint changed (see the
changelog). Policy files written by 0.1.0b1 to 1.0.0 (revision 6) load with
their fields unchanged, warn, and get a new fingerprint; the old one is kept
in `migrated_from`. Files from earlier pre-releases also take the defaults
they lacked. In 1.0.0 the default
policies also watch "higher" and "lower", so the built-in fingerprints
changed. A calibration stays bound to the policy it was made for: use it
with that policy's file, or recalibrate.
The repository includes the 0.1.0b4 default policy as
`tests/fixtures/gtrex-0.1.0b4-policy.json` for reproducing that beta's
calibrated decisions. Its stored fingerprint is `63c0146e38cdb3cd`
(`df7bcdec27c3b89e` once migrated to revision 7).

## License

Apache-2.0.
