Views & the offset discipline

First, bank the win from Lesson 0001

Last time: every grammar implements recognize(text) returning RecognitionMatchs whose spans live in the original input's coordinates. Today you learn how that survives when matching has to happen on something that isn't the original input — folded case, stripped accents, deleted separators.

The bridge from regex: the translation problem

You've probably done this in pure regex:

m = re.search(r"kilogram", text.lower())   # matched on a COPY
m.span()                                   # …but these offsets point into the copy!

If text.lower() had changed any lengths, m.start() would be a lie about text. .lower() luckily preserves length — but accent-stripping, dropping separators, or collapsing whitespace do not. Paxman's answer: transformations always travel with a map back.

A view is a normalized copy of the text plus an offset map back to the original. Matches happen on views; spans are always reported through the map.

The machinery, in code

All three pieces live in paxman/core/grammar/scan_context.py. The map itself:

@dataclass(frozen=True, slots=True)
class View:
    subject: str                      # the normalized copy
    offsets: tuple[int, ...] | None   # None means "same length, 1:1"

    def original_span(self, s: int, e: int) -> tuple[int, int]:
        if self.offsets is None:
            return (s, e)
        return (self.offsets[s], self.offsets[e])

A normalizer produces exactly that pair (normalizers.py, Normalizer protocol):

def normalize(self, text: str) -> tuple[str, tuple[int, ...] | None]: ...

And ScanContext.view(name, normalizer) builds each view lazily, caches it by name, and asserts two invariants before trusting it: len(offsets) == len(subject) + 1, and every interval [offsets[i], offsets[i+1]) is non-empty and inside the text. The final entry is a sentinel mapping "just past the end of the subject". Matchability aside, that structure is why a map can never silently misalign.

(Each normalizer carries its own Provenance — CLDR, BCP 47, BIPM — because even "how we fold text" is a spec-backed decision here.)

Worked trace: follow one character home

Input "Côte d’Ivoire" (13 chars) under CountryNameFold:

The combining mark on ô evaporated, the apostrophe at index 6 was dropped, so the subject is 12 chars plus one sentinel entry — 13 offsets total. Now read matches off the map:

• subject hit [0, 12) → original_span(0, 12) = (0, 13) — the whole input, apostrophe and all.
• subject hit [5, 6) (just "d") → (5, 7) — the dropped apostrophe got absorbed into the region. That's legal: raw_text is simply text[5:7], deleted characters included.

Why the engine cares

Deduplication ("longer wins") compares spans numerically. If one grammar reported spans in view coordinates and another in original ones, containment math would be garbage. So the stage contract forbids the shortcut: core/grammar/stages.py — "stages must place normalized or transformed views in scratch… preserving RecognitionMatch offsets relative to the original input"; state.text must come through unchanged. Views are why grammars can be aggressive with folding and stay honest about position anyway.

Practice: translate spans yourself

Each round shows the original input and a hit reported in subject coordinates. Click the first and last characters of the corresponding span in the original — exactly what view.original_span() computes.

Retrieval checks

Read this next (primary source)

• Highest value, ten minutes: paxman/core/grammar/scan_context.py — it's ~80 lines. Read View and ScanContext.view() until the two asserts feel inevitable.
• Then skim: paxman/core/grammar/normalizers.py — spot the three archetypes among the real classes, and note the per-normalizer provenance.
• Connections: Country/grammar/name_recognition.py shows the full dance in situ — view request, matcher, original_span loop, plus a whole-input parity check kept deliberately redundant.