Metadata-Version: 2.5
Name: whoosh-compat
Version: 0.2.0
Summary: Whoosh query-language parser emitting programmatic tantivy queries
Project-URL: Repository, https://github.com/stumpylog/whoosh-compat
Author-email: Trenton H <797416+stumpylog@users.noreply.github.com>
License-Expression: BSD-2-Clause
License-File: LICENSE
License-File: NOTICE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.11
Requires-Dist: python-dateutil>=2.8
Provides-Extra: tantivy
Requires-Dist: tantivy>=0.26.0; extra == 'tantivy'
Description-Content-Type: text/markdown

# whoosh-compat

A standalone, typed, Python 3.11+ library that parses the
[Whoosh](https://github.com/whoosh-community/whoosh) query language into a
backend-neutral AST and emits **programmatically constructed**
[`tantivy.Query`](https://github.com/quickwit-oss/tantivy-py) objects: never
via tantivy's own string query parser.

It exists so that applications which used to build Whoosh query objects
directly (or hand-translate Whoosh-style query strings into another engine's
query language) can keep their existing, lenient, natural-language search
syntax while running on [tantivy](https://github.com/quickwit-oss/tantivy).
The motivating case is [paperless-ngx](https://github.com/paperless-ngx/paperless-ngx),
which migrated its search backend from Whoosh to tantivy but kept Whoosh's
query syntax as its user-facing search language. See
[paperless-ngx#13568](https://github.com/paperless-ngx/paperless-ngx/issues/13568)
for a concrete example of a query (`title:202[0-3]*`, a bracket-class
wildcard) that a naive string-translation layer gets wrong.

## Installation

```bash
pip install whoosh-compat[tantivy]
```

The core package (`pip install whoosh-compat`) depends only on
`python-dateutil`. The `tantivy` extra pulls in `tantivy-py`, which is
required to use `whoosh_compat.emitters.tantivy_`. The AST itself
(`whoosh_compat.ast`) and the parser (`whoosh_compat.parse`) have no tantivy
dependency, so a future backend (e.g. Meilisearch) can reuse them.

## Usage

```python
import tantivy

import whoosh_compat as wc
from whoosh_compat.emitters.tantivy_ import emit

# 1. Describe the fields the parser and emitter need to know about.
registry = wc.FieldRegistry(
    [
        wc.FieldSpec("content", wc.FieldKind.TEXT, analyzer=str.split),
        wc.FieldSpec("tag", wc.FieldKind.KEYWORD, comma_values=True, analyzer=str.split),
        wc.FieldSpec("created", wc.FieldKind.DATE, date_only=True, fast=True),
    ]
)

# 2. Parse a whoosh-style query string into a backend-neutral AST.
result = wc.parse(
    "tag:steuer AND created:[2020 TO 2020]",
    registry=registry,
    default_fields=["content"],
)
result.ast  # normalized AST root
result.diagnostics  # tuple[Diagnostic, ...], e.g. invalid dates

# 3. Emit a tantivy.Query against a real index and search it.
query = emit(result.ast, index=index, registry=registry)
searcher.search(query, limit=10)
```

`FieldSpec.analyzer` is how index-time tokenization parity is achieved: it's
a plain `Callable[[str], list[str]]` the host supplies (e.g.
`tantivy.TextAnalyzer.analyze`), called at *emit* time on `Term`/`Phrase`
text. tantivy's `term_query` does not tokenize on its own, so skipping this
step means a query for `"Invoices"` silently fails to match an index that
stored the lowercased token `"invoices"`.

See [ARCHITECTURE.md](./ARCHITECTURE.md) for the full `FieldSpec`/
`FieldRegistry` shape and how a query string gets from string to
`tantivy.Query`.

## API Stability

The public API boundary, as of 0.1.0:

**Stable API** (guaranteed across minor and patch releases):

- `whoosh_compat.parse()` and `whoosh_compat.ParseResult`
- `whoosh_compat.ast` module (the backend-neutral query tree), including
  `whoosh_compat.free_text_tokens()` (also exported at top level)
- `whoosh_compat.fields` module (`FieldSpec`, `FieldRegistry`, `FieldKind`, etc.)
- `whoosh_compat.errors` module (exception types)
- `whoosh_compat.emitters.tantivy_.emit()` function and the `Emitter` protocol in `whoosh_compat.emitters.base`

**Internal / not stable** (usable, but subject to change without notice between versions):

- `whoosh_compat.parser.*`: the forked whoosh tagger and filter pipeline. The parser is a fork of whoosh's own query parser, kept close to upstream so it stays diffable and easy to maintain. Because it tracks a third-party codebase, its internals and behavior may change between whoosh-compat releases, even minor ones.

The `parser.*` exemption forecloses nothing: it can always be promoted to stable later. In the meantime, if you import directly from `whoosh_compat.parser`, your code may need updates on whoosh-compat releases.

### Module naming: why `emitters.tantivy_`?

The `emitters.tantivy_` module uses a trailing underscore to signal that this is a backend-specific module. This naming is deliberate and permanent at the 0.1.0 release. While a future version could add a lazy re-export (e.g., `emitters.tantivy` without the underscore) to provide an alternative import path, the canonical import is and will remain `from whoosh_compat.emitters.tantivy_ import emit`. The underscore also reinforces that tantivy is an optional dependency: the emitter won't be imported unless you ask for it.

### The host contract

A host embedding this library (mapping failures to an HTTP 400, for
example) needs to check for **two** independent failure modes, not one:

1. **`ParseResult.diagnostics` is non-empty.** `parse()` never raises for
   bad *query* input; a malformed date, an out-of-domain number, or other
   field-kind-specific problem becomes a `Diagnostic` plus an `ErrorLeaf` in
   the tree instead. Calling `emit()` on a tree containing an `ErrorLeaf`
   raises `QueryError`.
2. **`emit()` raises `QueryError`.** This can happen even when
   `diagnostics` is empty: some query shapes parse cleanly but have no way
   to execute against tantivy today. The canonical example is a text-field
   range, `title:[a TO b]`: whoosh supported this, but tantivy-py has no
   programmatic text-range API (`DIVERGENCES.md` entry 5), so the query
   parses with `diagnostics == ()` and only fails once `emit()` is called.
   There is a single exception type now: `QueryError` always carries a
   `Diagnostic` (`err.diagnostic`) describing why.

**An empty `diagnostics` tuple does not, by itself, mean emitting is safe.**
Both checks matter; a host that only looks at `diagnostics` will still see
an uncaught `QueryError` bubble up for shapes like the one above.

**The one exception `parse()` itself can raise is `QueryParserError`, and it
never means the query was bad.** It means a defect in this library: the parse
pipeline is wrapped in a backstop that converts any unexpected exception into
that type, chaining the original as `__cause__`, so a host routes it to a
monitorable 500 instead of seeing (for example) a bare `RecursionError` from a
pathologically nested query. It is deliberately not a `Diagnostic`: reporting
an unknown internal failure as a 400 would blame the user for a bug on this
side and hide it from monitoring. (Misconfiguration passed to `parse()` --
an empty or unknown `default_fields`, a naive `basedate` -- still raises
`ValueError` eagerly, as documented on the function.)

**Branch on `Diagnostic.kind` and `Diagnostic.cause`; treat `message` as
log output, never parse it.** Both `ParseResult.diagnostics` entries and a
caught `QueryError`'s `.diagnostic` are structured records: `kind` is a
stable `DiagnosticKind` member a host can switch on (for example, mapping
`BAD_DATE` to a typed `InvalidDateQuery`), and `cause` is a coarser `Cause`
a host can use for routing without knowing every `DiagnosticKind`:

| `Cause`         | Meaning                                    | Typical host response       |
| --------------- | ------------------------------------------- | ---------------------------- |
| `INVALID_INPUT` | The query text itself is malformed          | HTTP 400                     |
| `UNSUPPORTED`   | The query is well-formed but this backend can't run it | HTTP 400        |
| `MISCONFIGURED` | The registry/schema setup is wrong          | Operator alert **and** HTTP 400 |
| `INTERNAL`      | The AST violates an invariant `parse()` would never produce | HTTP 500 |

`MISCONFIGURED` is the one cause that is two responses rather than one. It
means the registry and the index schema disagree, which only an operator can
fix, so it must raise an alert. But every `MISCONFIGURED` kind is reachable
from ordinary query text (`notes.user:*` for `EXISTS_REQUIRES_FAST`, and any
query naming a field the registry knows and the index schema does not for
`SCHEMA_FIELD_MISSING`), so a request is waiting on an answer that the alert
does not provide. The query cannot run whether or not anyone reads the
alert, and reporting it as a 500 would claim a defect in this library that
isn't there, so the request gets a 400.

`SCHEMA_FIELD_MISSING` is reported uniformly by every leaf that queries a
resolved field (term, phrase, prefix, wildcard, numeric and date range,
bare-`*` existence, and JSON subpaths): the drift is a property of the
field, not of the spelling that reaches it, so `content:x` and `content:x*`
never land on opposite sides of the 400/500 line for the same broken
deployment. Only the confirmed missing-field condition is reclassified; any
other refusal from tantivy-py remains `BACKEND_REJECTED`/`INTERNAL`, so a
genuine defect in this library is never hidden behind a 400.

`Diagnostic.message` (and the `QueryError` exception message, which is the
same string) carries no stability guarantee and may reword without notice.
Everything a host needs to act on is on the record's own fields instead: a
`DIVERGENCES.md` entry number on `divergence`, the field and its kind on
`field`/`field_kind`, the offending literal on `raw_value`, the span in
the query string on `startchar`/`endchar`, and, where a single concrete
rewrite of the query text would work, that rewrite on `suggestion` (see
"Adopting the library" below for how to apply it, and why it is a separate
field rather than a new `DiagnosticKind`).
Treat the message as developer/log output: a host showing errors to end
users should build its own copy from `kind`/`cause`, not display or parse
`message`; the paperless-ngx integration does exactly this.

A `Diagnostic`'s severity is fatal-only, and always will be: there is no
`severity` field, and none is planned. Any diagnostics present means the
query cannot be emitted, full stop; there is no "warning" tier to weigh
differently. A future informational-only signal (for example, reporting
that a zero-token term was silently dropped during analysis) would use a
separate channel, never `ParseResult.diagnostics`.

### Free-text tokens for secondary clauses

`whoosh_compat.free_text_tokens(node, registry=..., fields=...)` answers
"which plain words does this query search for?" for hosts that blend a
secondary text clause alongside the emitted query: the motivating case is
paperless-ngx's fuzzy-matching blend, which re-parses a word string
through tantivy's own query parser and must never receive whoosh grammar.
It returns the analyzed `Term`/`Phrase` tokens on the requested
TEXT/KEYWORD fields, deduplicated in first-appearance order, with the
subtle rules handled here rather than in each host: negated subtrees
(`NOT x`, `ANDNOT`'s negative side) contribute nothing, patterns and
ranges contribute nothing, and a word the multifield expansion copied
onto several default fields counts once. Tokens are the field analyzer's
output verbatim, never re-split; see the function's docstring for the
full contract.

Pass the tree **as parsed** (`ParseResult.ast`). The negation rule is why:
`analyze()` deliberately collapses an `AndNot` whose positive side
analyzed to nothing, leaving the negative side standing alone as an
ordinary positive node (`DIVERGENCES.md` entry 23), so a tree you analyzed
yourself before calling no longer records what the user excluded, and no
walk can recover it.

Pass `analyzed=False` to get the raw text each contributing leaf was
parsed from instead of the analyzer's output. Do that whenever the tokens
are going back into a parser that will analyze them again: analysis is
not generally idempotent (a stemmer maps `universities` to `univers` and
then `univers` to `univ`), so re-analyzing analyzed output searches for
something the index does not contain. In that mode the analyzer is never
consulted. Which *nodes* contribute is structural (negation, patterns,
kinds and dedupe never vary by mode) with one exception, the zero-token
leaf in the first row below; the other two rows differ in text only:

| query | `analyzed=True` | `analyzed=False` |
| --- | --- | --- |
| `the` (a stopword) | `()` | `('the',)` |
| `"tax reports"` | `('tax', 'report')` | `('tax reports',)` |
| `alpha-beta` (analyzer splits it) | `('alpha', 'beta')` | `('alpha-beta',)` |

An all-stopword leaf analyzes to nothing, so it contributes no entry at
all in analyzed mode, while unanalyzed mode reports its raw text: that is
a node present in one mode and absent from the other, not two spellings of
one node. Deciding *membership* by the analyzer while refusing its
*output* would be a half-analysis this mode's contract denies; the
re-parse downstream applies its own stopword list, in its own index's
terms.

So an entry in this mode can contain whitespace and punctuation, and the
number of entries is the number of contributing leaves rather than of
words. It can also contain characters a *re-parse* would read as grammar
(a colon, a bracket) even though the query grammar around them is gone:
quote or escape before re-parsing, and note that a whole-token filter
(`\w+`) over these entries drops every hyphenated, dotted or
quoted-phrase term outright, because the text is untokenized.

**Cap query length at the host boundary.** Parse time is quadratic in the
length of a long run of word characters containing no `:` (the fieldname
tagger's regex, `[\w.]+:`, scans toward end-of-input and fails at each
successive position). Measured on one developer machine, so treat these
as order-of-magnitude and the ~4x-per-doubling growth as the durable
claim: 10KB ~1 second, 20KB ~4 seconds, 40KB ~15 seconds, 60KB ~34
seconds. This library's tagger regex differs from upstream whoosh's in
exactly one way, permitting `.` inside a field name so that dotted JSON
subpaths tag (`[\w.]+:` here against whoosh's `\w+:`); that adds no `:` to
the run being scanned and so does not change the scan's character. The
parity claim rests on measurement rather than on sameness: real whoosh
shows the same curve on the same input (measured 10KB ~1.1 seconds, 20KB
~4.0 seconds against the pinned oracle). The cost is inherited
deliberately rather than fixed with a rewritten tagger regex whose subtle
behavior differences would risk parity. The parser's own
nesting-depth cap bounds recursion, not CPU time, so a host accepting
untrusted query strings should enforce its own length limit (a few KB
comfortably covers any human-written query) before calling `parse()`.

### Adopting the library: sweep stored queries first

A host switching to this library from real Whoosh usually carries a body of
stored queries written against the old engine: saved views, bookmarks,
scheduled searches. Some of those queries never worked the way their author
intended, and real Whoosh gave no sign of it. Parsed by the pinned oracle at
a base date in 2026, `created:december 2019` resolves to a window over
**December 2026** and searches for `2019` as a free-text word. Nobody who
saved that query was told anything was wrong.

Where this library rejects such a value instead (`DIVERGENCES.md` entry 61),
the stored query stops returning wrong documents and starts returning a
diagnostic. That is the intended improvement, but it lands on users who were
not aware they had a broken query, so it is worth doing before cutting over
rather than discovering it in production.

Parse every stored query and look at what comes back. Three outcomes matter:

1. **No diagnostics, `emit()` succeeds.** Nothing to do.
2. **A diagnostic carrying a `suggestion`.** The unquoted multi-word date
   family is the case that has one today. `suggestion` is the replacement
   text for that diagnostic's own `startchar`/`endchar` span, so the host
   splices rather than re-deriving the rule. Apply one query's diagnostics
   in descending `startchar` order, so rewriting one value does not shift
   the spans before it:

   ```python
   result = whoosh_compat.parse(q, registry=..., default_fields=..., tz=..., basedate=...)
   out = q
   for d in sorted(result.diagnostics, key=lambda d: -(d.startchar or 0)):
       if d.suggestion is not None:
           out = out[: d.startchar] + d.suggestion + out[d.endchar :]
   ```

   | stored query | rewritten |
   | --- | --- |
   | `created:december 2019` | `created:"december 2019"` |
   | `created:2020 to 2021` | `created:"2020 to 2021"` |
   | `created:previous month to now` | `created:"previous month to now"` |
   | `created:december 2019 OR added:2020 august 4` | `created:"december 2019" OR added:"2020 august 4"` |

3. **A diagnostic with `suggestion is None`.** No single rewrite of the
   query text would work: a malformed date (`created:20231340`), a value
   the date grammar does not recognise at all (`created:last week`, where
   quoting does not help either), a pattern on a numeric or BOOLEAN_EXISTS
   field (`type_id:1*`), or a shape with two possible fixes that mean
   different things (`title:200[1-9]`). These need a human, or a decision
   to drop the clause.

Branch on `suggestion is not None`, not on `kind`. The same `kind` carries a
suggestion for one query and not for another: `BAD_DATE` covers both
`created:december 2019`, which has a working quoted spelling, and
`created:20231340`, which has none. That is why the signal is a separate
field rather than a new `DiagnosticKind`, which would also have broken every
host already branching on `BAD_DATE`.

Re-parsing the rewritten query before storing it is still worth doing as a
belt-and-braces check, but it is no longer what tells the two cases apart.

## Supported query syntax

Parity target is **Whoosh's intended grammar**, not every Whoosh plugin.
See `ARCHITECTURE.md` for why the parser is a fork rather than a
reimplementation, and `DIVERGENCES.md` for every point where whoosh-compat's
behavior intentionally differs from real Whoosh.

| Syntax | Example | Notes |
|---|---|---|
| Bare terms, implicit AND | `invoice total` | both terms required (Whoosh's own semantics) |
| Boolean operators (uppercase only) | `a AND b`, `a OR b`, `NOT a`, `a ANDNOT b`, `a ANDMAYBE b`, `a REQUIRE b` | lowercase `and`/`or`/`not` are plain text, matching Whoosh's operator regexes; `REQUIRE` is infix like the others |
| Grouping | `(a OR b) AND c` | |
| Fielded terms | `title:invoice` | |
| Field aliases | `type:invoice` → `document_type:invoice` | host-configured on `FieldSpec.aliases` |
| Phrases | `"exact phrase"`, `"a b"~2` | slop follows Whoosh semantics: `1` = adjacent |
| Wildcards | `inv*`, `inv?ice`, `202[0-3]*` | glob syntax including bracket character classes |
| Prefix | `inv*` (no other wildcard chars) | folds to a prefix query |
| Ranges | `created:[2020 TO 2025]`, `asn:[100 to]` | numeric and date; open-ended on either side |
| Boost | `title:invoice^2.0` | |
| Comma value lists | `tag:foo,bar` → `tag:foo AND tag:bar` | per-field opt-in (`FieldSpec.comma_values`); `tag:'foo,bar'` quoting keeps it a single literal |
| Every / exists | `*`, `*:*`, `title:*` | |
| Dates | `created:2020`, `created:today`, `created:previous month`, `created:'previous month'`, `created:now-7d` | full grammar: ISO/compact forms, natural-language keywords, relative offsets (`now-7d`, `-1 week`); the six multi-word keywords (`previous week`/`month`/`quarter`/`year`, `this month`/`year`) parse quoted or bare, an extension over Whoosh, where a value ends at the first space; a time of day written after one of them belongs to the date value too (narrowing it for the four calendar-unit keywords, rejected for the two span-valued ones), see `DIVERGENCES.md` entries 19 and 52. Every *other* unquoted multi-word date value is rejected with a `BAD_DATE` diagnostic naming the whole value rather than silently truncated to its first word: `created:december 2019`, `created:2020 to 2021` and even `created:previous month to now` (a keyword phrase is joined first, then read as the start of a longer run) all error, and the quoted or bracketed spelling (`created:"december 2019"`, `created:[2020 TO 2021]`) is the one that works, see entry 61 |
| RFC3339 datetimes | `created:[2020-01-01T00:00:00Z TO 2020-06-01T00:00:00Z]`, `created:'2020-01-01T00:00:00Z'` | `T` joins the separator class and a trailing `Z` is honored as the UTC designator (an absolute instant, not local time); an extension over Whoosh, which cannot parse these correctly (quoted `T`/`Z` values parse to nothing; range bounds collapse to their leading year), see `DIVERGENCES.md` entries 12 and 48-50. Quote it or bracket it: the bare unquoted spelling (`created:2020-01-01T00:00:00Z`) is split at its colons by the field-separator rule before any date parsing happens, and the half of it that survives is rejected as a bad date rather than searched as the shorter period it looks like, see entry 54 |
| JSON subpaths | `notes.user:alice` | an extension with no equivalent in Whoosh itself (registered per-field via `FieldSpec.subpaths`) |
| Default JSON subpath | `notes:alice` → `notes.user:alice` | per-field opt-in: the one subpath declared `SubpathSpec(default=True)` is what a bare mention of the field means, so a host never has to rewrite `notes:` in the raw query string. Without a default, a bare JSON field name stays unrecognized and demotes to text |

Not carried over from Whoosh (not currently implemented, kept cheap to add
via the forked plugin architecture): `asn:>100` (`GtLtPlugin`), `term~2`
fuzzy matching, `r"regex"` literal regex queries, `SequencePlugin`,
`-foo`/`+foo` as negation/requirement shorthand (in the whoosh grammar this
library targets, `-foo` was plain text whose analyzer typically dropped the
dash: `NOT` was the only negation operator), and free-date mode (implicit
date parsing in an unfielded-date context; the parser defaults to the
date-parsing plugin for fielded dates when the host calls `parse()` with a
date-aware registry instead).

### Divergences from real Whoosh

**AST-level divergence does not always mean result-level divergence.**
whoosh-compat is tested at two separate layers. See
[`DIVERGENCES.md`](./DIVERGENCES.md) entry 16 for the full explanation, and
`tests/emitter/test_acceptance_e2e.py`'s module docstring for worked
examples: a documented, real difference in the *parsed AST* between
whoosh-compat and real Whoosh (e.g. how a wildcard pattern's case-folding is
sequenced) can still produce the *same final matched-document set*, because
the divergence gets absorbed somewhere downstream (e.g. both sides' text
analyzers end up lowercasing the same way regardless). Read
`DIVERGENCES.md` for what's intentionally different and why; don't assume an
entry there implies a query result bug without checking whether it's one of
the entries called out as AST-only.

### The analyzer / pattern_normalizer seam

Two separate callables on `FieldSpec`, deliberately not unified into one:

- **`analyzer`** (`Callable[[str], list[str]]`): the *full* token-level
  chain (lowercase, ASCII-fold, stemming, stopword removal, whatever the
  host's index-time analyzer does) applied to `Term`/`Phrase` query text at
  emit time, so query tokens match what's actually in the index.
- **`pattern_normalizer`** (`PatternNormalizer`, i.e.
  `Callable[[str], str | Sequence[str]]`): a *narrower*, fragment-level
  transform applied to each literal segment of a `Wildcard`/`Prefix`
  pattern. It never tokenizes and never drops a fragment; beyond that it is
  usually lowercase + ASCII-fold, and on a stemmed field it also offers the
  segment's stem. It may return **one** form of the segment (a bare `str`)
  or **several alternatives** (a sequence); the emitter matches a term
  satisfying *any* of them. Inside a bracket class it is additionally held
  to being character-level, because that is the one place a fragment is a
  single character and must stay one (see below).

These have to be different callables. The analyzer answers "what tokens does
this *value* become"; the pattern normalizer answers "what could this
*fragment of a glob* look like in the index", which is not the same question:
a fragment is usually not a word (`inv*oices`), it can never be split into
several tokens or dropped, and inside a bracket class it has to stay exactly
one character long.

The alternatives are what let a pattern reach a **stemmed** index without
giving up the spelling the user typed. English Snowball substitutes rather
than truncates, so neither form alone is enough: `company` stems to
`compani` (only the stem finds the indexed term) while `copyright` is its own
stem (only the typed run finds it). A host with a stemmed field returns both
forms and gets both documents; measured over a 4,977-word vocabulary, 3.5% of
words stem to something that is not a prefix of themselves, so no
"use the stem when ..." rule over a single string separates the two cases.
Each literal run alternates independently, so a many-run pattern costs the
*sum* of its alternatives, not the product.

Two properties of the seam survive that widening. Inside a bracket class the
normalizer is still applied one character at a time and only when it answers
with exactly one alternative exactly one character long (a class position
matches one character, and every offset in the glob translation is taken
against the body's length). And a normalizer is never asked to be
*correct* on a fragment that is not a word: stemming `oices` (from
`inv*oices`) is morphological nonsense, but as an *added* alternative it can
only widen the match, never move it, which is why alternatives replaced the
single-string form rather than joining it.

### Timezone handling

`DateRange` bounds inside the AST are always timezone-aware UTC `datetime`s.
The tantivy emitter converts them to **naive UTC** before calling
`Query.range_query` (see `_to_naive_utc` in
`src/whoosh_compat/emitters/tantivy_.py`), because `tantivy-py <= 0.26.0`'s
`range_query` only accepts naive datetimes for `FieldType.Date`. Passing a
tz-aware one raises `ValueError`. This is worked around here rather than
relied on upstream because the fix
([tantivy-py#666](https://github.com/quickwit-oss/tantivy-py/pull/666))
merged *after* 0.26.0 was tagged. Naive input is passed straight through
(tantivy already treats naive datetimes as UTC at index time, matching how
documents are indexed).

### The JSON subpath carve-out

`notes.user:alice`-style dotted-field queries (`FieldKind.JSON`) are the one
place this library emits through `index.parse_query()` instead of
constructing a `tantivy.Query` programmatically: the installed `tantivy-py`'s
`Query.term_query` cannot address a JSON subpath by exact field name: it
raises as if the field didn't exist at all. The emitter feature-detects this
per process and falls back to a strictly escaped/quoted `parse_query` call
for just that one leaf. This carve-out retires itself automatically once
[tantivy-py#716](https://github.com/quickwit-oss/tantivy-py/pull/716) (which
routes `make_term` through JSON path resolution) lands and ships. No code
change is needed here, the feature-detection just starts taking the other
branch.

## Development

Install with dev dependencies (uses [uv](https://docs.astral.sh/uv/)):

```bash
uv sync --group dev
```

Run the checks CI runs:

```bash
uv run ruff check .
uv run mypy src
uv run pytest tests
```

This repo also ships a [`.pre-commit-config.yaml`](./.pre-commit-config.yaml)
covering the cheap, mechanical checks (whitespace, YAML/TOML validity,
codespell, `zizmor` for the GitHub Actions workflow, `ruff check`, keeping
`uv.lock` in sync). Run it with [prek](https://github.com/j178/prek) (a
faster, dependency-free reimplementation of `pre-commit` that reads the
same config file) or `pre-commit` itself:

```bash
uvx prek run --all-files
# or: uvx pre-commit run --all-files
```

### Testing layers

1. **Unit tests** (`tests/`): parser, AST, `normalize()`, and `FieldSpec`/
   `FieldRegistry` behavior in isolation.
2. **Differential tests** (`tests/differential/`): parse the same corpus of
   query strings through both whoosh-compat and a **real, pinned Whoosh**
   (a test-only dependency; see the git-ref pin in `pyproject.toml`'s
   `dev` group, this fork carries parser fixes absent from the PyPI
   release) and compare the resulting trees. Divergences must match an
   explicit allowlist (`tests/differential/allowlist.py`), each entry cross-
   referencing a `DIVERGENCES.md` entry for *why* it's expected.
3. **End-to-end acceptance tests** (`tests/emitter/test_acceptance_e2e.py`):
   the same fixture documents indexed twice (once in a real Whoosh index,
   once in tantivy), full query strings run against both, and the *matched
   document ID sets* compared: result-level parity, not tree shape.

Because layer 2 needs a real Whoosh installation as an oracle, it's pulled
in as a dev-only dependency (pinned by git ref, not the PyPI 2.7.4 release)
rather than a runtime dependency of the library itself.

### Property-based / fuzz testing

`tests/differential/strategies.py` is a grammar-aware [Hypothesis](https://hypothesis.readthedocs.io/)
strategy covering the whole supported query language (README's syntax
table above): nested groups, every operator, wildcards/ranges/phrases/
comma-lists/boosts/JSON subpaths, and deliberately placed zero-token
values (an all-stopword term/phrase, see `strategies.ZERO_TOKEN_WORDS`).
It drives five properties:

- `tests/differential/test_hypothesis.py::test_fuzz_grammar_matches_oracle`:
  the same AST-shape parity check as the static corpus (layer 2 above), but
  over generated, nested queries, guided by `hypothesis.target()` toward
  structurally rich examples (more nodes, deeper nesting, more distinct
  node types, more zero-token leaves buried inside a larger structure).
  Seeded with every static corpus line plus a few strings pulled directly
  from `DIVERGENCES.md` entries, via `@example()`.
- `tests/differential/test_hypothesis.py::test_normalize_is_total_and_idempotent`:
  `normalize(normalize(x)) == normalize(x)` for every freshly parsed AST,
  and `normalize()` never raises.
- `tests/test_parse_never_raises.py::test_parse_raises_nothing_but_query_parser_error`:
  over the same generated grammar (drawn deeper than the shared strategy's
  committed default), `parse()` raises nothing but `QueryParserError`, the
  documented "library defect" type. Sits alongside regression anchors for
  every escape route found so far.
- `tests/emitter/test_hypothesis_e2e.py::test_emit_never_raises_except_unsupported`:
  parsing a query that produced no diagnostics, then emitting it against a
  real in-memory tantivy index, never raises a `QueryError` whose
  `diagnostic.cause` is anything other than `Cause.UNSUPPORTED` (the
  documented case of a construct that parses cleanly but has no way to
  execute against tantivy, such as `DIVERGENCES.md` entry 5).
- `tests/emitter/test_hypothesis_e2e.py::test_normalize_idempotent_on_emitter_registry_grammar`:
  the same `normalize()` property again, against the emitter registry's
  own (smaller, JSON/BOOLEAN_EXISTS-carrying) field vocabulary.

These run at a modest `max_examples` in CI so the suite stays fast. To run
a longer local soak (recommended before a release, or after touching
`parser/`, `ast.py`, or `emitters/`), raise the example count for a single
run without editing the files.

Note that a Hypothesis **profile cannot do this**. Registering a profile
with a higher `max_examples` and loading it has no effect here, because
every property in this repository sets `max_examples` explicitly in its
own `@settings(...)`, and an explicit value beats the profile default no
matter when the profile is loaded. A run set up that way silently executes
the committed CI counts while appearing to run thousands of examples
(`--hypothesis-show-statistics` reports the real number, and says
`Stopped because settings.max_examples=300`).

What does work is rewriting the settings object the `@given` wrapper reads
at call time. Save this as `soak_plugin.py` somewhere on `PYTHONPATH` and
pass it with `-p`:

```python
import os

import hypothesis

_TARGET = int(os.environ.get("SOAK_MAX_EXAMPLES", "5000"))


def pytest_collection_modifyitems(session, config, items):
    for item in items:
        fn = getattr(item, "obj", None)
        current = getattr(fn, "_hypothesis_internal_use_settings", None)
        if current is None or current.max_examples >= _TARGET:
            continue
        fn._hypothesis_internal_use_settings = hypothesis.settings(
            current, max_examples=_TARGET, deadline=None
        )
```

```bash
SOAK_MAX_EXAMPLES=5000 uv run pytest   tests/differential/test_hypothesis.py tests/emitter/test_hypothesis_e2e.py   -p soak_plugin -q --hypothesis-show-statistics
```

Two properties want their own (smaller) target rather than this one.
`tests/emitter/test_acceptance_property.py`'s generated-query property
runs a real whoosh search *and* a real tantivy search per example, and
takes its count from the `WHOOSH_COMPAT_ACCEPTANCE_SOAK_EXAMPLES`
environment variable instead. And `test_alternating_nesting_depth_cost_budget`
asserts on elapsed wall-clock time, so it is marked `wall_clock`: any
runner that adds instrumentation must deselect it with
`-m "not wall_clock"`.

Editing the `max_examples` values in those files directly, for the
duration of the run, also works and needs no plugin. Keep long soaks
(thousands of examples) **out of CI**: they take minutes, not seconds, and
are meant for local/pre-release verification, not every push.

**HypoFuzz** (coverage-guided fuzzing that runs existing Hypothesis
properties under instrumentation) was evaluated for this purpose and is
**not used**: its license (`Zac-HD/hypofuzz`'s `LICENSE`, checked directly)
grants use "for non-commercial purposes only", explicitly requires a
separate paid commercial license for "use within a commercial
organization, including internal tooling or testing" and "use in
continuous integration or development pipelines for commercial products",
and prohibits modification/redistribution without permission. That's
incompatible with this BSD-2-Clause project (whoosh-compat is itself used
by, and expected to be run in CI by, commercial downstream users like
paperless-ngx installs), so it was not added as a dependency. The plain-
Hypothesis soak profile above is the recommended way to get a similar
"run longer, look harder" effect without it.

## License

BSD-2-Clause, see [LICENSE](./LICENSE). This project's parser
(`whoosh_compat/parser/`) is forked from
[whoosh-community/whoosh](https://github.com/whoosh-community/whoosh) (a
fork of Matt Chaput's original [Whoosh](https://github.com/mchaput/whoosh),
also on [PyPI](https://pypi.org/project/Whoosh/); the whoosh-community fork
is itself unmaintained). Forked files retain their original BSD-2-Clause
header. See [NOTICE](./NOTICE).
