Metadata-Version: 2.5
Name: internetbook-analysis
Version: 0.1.0
Summary: Text analysis and BM25 weighting shared by internetbook indexing and search
Project-URL: Repository, https://github.com/gyldendalsolutions/internetbook-analysis
Author: Gyldendal Solutions
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.12
Requires-Dist: snowballstemmer==3.1.1
Provides-Extra: dev
Requires-Dist: pytest<10.0.0,>=8.0.0; extra == 'dev'
Description-Content-Type: text/markdown

# internetbook-analysis

Text analysis and BM25 weighting shared by internetbook indexing and search.

The internetbook indexer and the search API must turn text into terms the same way: a query term the indexer spelled differently matches nothing, and nothing reports it. This package is the one implementation both use.

## What it does

- **Analysis:** tokenises, case-folds, drops stopwords, optionally adds compound-word parts, and stems with Snowball. Supported: Danish, English, German, Spanish, French, Italian, Norwegian and Swedish. Faroese, Icelandic and Greenlandic have no Snowball stemmer or stopword list; they are left unstemmed and keep every word, rather than borrowing a neighbour's rules.
- **BM25, split between documents and queries:**
  - The document side stores saturated, length-normalised term frequencies.
  - The query side applies IDF.
  - Their dot product is the BM25 score. Document vectors don't change when document frequencies do.
- **pgvector sparse vectors:** produces the text form of `sparsevec` values.

```python
from internetbook_analysis import (
    TermStatistics, analyzer_for, document_weights, query_weights,
    term_frequencies, to_sparsevec,
)

terms = analyzer_for("da").analyze("Planterne laver fotosyntese")
# ['plant', 'lav', 'fotosyntes']

# Document side: term ids come from the indexer's term dictionary.
ids = {"plant": 1, "lav": 2, "fotosyntes": 3}
doc = to_sparsevec(document_weights(
    term_frequencies(ids[t] for t in terms), len(terms), average_length=120.0,
))

# Query side: IDF from document frequencies.
stats = {"plant": TermStatistics(term_id=1, document_frequency=420)}
query = to_sparsevec(query_weights(analyzer_for("da").analyze("planter"), stats, 3_500_000))
```

## Versioning

`ANALYZER_VERSION` names the analysis output. It changes whenever the same input would produce different terms: a new stopword, tokeniser rule, decompounder or stemmer version. A change means re-indexing everything, so:

- Consumers pin this package exactly (`==`).
- `snowballstemmer` is pinned exactly here, and upgrading it is a new `ANALYZER_VERSION`.
- `tests/golden/` holds expected output for real sentences. A change to it is a change to `ANALYZER_VERSION`.

The package version is separate. Releases that don't change output (documentation, new helpers) keep `ANALYZER_VERSION`.

## Stopwords

The lists in `src/internetbook_analysis/stopwords/` are the Snowball project's `stop.txt` files, copied unmodified; their licence is in `NOTICE`. Solr's EXT:solr configsets use slightly edited copies of the same lists (English swaps `a` for `us`, Swedish has `sitta` for `sitt`, French adds ten forms). This package follows Snowball, not those edits.

## Constants that must match elsewhere

- `BM25_K1`, `BM25_B`: the `k1`/`b` options of the pg_textsearch BM25 index.
- `SPARSE_DIMENSIONS`: the dimension of every `sparsevec` written or queried.
