Metadata-Version: 2.5
Name: laghu
Version: 0.1.1
Summary: model2vec static models in a fraction of the memory, with the same vectors
Project-URL: Repository, https://github.com/vedicreader/laghu
Project-URL: Documentation, https://vedicreader.github.io/laghu/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: embeddings,mmap,model2vec,nbdev,static-embeddings,tokenizers,unigram
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: fastcore>=1.8
Requires-Dist: huggingface-hub>=0.24
Requires-Dist: numpy>=1.24
Requires-Dist: tokenizers>=0.20
Provides-Extra: fallback
Requires-Dist: model2vec>=0.9; extra == 'fallback'
Description-Content-Type: text/markdown

# laghu


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

A [model2vec](https://github.com/MinishLab/model2vec) static embedding model loaded with laghu (Sanskrit for light) has `encode`, `encode_as_sequence` and `tokenize`. Its output matches model2vec’s bit for bit, apart from the empty-input difference documented below.

model2vec reads the embedding table into each process. For potion-multilingual-128M, the table takes 512 MB. Its Unigram tokenizer adds a character trie over 500,353 pieces, about another 610 MB. After encoding, the process uses about 1.3 GB of private memory.

laghu maps the table read-only from `model.safetensors`. Processes share the mapped pages through the operating system’s page cache. The operating system can reclaim clean pages under memory pressure. For supported Unigram models, laghu replaces the trie with Python Viterbi over a dict of pieces. The normalizer and pre-tokenizer come from `tokenizers`, built from the same tokenizer.json with the vocabulary removed. The same model then takes about 135 MB of private memory per process.

WordPiece, BPE and word-level models still use `tokenizers`. laghu maps their embedding tables but does not replace their tokenizers. The memory saving depends on the table’s size relative to the tokenizer.

## Install

``` sh
uv add laghu        # or: pip install laghu
```

model2vec itself is not a dependency. For the fallback described below, and for [`same_as_model2vec`](https://vedicreader.github.io/laghu/encoder.html#same_as_model2vec), install the extra:
`uv add 'laghu[fallback]'`.

## Use

``` python
from laghu import load

m = load('minishlab/potion-multilingual-128M')      # a folder, or a Hugging Face repo id (the local cache is tried first)
v = m.encode(['The quick brown fox.', 'कर्मण्येवाधिकारस्ते'])   # float32, (2, 256), unit rows: the same array model2vec returns
m.tokenize(['hello world'])
m.encode_as_sequence('hello world').shape          # (n_tokens, 256)
```

    (2, 256)

[`load`](https://vedicreader.github.io/laghu/encoder.html#load) returns a [`LeanModel`](https://vedicreader.github.io/laghu/encoder.html#leanmodel). If loading fails, it logs the cause and tries model2vec’s `StaticModel`, which needs `laghu[fallback]`. If the fallback also fails, [`load`](https://vedicreader.github.io/laghu/encoder.html#load) raises the original laghu error. `load(..., fallback=False)` raises without trying model2vec. Both loaders refuse bfloat16 and float8 tables because numpy cannot hold those dtypes.

`same_as_model2vec(path)` loads a model both ways and compares them on the built-in corpus, or on texts of your own:

``` python
from laghu import same_as_model2vec
same_as_model2vec('minishlab/potion-base-8M', ['my', 'own texts'])
```

    /Users/71293/code/personal/orgs/laghu/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
      from .autonotebook import tqdm as notebook_tqdm

    {'lean': False,
     'n': 2,
     'ids': True,
     'dtype': True,
     'f32': True,
     'f16': True,
     'exact': True,
     'seq': True,
     'median': True,
     'unk': True,
     'tokens': True}

It compares the token ids and the vectors cast to float32 and to float16. It also compares the exact dtype and bits, `encode_as_sequence`,
`median_token_length`, the unknown token id and the token list.

## What matches, and what it costs

Every model below matches model2vec’s ids and vectors, bit for bit, on 4,726 texts. The corpus includes prose in 36 languages and random strings from 30 Unicode blocks. It also tests control characters, special tokens, emoji sequences, combining marks, bidi controls and a 200,000-character string. Each memory measurement uses a fresh process. On macOS the helper reports `phys_footprint` after encoding and `lifetime_max_phys_footprint` for peak. On Linux it reports `RssAnon` and peak resident memory (`VmHWM`). Speed is `encode` over 3,226 texts on an M3 Max.

| model | tokenizer | laghu tokenizes with | vocab × dims | dtype | norm | identical | private MB, model2vec → laghu | peak MB, model2vec → laghu | texts/s, model2vec → laghu |
|----|----|----|----|----|----|----|----|----|----|
| potion-base-2M | WordPiece | tokenizers | 29,528 × 64 | float32 | on | yes | 81 → 58 | 82 → 58 | 84,510 → 92,982 |
| potion-base-4M | WordPiece | tokenizers | 29,528 × 128 | float32 | on | yes | 94 → 62 | 95 → 62 | 87,248 → 90,242 |
| potion-base-8M | WordPiece | tokenizers | 29,528 × 256 | float32 | on | yes | 114 → 70 | 115 → 70 | 79,415 → 81,810 |
| potion-base-32M | WordPiece | tokenizers | 63,091 × 512 | float32 | on | yes | 244 → 97 | 244 → 97 | 75,009 → 81,379 |
| potion-retrieval-32M | WordPiece | tokenizers | 63,091 × 512 | float32 | on | yes | 244 → 97 | 244 → 97 | 70,799 → 79,717 |
| potion-science-8M | WordPiece | tokenizers | 29,528 × 256 | float32 | on | yes | 113 → 70 | 114 → 70 | 81,008 → 84,532 |
| potion-code-16M | WordPiece | tokenizers | 61,826 × 256 | float32 +map +weights | on | yes | 147 → 110 | 211 → 117 | 10,674 → 10,425 |
| potion-code-16M-v2 | WordPiece | tokenizers | 63,457 × 256 | float16 | on | yes | 108 → 101 | 186 → 119 | 21,819 → 22,164 |
| potion-multilingual-128M | Unigram | Python Viterbi | 500,353 × 256 | float32 | on | yes | 1,348 → 135 | 1,348 → 135 | 39,105 → 12,795 |
| M2V_base_output | WordPiece | tokenizers | 29,528 × 256 | float32 | off | yes | 112 → 69 | 113 → 69 | 65,460 → 87,100 |
| M2V_multilingual_output | WordPiece | tokenizers | 501,054 × 256 | float32 | off | yes | 834 → 198 | 842 → 295 | 42,565 → 90,283 |
| M2V_base_glove | WordLevel | tokenizers | 400,002 × 256 | float32 | off | yes | 656 → 186 | 656 → 194 | 67,028 → 85,058 |
| M2V_base_glove_subword | WordPiece | tokenizers | 400,779 × 256 | float32 | off | yes | 647 → 178 | 649 → 195 | 66,256 → 85,430 |
| jina-embeddings-v3-separation-distilled | Unigram | Python Viterbi | 250,002 × 256 | float32 | off | yes | 722 → 88 | 722 → 88 | 51,836 → 20,840 |
| multilingual-e5-large-m2v | Unigram | Python Viterbi | 250,002 × 512 | float32 | on | yes | 981 → 109 | 981 → 109 | 24,139 → 19,670 |
| m2v-gemma-embedding-300m | Unigram | Python Viterbi | 255,732 × 256 | float16 +weights | on | yes | 542 → 93 | 542 → 93 | 20,319 → 12,134 |
| potion-multilingual-128M-i8-tokenade | Unigram | Python Viterbi | 500,353 × 256 | int8 | on | yes | 985 → 138 | 985 → 138 | 27,676 → 12,090 |
| m2v-qwen3-embedding-0.6b-1024d | Unigram | Python Viterbi | 151,644 × 1024 | float16 +weights | on | yes | 602 → 132 | 602 → 132 | 16,792 → 7,168 |
| ovos-m2v-intents-es-ES-bne-v5 | Unigram | Python Viterbi | 50,259 × 64 | float32 | on | yes | 169 → 52 | 169 → 52 | 11,786 → 11,265 |
| m2v-gte-256-edu | Unigram | tokenizers | 313,845 × 256 | float32 | on | yes | 1,028 → 663 | 1,028 → 675 | 25,363 → 24,981 |
| bge-m3-m2v-256 | Unigram | Python Viterbi | 250,002 × 256 | float32 | off | yes | 725 → 89 | 725 → 89 | 34,518 → 20,670 |
| potion-mxbai-micro | WordPiece | tokenizers | 2,000 × 256 | int8 +map +weights | on | yes | 90 → 72 | 91 → 72 | 68,385 → 70,998 |
| NIFE-mxbai-embed-large-v1_model2vec | WordPiece | tokenizers | 107,962 × 1024 | float32 | off | yes | 558 → 106 | 572 → 116 | 41,042 → 26,756 |
| ovos-m2v-intents-pt-PT-albertina-v5 | BPE | tokenizers | 50,262 × 256 | float32 | on | yes | 162 → 90 | 162 → 90 | 46,343 → 22,813 |
| F2LLM-v2-330M-model2vec | BPE | tokenizers | 151,644 × 512 | float16 | on | yes | 356 → 171 | 356 → 171 | 28,795 → 40,132 |
| static-retrieval-multilingual-69m-v1 | BPE | tokenizers | 179,936 × 384 | float32 | off | yes | 537 → 229 | 541 → 263 | 32,235 → 31,848 |

Unigram throughput ranges from about a third of model2vec’s rate to about the same. `tokenizers` parallelizes batches across cores. Python Viterbi runs on one core. These batch measurements do not establish latency for one query.

The word cache helps when the pre-tokenizer splits words, as in jina-v3, bge-m3 and multilingual-e5. Natural text runs 1.6 to 1.9 times faster on a first pass and 3 to 5 times faster on repeated text. The `bench` notebook records these measurements and the reasons for rejecting tokie and gigatoken.

Other models use the same `tokenizers` code in both loaders. The timing samples last about 0.1 seconds and vary by up to half on repeated runs. They do not establish a speed advantage.

## What is not supported

- Quantizing or truncating on load (`quantize_to`, `dimensionality`, `vocabulary_quantization`): each would copy the table, which is
  what laghu exists to avoid. Load with model2vec for those.
- `single_word` added tokens on a Unigram model fall back to `tokenizers` for tokenizing (the table is still mapped). Rust decides a
  word boundary by Unicode’s Alphabetic property, which Python’s `str.isalpha` does not match.
- `use_multiprocessing` is accepted and ignored; the result is the same.
- `encode([])` returns an empty `(0, dim)` array. model2vec raises.

## Developing

nbdev 3 on hatchling: the notebooks in `nbs/` are the source. `uv run nbdev-prepare` exports, tests and rebuilds this README;
`uv run nbdev-test --flags slow` also runs the bench, which downloads about 5 GB of models.

## Licence

MIT.
