Metadata-Version: 2.5
Name: laghu
Version: 0.4.0
Summary: model2vec static models in a fraction of the memory, with the same vectors
Project-URL: Repository, https://github.com/vedicreader/laghu
Project-URL: Documentation, https://vedicreader.github.io/laghu/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: embeddings,mmap,model2vec,nbdev,static-embeddings,tokenizers,unigram
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: fastcore>=1.8
Requires-Dist: huggingface-hub>=0.24
Requires-Dist: marisa-trie>=1.2
Requires-Dist: numpy>=1.24
Requires-Dist: tokenizers>=0.20
Provides-Extra: dev
Requires-Dist: ipykernel>=7.3.0; extra == 'dev'
Requires-Dist: model2vec<0.10,>=0.9; extra == 'dev'
Requires-Dist: nbdev>=3.3.14; extra == 'dev'
Requires-Dist: notebook>=7.6.3; extra == 'dev'
Provides-Extra: fallback
Requires-Dist: model2vec<0.10,>=0.9; extra == 'fallback'
Provides-Extra: rust
Requires-Dist: laghu-rs>=0.2.0; (python_version >= '3.11') and extra == 'rust'
Description-Content-Type: text/markdown

# laghu


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

[model2vec](https://github.com/MinishLab/model2vec) makes static embedding models. A static embedding model changes a text into a vector. It splits the text into tokens. It finds a stored vector for each token. Then it calculates the average of these vectors. No neural network runs, so these models are very fast. Each model has two main parts: an embedding table with one row for each token, and a tokenizer that splits text into tokens.

laghu (लघु, Sanskrit for “light”) loads the same models with much less memory. It gives the same vectors as model2vec, bit for bit. A model that laghu loads has the same `encode`, `encode_as_sequence` and `tokenize` methods as a model2vec model. The section [What is not supported](#what-is-not-supported) gives the one difference.

## Where the memory goes

A process is one running program. Private memory is memory that only one process can use. model2vec copies two large parts of the model into the private memory of each process. laghu shares these parts between processes.

| Part of potion-multilingual-128M | model2vec | laghu |
|----|----|----|
| Embedding table | 512 MB, copied into each process | Mapped from `model.safetensors` and shared |
| Tokenizer search structure for 500,353 pieces | About 610 MB in each process | Mapped from a vocabulary file and shared |
| Private memory after `encode` | About 1.3 GB | About 60 MB, mostly Python and its imports |

laghu maps the embedding table from the file. The operating system reads a page of the file only when laghu uses it. The operating system keeps the page in its cache. It shares the page with every process that maps the same file. When memory is low, the operating system can remove the page and read it again later.

For a Unigram model, laghu also replaces the search structure of the tokenizer. A Unigram tokenizer selects the most likely sequence of pieces from a long list. laghu builds a vocabulary file one time for each tokenizer.json. The file holds the pieces in a compact [marisa trie](https://github.com/pytries/marisa-trie), with their ids and scores. Every process maps this file and shares it. laghu then finds the best pieces with a Viterbi search. The search is written in Python. With the optional `laghu-rs` wheel, the search runs in Rust. The text normalization and the first split into words come from `tokenizers`. laghu builds them from the same tokenizer.json, without the vocabulary.

Models with a BPE or WordLevel tokenizer use `tokenizers` to tokenize. WordPiece models also use `tokenizers` when `laghu-rs` is not installed. laghu maps the table of these models too. The memory that you save depends on the size of the table compared with the size of the tokenizer.

## Install

``` sh
uv add laghu        # or: pip install laghu
```

model2vec is not a dependency. Install the `fallback` extra, `uv add 'laghu[fallback]'`, in these two cases:

- You want [`load`](https://vedicreader.github.io/laghu/encoder.html#load) to use model2vec when laghu cannot read a model.
- You want to use [`same_as_model2vec`](https://vedicreader.github.io/laghu/encoder.html#same_as_model2vec).

The extra installs model2vec 0.9, which laghu matches. model2vec 0.10 gives different vectors, and its `tokenize` does not take `max_length`.

The optional Rust core is the wheel `laghu-rs` (module `laghu_rs`). It needs Python 3.11 or later. Install it with `pip install laghu-rs`, or with the extra: `uv add 'laghu[rust]'`.

## The optional Rust core

With `laghu-rs` installed, laghu tokenizes these models in Rust, on all CPU cores:

- Unigram models. `laghu_rs` normalizes the text, finds the added tokens and splits the text into words with the `tokenizers` crate. Then it runs the Viterbi search of laghu over a byte trie in the shared vocabulary file.
- WordPiece models that have a BertNormalizer, a BertPreTokenizer and a WordPiece model. The English potion models are examples. `laghu_rs.WordPiece` is a port of `FastWordPiece` from [model2vec-rs](https://github.com/MinishLab/model2vec-rs). It calculates only the token ids. It does not calculate the offsets that `tokenizers` keeps.

For a float32 table without weights or a token mapping, `laghu_rs` also calculates the average vectors. On every path, the ids and vectors are the same as the ids and vectors of model2vec, bit for bit.

This table gives the speed and the private memory for 500-character chunks of text. The computer was an M3 Max with 14 cores. The measurements are from 2026-10-24, with a load average of 6.5 to 9. Each speed is the best of three runs of [`laghu.bench.speed`](https://vedicreader.github.io/laghu/bench.html#speed).

| Model | Loader | Chunks per second | Private memory |
|----|----|----|----|
| potion-multilingual-128M (Unigram) | laghu with `laghu-rs` | 42,385 | 72 MB |
|  | laghu without `laghu-rs` | 1,940 | 67 MB |
|  | model2vec 0.9 | 21,105 | 1,369 MB |
| potion-base-8M (WordPiece) | laghu with `laghu-rs` | 432,938 | 81 MB |
|  | laghu without `laghu-rs` | 55,514 | 93 MB |
|  | model2vec 0.9 | 55,600 | 140 MB |

laghu 0.3.0 gave 5,276 chunks per second for model2vec on potion-multilingual-128M. That number was measured with `tokenizers` on one core, probably because `TOKENIZERS_PARALLELISM=false` was set. On one core, model2vec gives 5,462 chunks per second on the same computer.

### When laghu does not use Rust

For a Unigram model, laghu tokenizes in pure Python when one of these conditions is true:

- laghu cannot import `laghu_rs`.
- `LAGHU_RUST=0` is set when the model loads.
- The model loads with `shared=False`.
- The tokenizer.json truncates or pads.

For a Unigram model, laghu splits the text in Python and does the search in Rust when one of these conditions is true:

- The Python `tokenizers` has a different major or minor version from `laghu_rs.TOKENIZERS`. A different patch version is accepted. `laghu-rs` 0.1.0 and 0.2.0 use `tokenizers` 0.23.2. They accept all Python `tokenizers` 0.23.x versions.
- The `tokenizers` crate cannot read the tokenizer.json.
- The crate splits a text in `PROBE` differently from the Python split of laghu. `PROBE` is a set of 46 difficult test texts. laghu logs a warning.

laghu tokenizes a WordPiece model with `tokenizers` when one of these conditions is true:

- laghu cannot import `laghu_rs`, or `laghu_rs` has no `WordPiece`.
- `LAGHU_RUST=0` is set when the model loads.
- The model loads with `lean=False`.
- The tokenizer.json truncates or pads.
- The Python `tokenizers` has a different major or minor version from `laghu_rs.TOKENIZERS`.
- The normalizer, pre-tokenizer and model are not BertNormalizer, BertPreTokenizer and WordPiece.
- The unknown token is not in the vocabulary.
- An added token is `normalized`, or has `lstrip`, `rstrip` or `single_word` set.
- `laghu_rs.WordPiece` gives different ids from `tokenizers` on a text in `PROBE`. laghu logs a warning.

laghu does not use `laghu-rs` for BPE and WordLevel models. It does not use `laghu-rs` for Unigram models with a feature that laghu does not copy.

The ids are the same on every path.

## Use

``` python
from laghu import load

m = load('minishlab/potion-multilingual-128M')      # a folder, or a Hugging Face repo id (the local cache is tried first)
v = m.encode(['The quick brown fox.', 'कर्मण्येवाधिकारस्ते'])   # float32, (2, 256), unit rows: the same array model2vec returns
m.tokenize(['hello world'])
m.encode_as_sequence('hello world').shape          # (n_tokens, 256)
```

    (2, 256)

[`load`](https://vedicreader.github.io/laghu/encoder.html#load) takes a local folder or a Hugging Face repo id. It returns a [`LeanModel`](https://vedicreader.github.io/laghu/encoder.html#leanmodel). If the repo is in the local Hugging Face cache, [`load`](https://vedicreader.github.io/laghu/encoder.html#load) does not use the network.

If laghu cannot load the model, [`load`](https://vedicreader.github.io/laghu/encoder.html#load) logs the reason. Then it tries model2vec’s `StaticModel`, which needs `laghu[fallback]`. If model2vec also fails, [`load`](https://vedicreader.github.io/laghu/encoder.html#load) raises the first error from laghu. Use `fallback=False` to get the error without a try with model2vec.

laghu and model2vec cannot read tables stored as bfloat16 or float8, because numpy does not have these types.

laghu keeps the vocabulary file of a Unigram model in `$LAGHU_CACHE`. If `$LAGHU_CACHE` is not set, laghu uses `$XDG_CACHE_HOME/laghu`. If that is not set, laghu uses `~/.cache/laghu`. If laghu cannot write to the folder, it puts the file next to tokenizer.json.

| Vocabulary file | Time |
|----|----|
| First load, which builds the file | 1 to 2 seconds |
| Later loads, in any process, which map the file | A few hundredths of a second |

If tokenizer.json changes, or if the file is damaged, laghu builds the file again. `load(..., shared=False)` keeps the vocabulary in a Python dict. This uses about 80 MB more private memory in each process.

To check your own model, use `same_as_model2vec(path)`. It loads the model with laghu and with model2vec, and it compares the results. It uses the built-in test texts of laghu, or the texts that you give:

``` python
from laghu import same_as_model2vec
same_as_model2vec('minishlab/potion-base-8M', ['my', 'own texts'])
```

    /Users/71293/code/personal/orgs/laghu/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
      from .autonotebook import tqdm as notebook_tqdm

    {'lean': False,
     'n': 2,
     'ids': True,
     'dtype': True,
     'f32': True,
     'f16': True,
     'exact': True,
     'seq': True,
     'median': True,
     'unk': True,
     'tokens': True}

The result shows, for each check, if the two loaders agree:

| Key      | What it checks                                     |
|----------|----------------------------------------------------|
| `ids`    | The token ids                                      |
| `f32`    | The vectors, changed to float32                    |
| `f16`    | The vectors, changed to float16                    |
| `dtype`  | The exact data type                                |
| `exact`  | The exact bits                                     |
| `seq`    | `encode_as_sequence`                               |
| `median` | `median_token_length`                              |
| `unk`    | The unknown token id                               |
| `tokens` | The token list                                     |
| `lean`   | laghu used its own Unigram tokenizer (not a check) |
| `n`      | The number of texts (not a check)                  |

## What matches, and what it costs

Every model in the table below gives the same ids and vectors as model2vec, bit for bit, on 4,726 texts. The texts include prose in 36 languages and random strings from 30 Unicode blocks. They also include control characters, special tokens, emoji sequences, combining marks, bidirectional text controls and a string of 200,000 characters.

These rules apply to the table:

- Each memory number comes from a new process. On macOS, private memory is `phys_footprint` after `encode`, and peak memory is `lifetime_max_phys_footprint`. On Linux, the same columns would hold `RssAnon` and `VmHWM`.
- Speed is `encode` over 3,226 texts on an M3 Max.
- “laghu tokenizes with” gives the tokenizer that laghu used: its Viterbi search in Python, its Viterbi search in Rust with `laghu-rs`, or `tokenizers`.
- In the private memory and speed columns, the first two numbers were measured without `laghu-rs`. The third number was measured with `laghu-rs`, in the same way.
- `—` means that `laghu-rs` does not change the row. This is true for BPE and WordLevel models. It is also true for potion-code-16M-v2, because its tokenizer.json truncates and pads.
- “not measured” means that nobody measured the row with `laghu-rs`. These rows were checked without `laghu-rs`. With `laghu-rs`, they use Rust when their tokenizer qualifies.
- potion-base-8M and potion-code-16M-v2 were checked again with `laghu-rs` on 2026-10-24. Both stay identical.

The `laghu-rs` numbers come from different sessions than the older numbers. Compare each `laghu-rs` number with the numbers from its own session:

| Model | Date | model2vec, texts per second | laghu without `laghu-rs`, texts per second | laghu with `laghu-rs`, texts per second | Notes |
|----|----|----|----|----|----|
| potion-multilingual-128M | 2026-10-23 | 22,846 | 14,859 | 130,007 | Best of 3 runs. Load average 6.5. The slowest Rust run gave 41,988 texts per second. |
| potion-base-8M | 2026-10-24 | 82,027 | not measured | 650,395 | Best of 3 runs. Each Rust run took about 0.003 seconds. |

| model | tokenizer | laghu tokenizes with | vocab × dims | dtype | norm | identical | private MB, model2vec → laghu → laghu-rs | peak MB, model2vec → laghu | texts/s, model2vec → laghu → laghu-rs |
|----|----|----|----|----|----|----|----|----|----|
| potion-base-2M | WordPiece | tokenizers | 29,528 × 64 | float32 | on | yes | 81 → 58 → not measured | 82 → 58 | 84,510 → 92,982 → not measured |
| potion-base-4M | WordPiece | tokenizers | 29,528 × 128 | float32 | on | yes | 94 → 62 → not measured | 95 → 62 | 87,248 → 90,242 → not measured |
| potion-base-8M | WordPiece | tokenizers, or Rust with laghu-rs | 29,528 × 256 | float32 | on | yes | 114 → 70 → 71 | 115 → 70 | 79,415 → 81,810 → 650,395 |
| potion-base-32M | WordPiece | tokenizers | 63,091 × 512 | float32 | on | yes | 244 → 97 → not measured | 244 → 97 | 75,009 → 81,379 → not measured |
| potion-retrieval-32M | WordPiece | tokenizers | 63,091 × 512 | float32 | on | yes | 244 → 97 → not measured | 244 → 97 | 70,799 → 79,717 → not measured |
| potion-science-8M | WordPiece | tokenizers | 29,528 × 256 | float32 | on | yes | 113 → 70 → not measured | 114 → 70 | 81,008 → 84,532 → not measured |
| potion-code-16M | WordPiece | tokenizers | 61,826 × 256 | float32 +map +weights | on | yes | 147 → 110 → not measured | 211 → 117 | 10,674 → 10,425 → not measured |
| potion-code-16M-v2 | WordPiece | tokenizers | 63,457 × 256 | float16 | on | yes | 108 → 101 → — | 186 → 119 | 21,819 → 22,164 → — |
| potion-multilingual-128M | Unigram | Python Viterbi, or Rust with laghu-rs | 500,353 × 256 | float32 | on | yes | 1,348 → 61 → 59 | 1,348 → 61 | 39,105 → 10,902 → 130,007 |
| M2V_base_output | WordPiece | tokenizers | 29,528 × 256 | float32 | off | yes | 112 → 69 → not measured | 113 → 69 | 65,460 → 87,100 → not measured |
| M2V_multilingual_output | WordPiece | tokenizers | 501,054 × 256 | float32 | off | yes | 834 → 198 → not measured | 842 → 295 | 42,565 → 90,283 → not measured |
| M2V_base_glove | WordLevel | tokenizers | 400,002 × 256 | float32 | off | yes | 656 → 186 → — | 656 → 194 | 67,028 → 85,058 → — |
| M2V_base_glove_subword | WordPiece | tokenizers | 400,779 × 256 | float32 | off | yes | 647 → 178 → not measured | 649 → 195 | 66,256 → 85,430 → not measured |
| jina-embeddings-v3-separation-distilled | Unigram | Python Viterbi, or Rust with laghu-rs | 250,002 × 256 | float32 | off | yes | 722 → 88 → not measured | 722 → 88 | 51,836 → 20,840 → not measured |
| multilingual-e5-large-m2v | Unigram | Python Viterbi, or Rust with laghu-rs | 250,002 × 512 | float32 | on | yes | 981 → 109 → not measured | 981 → 109 | 24,139 → 19,670 → not measured |
| m2v-gemma-embedding-300m | Unigram | Python Viterbi, or Rust with laghu-rs | 255,732 × 256 | float16 +weights | on | yes | 542 → 93 → not measured | 542 → 93 | 20,319 → 12,134 → not measured |
| potion-multilingual-128M-i8-tokenade | Unigram | Python Viterbi, or Rust with laghu-rs | 500,353 × 256 | int8 | on | yes | 985 → 138 → not measured | 985 → 138 | 27,676 → 12,090 → not measured |
| m2v-qwen3-embedding-0.6b-1024d | Unigram | Python Viterbi, or Rust with laghu-rs | 151,644 × 1024 | float16 +weights | on | yes | 602 → 132 → not measured | 602 → 132 | 16,792 → 7,168 → not measured |
| ovos-m2v-intents-es-ES-bne-v5 | Unigram | Python Viterbi, or Rust with laghu-rs | 50,259 × 64 | float32 | on | yes | 169 → 52 → not measured | 169 → 52 | 11,786 → 11,265 → not measured |
| m2v-gte-256-edu | Unigram | tokenizers | 313,845 × 256 | float32 | on | yes | 1,028 → 663 → — | 1,028 → 675 | 25,363 → 24,981 → — |
| bge-m3-m2v-256 | Unigram | Python Viterbi, or Rust with laghu-rs | 250,002 × 256 | float32 | off | yes | 725 → 89 → not measured | 725 → 89 | 34,518 → 20,670 → not measured |
| potion-mxbai-micro | WordPiece | tokenizers | 2,000 × 256 | int8 +map +weights | on | yes | 90 → 72 → not measured | 91 → 72 | 68,385 → 70,998 → not measured |
| NIFE-mxbai-embed-large-v1_model2vec | WordPiece | tokenizers | 107,962 × 1024 | float32 | off | yes | 558 → 106 → not measured | 572 → 116 | 41,042 → 26,756 → not measured |
| ovos-m2v-intents-pt-PT-albertina-v5 | BPE | tokenizers | 50,262 × 256 | float32 | on | yes | 162 → 90 → — | 162 → 90 | 46,343 → 22,813 → — |
| F2LLM-v2-330M-model2vec | BPE | tokenizers | 151,644 × 512 | float16 | on | yes | 356 → 171 → — | 356 → 171 | 28,795 → 40,132 → — |
| static-retrieval-multilingual-69m-v1 | BPE | tokenizers | 179,936 × 384 | float32 | off | yes | 537 → 229 → — | 541 → 263 | 32,235 → 31,848 → — |

`+map` marks a vocabulary-quantized model. This model stores fewer rows than it has tokens, and a map from each token to its row. `+weights` marks a model that multiplies the vector of each token by a stored weight.

The potion-multilingual-128M row was measured again on 2026-10-13 with the shared vocabulary file. The other Unigram rows were measured with the vocabulary in a dict. For potion-multilingual-128M, the dict used about 80 MB more private memory in each process.

Without `laghu-rs`, laghu is slower than model2vec on Unigram models. Its speed is between about one third of the speed of model2vec and about the same speed. The reason is that `tokenizers` uses all cores for a batch, but the Python search uses one core. With `laghu-rs`, laghu also uses all cores. On potion-multilingual-128M, laghu with `laghu-rs` is about two times as fast as model2vec. The table in [The optional Rust core](#the-optional-rust-core) gives the numbers. The other Unigram models were not measured with `laghu-rs`. All these numbers are for batches of texts. They do not show the time to encode one query.

laghu keeps a cache of words that it segmented before. The cache helps when the tokenizer first splits the text into words. The jina-v3, bge-m3 and multilingual-e5 tokenizers do this.

| Text                     | Speed with the word cache |
|--------------------------|---------------------------|
| Natural text, first pass | 1.6 to 1.9 times faster   |
| The same text again      | 3 to 5 times faster       |

The [bench](https://vedicreader.github.io/laghu/bench.html) page has these measurements. It also tells why laghu does not use two faster Rust tokenizers, tokie and gigatoken.

Without `laghu-rs`, laghu and model2vec tokenize the other models with the same `tokenizers` code. Each speed sample for these models lasts about 0.1 seconds. The samples change by up to half from one run to the next. Thus, the speed differences in these rows have no meaning.

## What is not supported

- laghu does not quantize or truncate a model when it loads the model. These are the model2vec options `quantize_to`, `dimensionality` and `vocabulary_quantization`. Each option makes a new copy of the table, and laghu exists to prevent that copy. Use model2vec if you need these options.
- laghu tokenizes a Unigram model with `single_word` added tokens with `tokenizers`. It still maps the table. The reason is that `tokenizers` finds the end of a word with the Unicode Alphabetic property, and Python’s `str.isalpha` does not match it.
- laghu accepts `use_multiprocessing` and ignores it. The result is the same.
- `encode([])` returns an empty array of shape `(0, dim)`. model2vec raises an error.

## Developing

laghu uses nbdev 3 with hatchling. The notebooks in `nbs/` are the source code. `uv run nbdev-prepare` exports the package, runs the tests and builds this README again. `uv run nbdev-test --flags slow` also runs the benchmark. The benchmark downloads about 5 GB of models.

## Licence

MIT.
