Metadata-Version: 2.2
Name: rabitqlib
Version: 0.5.1
Summary: RaBitQ Python bindings for indexes and clustering
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Project-URL: Homepage, https://vectordb-ntu.github.io/RaBitQ-Library/
Project-URL: Documentation, https://vectordb-ntu.github.io/RaBitQ-Library/
Project-URL: Repository, https://github.com/VectorDB-NTU/RaBitQ-Library
Project-URL: Issues, https://github.com/VectorDB-NTU/RaBitQ-Library/issues
Project-URL: Paper, https://doi.org/10.1145/3725413
Requires-Python: >=3.11
Requires-Dist: numpy
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Requires-Dist: numpy; extra == "test"
Description-Content-Type: text/markdown

<div align="center">

<h1>RaBitQ Library</h1>

<h3>Compact vectors. Accurate distances. Fast ANN search.</h3>

<p>
  A research-backed C++17 library with Python bindings for 1-bit and multi-bit<br>
  vector quantization, IVF, HNSW, SymphonyQG, and clustering.
</p>

<p>
  <a href="https://pypi.org/project/rabitqlib/"><img alt="PyPI" src="https://img.shields.io/pypi/v/rabitqlib.svg?cacheSeconds=300"></a>
  <a href="https://pypi.org/project/rabitqlib/"><img alt="Python versions" src="https://img.shields.io/badge/python-3.11--3.14-3776AB.svg?logo=python&amp;logoColor=white"></a>
  <a href="https://vectordb-ntu.github.io/RaBitQ-Library/"><img alt="Documentation" src="https://github.com/VectorDB-NTU/RaBitQ-Library/actions/workflows/docs.yml/badge.svg"></a>
  <a href="https://doi.org/10.1145/3725413"><img alt="Paper DOI" src="https://img.shields.io/badge/DOI-10.1145%2F3725413-blue"></a>
  <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-Apache--2.0-blue.svg"></a>
</p>

<p>
  <a href="https://vectordb-ntu.github.io/RaBitQ-Library/">Documentation</a> ·
  <a href="https://pypi.org/project/rabitqlib/">Python package</a> ·
  <a href="https://doi.org/10.1145/3725413">Paper</a> ·
  <a href="https://github.com/VectorDB-NTU/RaBitQ-Library/releases">Releases</a> ·
  <a href="ROADMAP.md">Maintenance</a>
</p>

</div>

> **Contributors welcome!** Help shape RaBitQ by reporting bugs, asking questions,
> suggesting features, or contributing code, tests, documentation, and examples.
> First-time contributors are welcome—[open an issue](https://github.com/VectorDB-NTU/RaBitQ-Library/issues/new/choose)
> or start with our [contribution guide](CONTRIBUTING.md#your-first-contribution)
> and [starter tasks](CONTRIBUTING.md#starter-tasks).

## News

- **September 2026 — IVF updates (0.5.1):** Add vectors to a built or loaded IVF
  index with `add()` and exclude vectors from search with `remove()`. See the [IVF update guide](docs/docs/index/ivf.md#updating-an-index)
  for costs and limits.

- **September 2026 — Native clustering (0.5.0):** Train k-means directly in C++
  or Python with RaBitQKMeans and QGKMeans, with approximate or exact final
  assignment. See the [clustering guide](docs/docs/clustering.md).

- **September 2026 — Platform support:** CPython 3.11–3.14 wheels cover Linux
  x86-64 and ARM64, Windows x86-64, and macOS 14+ ARM64. C++ source builds are
  validated on these platforms. See the
  [platform requirements](docs/docs/quick_start.md#requirements).

- **September 2026 — txtai integration:** [txtai](https://github.com/neuml/txtai)
  now includes `rabitqlib` as an ANN backend with IVF and HNSW modes. See its
  [RaBitQ configuration](https://github.com/neuml/txtai/blob/master/docs/embeddings/configuration/ann.md#rabitq)
  and the [integration discussion](https://github.com/VectorDB-NTU/RaBitQ-Library/issues/110).

## Install

```bash
python -m pip install --upgrade "rabitqlib>=0.5.1"
```

The build/search examples below require 0.5.0 or newer; IVF `add()`/`remove()`
require 0.5.1 or newer. For unreleased changes,
[install from a checkout](CONTRIBUTING.md#python-changes).

Wheels: CPython 3.11–3.14 on Linux x86-64 and ARM64, Windows x86-64, and
macOS 14+ ARM64 (Apple Silicon). x86-64 uses AVX2/FMA with optional AVX-512
acceleration; ARM64 uses NEON and portable scalar kernels. Linux ARM64 and
macOS wheels bundle OpenMP. Linux ARM64 wheels carry
`manylinux_2_27_aarch64` and `manylinux_2_28_aarch64` tags.

## Python quick start

Build and search a small IVF index using synthetic data:

```python
import numpy as np
from rabitqlib import FinalAssignmentMode, IvfIndex, RaBitQKMeans

rng = np.random.default_rng(42)
data = rng.standard_normal((500, 64)).astype(np.float32)
queries = rng.standard_normal((5, 64)).astype(np.float32)

clustering = RaBitQKMeans(
    64, 5, num_threads=2, final_assignment=FinalAssignmentMode.Exact
)
clustering.train(data)

index = IvfIndex(
    dim=64,
    max_elements=len(data),
    num_clusters=5,
    nbits=4,
    metric="l2",
)
index.build(data, clustering.centroids, clustering.assignments)

ids, distances = index.search(queries, k=10, nprobe=5)
print(ids.shape, distances.shape)  # (5, 10) (5, 10)
print(ids[0])
```

For all three indexes, `build` and `search` interpret `num_threads=0` as the
detected hardware thread count. Larger requests are capped at that count;
smaller positive requests are respected. Operations may use fewer workers when
there are fewer work items. If hardware detection is unavailable, one thread is
used. Python index methods default to one thread when `num_threads` is omitted.

IVF and HNSW examples save clusters for reuse across index configurations.
Choose RaBitQKMeans for flat assignment or QGKMeans for graph assignment;
FAISS is needed only for the optional clustering comparison.

Index save/load paths are UTF-8 strings on Windows and native path bytes on POSIX
in C++; Python paths are Unicode strings on all platforms.

See the [Python examples](sample/python/README.md) for IVF, HNSW, and SymphonyQG.
For clustering, use [RaBitQKMeans](docs/docs/clustering.md#rabitqkmeans) for small cluster
counts or [QGKMeans](docs/docs/clustering.md#qgkmeans) for graph assignment. The
[FAISS comparison](sample/python/compare_with_faiss.py) benchmarks both methods.

<details>
<summary>Build the Python bindings from source</summary>

Source builds require a C++17 compiler, CMake 3.20 or newer, and OpenMP. On
Windows, install Visual Studio 2026 with the Desktop development with C++
workload, then run `python -m pip install .` from the repository root.
On Ubuntu or Debian:

```bash
sudo apt-get update
sudo apt-get install -y build-essential cmake libomp-dev
git clone https://github.com/VectorDB-NTU/RaBitQ-Library.git
cd RaBitQ-Library
python -m pip install .
```

</details>

## Choose the right building block

| Component | Best fit | Storage and search profile |
| --- | --- | --- |
| **Quantizer** | Integrating RaBitQ into an existing system | Low-level 1-bit or multi-bit encoding and distance estimation. |
| **IVF** | Memory-efficient partitioned search | Stores quantized codes, or one-bit codes plus raw vectors for reranking. |
| **HNSW** | Graph search with compact vectors | Adds graph links and searches directly from quantized codes. |
| **SymphonyQG** | Fast graph search with a configurable memory/accuracy tradeoff | Uses raw vectors by default, or optional packed 4-bit/8-bit RaBitQ vectors, alongside per-neighborhood quantization data. |

IVF and SymphonyQG use [FastScan](https://arxiv.org/abs/1704.07355) for batched
estimates, while HNSW uses single-code kernels selected for the target architecture.

In typical workloads, 4-bit, 5-bit, and 7-bit quantization can achieve roughly
90%, 95%, and 99% recall, respectively, without reranking. Actual results
depend on the dataset, index configuration, and search parameters.

## Why RaBitQ?

| | |
| --- | --- |
| **Compact by design** | Choose [1-bit](https://doi.org/10.1145/3654970) or [multi-bit](https://doi.org/10.1145/3725413) codes to match your memory and accuracy target. |
| **Accurate estimates** | An asymptotically optimal theoretical error bound supports reliable ordering and reranking. |
| **Native CPU backends** | Runtime AVX2/AVX-512 selection on x86-64; NEON distance, packed-code, FastScan, rotation, query preparation, and HNSW search kernels on ARM64, with portable scalar fallbacks. |
| **Ready for ANN search** | Use the quantizer directly or build complete IVF, HNSW, and [SymphonyQG](https://dl.acm.org/doi/abs/10.1145/3709730) indexes. |
| **Native clustering** | Choose [RaBitQKMeans](docs/docs/clustering.md#rabitqkmeans) with flat RaBitQ assignment or [QGKMeans](docs/docs/clustering.md#qgkmeans) with SymphonyQG assignment; neither needs an external k-means package. |

The library supports Euclidean distance and inner product. Cosine search is
available by normalizing vectors before using inner product.

RaBitQ is developed by the
[VectorDB group](https://vectordb-ntu.github.io/) at Nanyang Technological
University, Singapore. A GPU implementation is also available in
[cuvs_rabitq](https://github.com/Stardust-SJF/cuvs_rabitq/tree/cuvs_ivf_rabitq).

## RaBitQ across the vector-search ecosystem

The projects below illustrate adoption of RaBitQ techniques across vector
search; this is not a list of direct dependencies on RaBitQ-Library.

**Integration story:** [How zvec integrates RaBitQ-Library](docs/docs/integrations/zvec.md)
traces its use of the library's quantizers and estimators inside zvec's IVF
and HNSW implementations, with links to the source code.

<table>
  <tr>
    <td align="center" width="20%">
      <a href="https://github.com/milvus-io/milvus"><img src="https://github.com/milvus-io.png?size=96" width="64" height="64" alt="Milvus logo"><br><strong>Milvus</strong></a>
    </td>
    <td align="center" width="20%">
      <a href="https://github.com/facebookresearch/faiss"><img src="https://github.com/facebookresearch.png?size=96" width="64" height="64" alt="Faiss logo"><br><strong>Faiss</strong></a>
    </td>
    <td align="center" width="20%">
      <a href="https://github.com/NVIDIA/cuvs"><img src="https://github.com/NVIDIA.png?size=96" width="64" height="64" alt="NVIDIA cuVS logo"><br><strong>NVIDIA cuVS</strong></a>
    </td>
    <td align="center" width="20%">
      <a href="https://github.com/microsoft/DiskANN/blob/main/diskann-quantization/src/lib.rs"><img src="https://github.com/microsoft.png?size=96" width="64" height="64" alt="Microsoft DiskANN logo"><br><strong>Microsoft DiskANN</strong></a>
    </td>
    <td align="center" width="20%">
      <a href="https://github.com/antgroup/vsag"><img src="https://github.com/antgroup.png?size=96" width="64" height="64" alt="VSAG logo"><br><strong>VSAG</strong></a>
    </td>
  </tr>
  <tr>
    <td align="center" width="20%">
      <a href="https://github.com/tensorchord/VectorChord"><img src="https://github.com/tensorchord.png?size=96" width="64" height="64" alt="VectorChord logo"><br><strong>VectorChord</strong></a>
    </td>
    <td align="center" width="20%">
      <a href="https://www.volcengine.com/docs/6465/1553583"><img src="https://github.com/volcengine.png?size=96" width="64" height="64" alt="Volcengine OpenSearch logo"><br><strong>Volcengine OpenSearch</strong></a>
    </td>
    <td align="center" width="20%">
      <a href="https://github.com/cockroachdb/cockroach"><img src="https://github.com/cockroachdb.png?size=96" width="64" height="64" alt="CockroachDB logo"><br><strong>CockroachDB</strong></a>
    </td>
    <td align="center" width="20%">
      <a href="https://github.com/elastic/elasticsearch"><img src="https://github.com/elastic.png?size=96" width="64" height="64" alt="Elasticsearch logo"><br><strong>Elasticsearch</strong></a>
    </td>
    <td align="center" width="20%">
      <a href="https://github.com/apache/lucene"><img src="https://github.com/apache.png?size=96" width="64" height="64" alt="Apache Lucene logo"><br><strong>Apache Lucene</strong></a>
    </td>
  </tr>
  <tr>
    <td align="center" width="20%">
      <a href="https://turbopuffer.com/blog/ann-v3#:~:text=ANN%20v3%20employs%20the%20RaBitQ"><img src="https://github.com/turbopuffer.png?size=96" width="64" height="64" alt="turbopuffer logo"><br><strong>turbopuffer</strong></a>
    </td>
    <td align="center" width="20%">
      <a href="https://github.com/alibaba/zvec"><img src="https://github.com/alibaba.png?size=96" width="64" height="64" alt="Zvec logo"><br><strong>Zvec</strong></a>
    </td>
    <td align="center" width="20%">
      <a href="https://docs.lancedb.com/indexing/quantization#rabitq-quantization"><img src="https://github.com/lancedb.png?size=96" width="64" height="64" alt="LanceDB logo"><br><strong>LanceDB</strong></a>
    </td>
    <td align="center" width="20%">
      <a href="https://docs.databricks.com/aws/en/oltp/projects/lakebase-vector"><img src="https://github.com/databricks.png?size=96" width="64" height="64" alt="Databricks logo"><br><strong>Databricks</strong></a>
    </td>
    <td align="center" width="20%">
      <a href="https://clickhouse.com/docs/engines/table-engines/mergetree-family/annindexes#quantized-codecs-methods"><img src="https://github.com/ClickHouse.png?size=96" width="64" height="64" alt="ClickHouse logo"><br><strong>ClickHouse</strong></a>
    </td>
  </tr>
  <tr>
    <td align="center" width="20%">
      <a href="https://qdrant.tech/articles/turboquant-quantization/#1-bit-rabitq-bit-plane-scoring"><img src="https://github.com/qdrant.png?size=96" width="64" height="64" alt="Qdrant logo"><br><strong>Qdrant</strong></a>
    </td>
    <td align="center" width="20%">
      <a href="https://docs.weaviate.io/weaviate/concepts/vector-quantization#rotational-quantization"><img src="https://github.com/weaviate.png?size=96" width="64" height="64" alt="Weaviate logo"><br><strong>Weaviate</strong></a>
    </td>
  </tr>
</table>

## C++ quick start

### Requirements

- CMake 3.20 or newer
- a C++17 compiler with OpenMP support
- an x86-64 CPU with AVX2 and FMA, or an ARM64 CPU on Linux or macOS

Clone and build the library and example programs:

```bash
git clone https://github.com/VectorDB-NTU/RaBitQ-Library.git
cd RaBitQ-Library

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel
```

For MSVC, follow the [Windows build instructions](tests/README.md#prerequisites).
For ARM64 source builds, see the [Linux ARM64](tests/README.md#linux-arm64)
and [macOS ARM64](tests/README.md#macos-arm64) instructions.
Local GCC/Clang builds enable `-march=native` by default; set
`-DRABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF` for portable binaries, as release
wheels do. See [CPU dispatch details](DEVELOPMENT.md#dispatch-conventions-and-coverage)
for backend requirements and fallbacks.

### Use RaBitQ-Library in another C++ project

The C++ API and ABI are still evolving. For reproducible builds, pin a release
or commit and include RaBitQ-Library as a Git submodule:

```bash
git submodule add https://github.com/VectorDB-NTU/RaBitQ-Library.git third_party/rabitqlib
git submodule update --init --recursive
```

Add the library and link its namespaced target in the consuming project's
`CMakeLists.txt`:

```cmake
set(RABITQ_BUILD_SAMPLES OFF CACHE BOOL "" FORCE)
add_subdirectory(third_party/rabitqlib)

target_link_libraries(my_program PRIVATE rabitqlib::rabitqlib)
```

Update the pinned revision deliberately when you are ready to adopt upstream
changes:

```bash
git -C third_party/rabitqlib fetch
git -C third_party/rabitqlib checkout <release-or-commit>
git add third_party/rabitqlib
```

<details>
<summary>Optional: install the C++ library</summary>

Installation is useful for package managers, container images, and shared
server environments. Disable native optimization when the installed library
may run on a different CPU from the build machine:

```bash
cmake -S . -B build \
  -DRABITQ_BUILD_SAMPLES=OFF \
  -DRABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX="$HOME/.local"
cmake --build build --parallel
cmake --install build
```

Consume the installed package with:

```cmake
find_package(rabitqlib CONFIG REQUIRED)
target_link_libraries(my_program PRIVATE rabitqlib::rabitqlib)
```

For a non-system prefix, point CMake to the installation when configuring the
consumer:

```bash
cmake -S . -B build -DCMAKE_PREFIX_PATH="$HOME/.local"
cmake --build build --parallel
```

The [downstream consumer test](tests/consumer/) provides a minimal complete
example of the installed-package workflow.

</details>

Both integration methods require OpenMP on the consuming system.

The index example executables are written to `bin/`. Their source code shows
the complete indexing and querying workflows:

- [IVF + RaBitQ](sample/cpp/ivf_rabitq_indexing.cpp)
- [HNSW + RaBitQ](sample/cpp/hnsw_rabitq_indexing.cpp)
- [SymphonyQG](sample/cpp/symqg_indexing.cpp)

A separate [RaBitQ quantization example](sample/cpp/quantizer.cpp) demonstrates
the lower-level quantizer API; it is provided as source and is not currently a
CMake target.

To build and run the C++ test suite:

```bash
cmake -S . -B build -DRABITQ_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel
ctest --test-dir build --output-on-failure
```

GoogleTest is downloaded during test configuration. For a full benchmark on
the GIST dataset, see [`example.sh`](example.sh). More detailed API and
algorithm guidance is available in the [documentation](docs/docs/index.md).

## Citation

If RaBitQ helps your research or system, please cite:

> Jianyang Gao, Yutong Gou, Yuexuan Xu, Yongyi Yang, Cheng Long, and Raymond
> Chi-Wing Wong. “Practical and Asymptotically Optimal Quantization of
> High-Dimensional Vectors in Euclidean Space for Approximate Nearest Neighbor
> Search.” *Proceedings of the ACM on Management of Data* 3, 3, Article 202
> (June 2025), 26 pages. [https://doi.org/10.1145/3725413](https://doi.org/10.1145/3725413).

> Yutong Gou, Jianyang Gao, Yuexuan Xu, and Cheng Long. “SymphonyQG: Towards
> Symphonious Integration of Quantization and Graph for Approximate Nearest
> Neighbor Search.” *Proceedings of the ACM on Management of Data* 3, 1,
> Article 80 (February 2025), 26 pages.
> [https://doi.org/10.1145/3709730](https://doi.org/10.1145/3709730).

> Jianyang Gao and Cheng Long. “RaBitQ: Quantizing High-Dimensional Vectors
> with a Theoretical Error Bound for Approximate Nearest Neighbor Search.”
> *Proceedings of the ACM on Management of Data* 2, 3, Article 167 (May 2024),
> 27 pages. [https://doi.org/10.1145/3654970](https://doi.org/10.1145/3654970).

## Contributing

Contributions are welcome, including documentation and examples. Start with
[your first contribution](CONTRIBUTING.md#your-first-contribution) or choose a
[small starter task](CONTRIBUTING.md#starter-tasks). The guide explains which
build, test, and formatting checks apply to your change.

See [maintenance and feedback](ROADMAP.md) for the current maintainer. Use
[GitHub Issues](https://github.com/VectorDB-NTU/RaBitQ-Library/issues/new/choose)
for bugs, feature requests, and usage or contribution questions.

## Acknowledgements

RaBitQ Library is developed by Yutong Gou, Jianyang Gao, Yuexuan Xu, Jifan Shi,
and Zhonghao Yang. We thank Alexandr Guzhva, Li Liu, Chao Gao, Silu Huang,
Jiabao Jin, Xiaoyao Zhong, and Jinjing Zhou for their valuable feedback.

## License

RaBitQ Library is available under the [Apache License 2.0](LICENSE).
