Metadata-Version: 2.4
Name: bertalign
Version: 2.0.0
Summary: Automatic multilingual sentence aligner based on LaBSE sentence embeddings.
Keywords: alignment,bitext,parallel-corpus,sentence-alignment,labse
Author: Jason
Author-email: Jason <bfsujason@163.com>
License-Expression: GPL-3.0-or-later
License-File: LICENCE
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: Linguistic
Requires-Dist: lingua-language-detector>=2.2.0
Requires-Dist: numba>=0.61.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: sentence-splitter>=1.4
Requires-Dist: sentence-transformers>=3.0.0
Requires-Dist: torch>=2.2.0
Requires-Dist: torchvision>=0.23.0 ; extra == 'sat'
Requires-Dist: wtpsplit>=2.2.1 ; extra == 'sat'
Maintainer: chikingsley
Maintainer-email: chikingsley <cheez2012@gmail.com>
Requires-Python: >=3.12
Project-URL: Homepage, https://github.com/chikingsley/bertalign
Project-URL: Repository, https://github.com/chikingsley/bertalign
Project-URL: Upstream, https://github.com/bfsujason/bertalign
Provides-Extra: sat
Description-Content-Type: text/markdown

# Bertalign

An automatic multilingual sentence aligner. Bertalign takes two texts that are
translations of each other and works out which sentences correspond — including
1-to-many, many-to-1, and many-to-many mappings — producing aligned sentence
pairs for translation study, corpus construction, or training data mining.

It implements the two-pass algorithm of
[Liu & Zhu (2022)](https://doi.org/10.1093/llc/fqac089): sentences (and their
overlapping concatenations) are embedded with
[LaBSE](https://huggingface.co/sentence-transformers/LaBSE), a first dynamic
programming pass finds approximate 1-1 anchors via top-k similarity search, and
a second pass finds the optimal m-n alignment along that path.

This is a modernized fork of
[bfsujason/bertalign](https://github.com/bfsujason/bertalign): packaged for
PyPI, fully typed, GPU-accelerated end to end, with offline language detection
and no network calls at alignment time.

## Install

```bash
uv add bertalign        # or: pip install bertalign
uv add "bertalign[sat]" # optional: neural sentence splitting (see below)
```

Python 3.12+. A CUDA GPU is used automatically when available, for both
embedding and similarity search; everything also runs on CPU.

## Usage

```python
from bertalign import Bertalign

aligner = Bertalign(src_text, tgt_text)
aligner.align_sents()
for src_segment, tgt_segment in aligner.sent_pairs():
    print(src_segment, "|||", tgt_segment)
```

Languages are detected automatically (offline, via
[lingua](https://github.com/pemistahl/lingua-rs)). If your texts already hold
one sentence per line, pass `is_split=True` to skip splitting.

Aligning many document pairs reuses one loaded model automatically; to be
explicit, share an encoder:

```python
from bertalign import Bertalign, Encoder

encoder = Encoder("LaBSE")
for src_text, tgt_text in documents:
    aligner = Bertalign(src_text, tgt_text, model=encoder)
    aligner.align_sents()
```

### Command line

```bash
bertalign source.txt target.txt > pairs.tsv
```

Inputs are plain-text files; output is one tab-separated pair per line
(internal whitespace is normalized during cleaning, so fields never contain
tabs). Options: `--model` (any sentence-transformers model), `--max-align`
(largest bead size), `--is-split`, `--sat`, `--trust-remote-code`,
`--version`.

## Choosing a model

Any [sentence-transformers](https://sbert.net) model works via
`model=`/`--model`. On the text+berg German–French gold standard (RTX 5070,
this repository's `scripts/benchmark_model.py`):

| Model                          | Strict F1 | Lax F1    | Load + align    |
| ------------------------------ | --------- | --------- | --------------- |
| **LaBSE** (default)            | **0.936** | 0.989     | **4.7 + 9.9 s** |
| BAAI/bge-m3                    | 0.935     | **0.992** | 38 + 35 s       |
| google/embeddinggemma-300m     | 0.928     | 0.986     | 33 + 29 s       |
| intfloat/multilingual-e5-large | 0.919     | 0.988     | 35 + 35 s       |
| Qwen/Qwen3-Embedding-0.6B      | 0.915     | 0.990     | 27 + 24 s       |

LaBSE remains the default: it was trained specifically for translation-pair
retrieval, and it is both the most accurate on strict F1 and by far the
fastest. Retrieval models that expect instruction prefixes (e5, Qwen3) lose
accuracy here because bertalign embeds raw sentences.

## Sentence splitting

The default splitter is rule-based
([sentence-splitter](https://github.com/mediacloud/sentence-splitter)) and
supports 25 languages: Catalan, Chinese, Czech, Danish, Dutch, English,
Finnish, French, German, Greek, Hungarian, Icelandic, Italian, Latvian,
Lithuanian, Norwegian, Polish, Portuguese, Romanian, Russian, Slovak,
Slovenian, Spanish, Swedish, and Turkish.

The `sat` extra adds neural splitting with
[wtpsplit's SaT models](https://github.com/segment-any-text/wtpsplit)
(state of the art on multilingual segmentation benchmarks, language-agnostic,
GPU-accelerated) — useful for noisy text or languages the rule splitter does
not cover:

```python
from bertalign import Bertalign
from bertalign.sat import sat_splitter

aligner = Bertalign(src_text, tgt_text, splitter=sat_splitter())
```

or `bertalign --sat source.txt target.txt` on the command line.

## Evaluation

The scorer from [Vecalign](https://github.com/thompsonb/vecalign) ships as
`bertalign.eval`, and the repository carries the text+berg evaluation corpus:

```bash
just eval                                            # gold-standard regression gate
uv run python scripts/benchmark_model.py BAAI/bge-m3 # benchmark another model
```

`just eval` must reproduce Strict F1 0.936 / Lax F1 0.989 exactly; it is the
correctness gate for any change to the aligner, kernels, or dependencies.

## Development

```bash
uv sync --extra sat
just check   # ruff (select = ALL), ty, vulture, pytest, build
```

## Licence and citation

GPL-3.0-or-later, inherited from
[upstream](https://github.com/bfsujason/bertalign). If you use Bertalign in
research, cite:

> Lei Liu & Min Zhu. 2022. Bertalign: Improved word embedding-based sentence
> alignment for Chinese–English parallel corpora of literary texts.
> *Digital Scholarship in the Humanities*.
> [https://doi.org/10.1093/llc/fqac089](https://doi.org/10.1093/llc/fqac089)
