Metadata-Version: 2.4
Name: lvnlp
Version: 0.2.0
Summary: Latvian NLP tools
Project-URL: Homepage, https://github.com/LUMII-AILab/lvnlp
Project-URL: Repository, https://github.com/LUMII-AILab/lvnlp
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: torch>=2.9.0
Requires-Dist: smart-open>=7.5.0
Requires-Dist: tqdm>=4.67.1
Requires-Dist: transformers>=4.57.1
Requires-Dist: huggingface-hub>=0.36.2
Requires-Dist: ufal.chu-liu-edmonds>=1.0.3
Requires-Dist: sentencepiece>=0.2.1
Requires-Dist: safetensors>=0.7.0
Provides-Extra: train
Requires-Dist: wandb>=0.23.0; extra == "train"

# lvnlp

Latvian NLP tools for morphosyntactic parsing.

## Installation

Install from PyPI:

```bash
pip install lvnlp
```

Or install the latest development version from GitHub:

```bash
pip install git+https://github.com/LUMII-AILab/lvnlp.git
```

## Morphosyntactic parsing

```python
from lvnlp.parser import Parser
from lvnlp.parser.tokenize import tokenize_sentences
from lvnlp.parser.utils import to_conll

parser = Parser.from_pretrained()
sentences = parser.parse(tokenize_sentences('Jānis brauca uz Rīgu. Viņš atgriezās vakarā.'))
print(to_conll(sentences))
```

The parser can optionally use the Latvian morphological analyzer during decoding. This mainly improves XPOS tagging and lemmatization: analyzer candidates are scored with the model's tag and lemma-rule probabilities, the best candidate's lemma is adopted, and its tag replaces the model's tag when it scores within a margin of it.

## Corpus tagging

`lvnlp tag` retags a corpus file in one streaming pass. Vertical (`.vert`) and jsonl files are
read and written, optionally gzipped; the format follows the file name.

```bash
lvnlp tag -i corpus.vert.gz -o tagged.vert.gz                        # form, lemma, xpos
lvnlp tag -i corpus.vert -o tagged.vert --columns 6                  # + upos, head, deprel
lvnlp tag -i corpus.vert -o tagged.vert --in-place                   # keep the input columns, overwrite the tagger's
lvnlp tag -i docs.jsonl -o tagged.jsonl.gz --device cuda:0*3         # {"id", "text"} documents; three model workers
lvnlp tag -i corpus.vert -o tagged.vert --analyzer --decode-workers 2  # rescore with the morphological analyzer
lvnlp tag -i corpus.vert -o tagged.vert --device cpu*8 --cpu-threads 1
```

The CLI can also run without installing: [uv](https://docs.astral.sh/uv/) fetches the package into a cached environment on first use.

```bash
# install uv once, then open a new shell
curl -LsSf https://astral.sh/uv/install.sh | sh

uvx lvnlp tag -i corpus.vert.gz -o tagged.vert.gz --device cuda:0
uvx lvnlp==0.2.0 tag -i corpus.vert.gz -o tagged.vert.gz --device cuda:0   # specific version
uvx --from 'git+https://github.com/LUMII-AILab/lvnlp.git@main' lvnlp tag -i corpus.vert.gz -o tagged.vert.gz   # development version
```

Main options: `--device cuda:0`, `cuda:0*N` (N model workers sharing one GPU), `cuda:0,cuda:1`
or `cpu*N`; `--decode-workers K` adds K processes per model worker that decode and rescore
batches, which is what makes `--analyzer` fast; `--batch-size` (128) and `--buffer-words`
(20000) size the forward passes; `--limit N` stops after N documents.

Steady-state throughput of `lv_ud2.18_lv-deberta-base` (bf16 on GPU, fp32 on CPU):

| setup | plain | with `--analyzer` | memory |
|---|---:|---:|---|
| one A100: `--device cuda:0` (+ `--decode-workers 2` with the analyzer) | 33k words/s | 21k words/s | 14 GB VRAM, 2-4 GB RAM |
| one A100: `--device cuda:0*3` (+ `--decode-workers 2` with the analyzer) | 53k words/s | 47k words/s | 35 GB VRAM, 6-11 GB RAM |
| 1 CPU core: `--device cpu --cpu-threads 1` | 73 words/s | ~70 words/s | 2 GB RAM |
| 8 CPU cores: `--device cpu*8 --cpu-threads 1 --buffer-words 1000` | ~0.6k words/s | ~0.6k words/s | 16 GB RAM |
| 64 CPU cores: `--device cpu*64 --cpu-threads 1 --buffer-words 1000` | 4.7k words/s | ~5k words/s | 126 GB RAM |

On CPU the forward pass dominates, so the analyzer costs little there; each CPU process needs about
2 GB RAM.

## Citation

```bibtex
@inproceedings{znotins-2026-improving,
  title = {Improving Latvian Morphosyntactic Parsing with Pretrained Encoders and Analyzer-Constrained Decoding},
  author = {Znotins, Arturs},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026)},
  month = {May},
  year = {2026},
  pages = {11724--11734},
  address = {Palma, Mallorca, Spain},
  publisher = {European Language Resources Association (ELRA)},
  editor = {Piperidis, Stelios and Bel, Núria and van den Heuvel, Henk and Ide, Nancy and Krek, Simon and Toral, Antonio},
  doi = {10.63317/5khpzsaiqrzw},
  url = {https://lrec.elra.info/lrec2026-main-918}
}
```

## Acknowledgements

This work was supported by the EU Recovery and Resilience Facility project
[Language Technology Initiative](https://www.vti.lu.lv)
(2.3.1.1.i.0/1/22/I/CFLA/002).
