Metadata-Version: 2.4
Name: wolof-normalizer
Version: 0.2.0
Summary: Normalise Wolof text written in French-style / SMS spelling into the official orthography (CLAD).
Author: Baye Lahad Mbacke
License-Expression: MIT AND CC-BY-SA-4.0
Project-URL: Homepage, https://github.com/LahadMbacke/wolof-normalizer
Project-URL: Model, https://huggingface.co/Lahad/wolof-normalizer-byt5-small
Project-URL: Dataset, https://huggingface.co/datasets/Lahad/wolof-normalization-pairs
Keywords: wolof,nlp,normalization,spelling,low-resource,senegal
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: model
Requires-Dist: torch>=2.6; extra == "model"
Requires-Dist: transformers>=5; extra == "model"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Provides-Extra: eval
Requires-Dist: pyarrow; extra == "eval"
Dynamic: license-file

# wolof-normalizer

Normalise Wolof text written in French-style / SMS spelling into the **official orthography** (CLAD alphabet).

```
$ wolof-normalize "Nanga def ? Mangui fi rek, dieuredieuf"
Na nga def ? Maa ngi fi rekk, jërëjëf
```

Most Wolof online (social media, messaging) is written with French spelling habits: `kh` for `x`, `ou` for `u`, `gn` for `ñ`, `dieu` for `jë`… This makes search, NLP tools and corpus building much harder. This small library maps these spellings back to the standard form.

## How it works

For each word, from safest to least safe:

1. **Lexicon** of known variants, including multi-word expressions (`nanga def` → `na nga def`):
   - [`data/lexicon.tsv`](https://github.com/LahadMbacke/wolof-normalizer/blob/main/src/wolof_normalizer/data/lexicon.tsv): hand-written, takes priority;
   - [`data/lexicon_mined.tsv`](https://github.com/LahadMbacke/wolof-normalizer/blob/main/src/wolof_normalizer/data/lexicon_mined.tsv): ~930 variants mined automatically from a parallel corpus (see [Evaluation](#evaluation)).
2. Words already in the **official vocabulary** ([`data/vocab.txt`](https://github.com/LahadMbacke/wolof-normalizer/blob/main/src/wolof_normalizer/data/vocab.txt)) are kept.
3. **Grapheme rules** (`tch`→`c`, `kh`→`x`, `ch`→`c`, `dj`→`j`, `gn`/`ny`→`ñ`, `ou`→`u`, `eu`→`ë`, `di`+vowel→`j`, final consonant gemination). A rule-based correction is applied **only if the result exists in the vocabulary**, to avoid breaking names and French words.
4. Otherwise the word is left unchanged.

Case and punctuation are preserved.

## Install & use

```bash
pip install wolof-normalizer            # lexicon + rules (lightweight)
pip install "wolof-normalizer[model]"   # + neural model (PyTorch, ~1.2 GB download on first use)
```

```python
from wolof_normalizer import normalize, normalize_with_changes

normalize("dafa beug wakh")          # 'dafa bëgg wax'
normalize_with_changes("kham nga ko ?")
# ('xam nga ko ?', [Change(original='kham', normalized='xam', reason='rules')])
```

Neural model (more accurate, slower):

```python
from wolof_normalizer import ModelNormalizer

normalizer = ModelNormalizer()   # downloads Lahad/wolof-normalizer-byt5-small
normalizer.normalize("gnou bari dagnou meuna dem keur gui souba")
# 'Ñu bari dañu mëna dem kër gi suba.'
```

Command line:

```bash
wolof-normalize "khale yi"            # xale yi
cat messages.txt | wolof-normalize -v # -v prints each change on stderr
wolof-normalize --model "khale yi dagnou beugue fo"   # neural model
```

From source: `pip install -e ".[dev]"`.

## Tests

```bash
pytest
```

## Evaluation

Evaluated on [soynade-research/Wolof-Non-Standard-Orthography](https://huggingface.co/datasets/soynade-research/Wolof-Non-Standard-Orthography) (3,438 non-standard / standard sentence pairs). The first 80% is used to mine the lexicon, the last 20% (688 sentences) for evaluation. Metric: word error rate (WER) against the standard sentence, lowercased, punctuation ignored.

| System | WER ↓ |
|---|---|
| No normalization | 64.9% |
| Hand lexicon + rules | 56.1% |
| + mined lexicon (≥ 3 occurrences) | 22.7% |
| **ByT5-small fine-tuned** (see below) | **11.2%** |
| Lexicon + rules, then ByT5 | 14.0% |

```bash
pip install -e ".[eval]"
cd scripts
python mine_lexicon.py   # writes candidates.tsv
python run_evaluation.py
```

Note: the non-standard side of this dataset is synthetic (generated by an LLM from real social-media patterns), so scores on real messages may differ.

## Neural model (ByT5)

Model on the Hub: [Lahad/wolof-normalizer-byt5-small](https://huggingface.co/Lahad/wolof-normalizer-byt5-small)

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LahadMbacke/wolof-normalizer/blob/main/notebooks/train_byt5_colab.ipynb)

A byte-level model ([`google/byt5-small`](https://huggingface.co/google/byt5-small)) is fine-tuned to handle words the lexicon does not know and to use sentence context. It halves the error rate of the lexicon (**11.2% vs 22.7% WER** on the test set). Running the lexicon before the model hurts (14.0%), so the model is best used on raw text.

Training: 5 epochs, effective batch 32, Adafactor, lr 5e-4, max 256 bytes, fp32, ~50 min on one V100 32GB; best validation WER 9.8%.

Training data (`scripts/make_synthetic.py`, ~32k pairs, published as [Lahad/wolof-normalization-pairs](https://huggingface.co/datasets/Lahad/wolof-normalization-pairs), CC BY-SA 4.0):
- the real pairs of the first 80% of the dataset above (200 kept aside for validation);
- standard Wolof sentences from [google/smol](https://huggingface.co/datasets/google/smol) (CC BY 4.0) and [Wolof Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) (CC BY-SA 4.0), turned into "SMS-style" Wolof with the mined lexicon used in reverse plus inverse grapheme rules (`x`→`kh`, `u`→`ou`, `ñ`→`gn`, dropped accents and geminates…). Test sentences are excluded from these sources.

```bash
# trains on Lahad/wolof-normalization-pairs from the Hugging Face Hub (GPU recommended, see notebook)
python scripts/train_byt5.py --output-dir runs/byt5-small
python scripts/run_evaluation.py --data-repo Lahad/wolof-normalization-pairs --model runs/byt5-small/best

# or rebuild the data locally
python scripts/make_synthetic.py
python scripts/train_byt5.py --local-data --output-dir runs/byt5-small
```

## Contributing

The lexicon and vocabulary are small for now and are the easiest place to help: add one `variant<TAB>official` line to `lexicon.tsv`, or one official word per line to `vocab.txt`.

## Roadmap

- Grow the lexicon and vocabulary from public Wolof resources
- Evaluation set (SMS sentences ↔ official spelling) and accuracy metrics
- Handle vowel length and accents (`a`/`à`/`aa`, `e`/`é`/`ë`)
- Learned model (character-level) for words outside the vocabulary
- Publish on PyPI and as a Hugging Face Space demo

## License

Code: MIT, Baye Lahad Mbacke.

`data/lexicon_mined.tsv` is derived from [soynade-research/Wolof-Non-Standard-Orthography](https://huggingface.co/datasets/soynade-research/Wolof-Non-Standard-Orthography) and is distributed under its license, [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/).
