Metadata-Version: 2.4
Name: afrisyl-tokenizer
Version: 0.1.1
Summary: A syllable-aware tokenizer for African languages
Author-email: Nkosilomusa Ncube <nkosilomusa955@gmail.com>
Maintainer-email: Nkosilomusa Ncube <nkosilomusa955@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/nkosilomusa-hue/afrisyl-tokenizer
Project-URL: Repository, https://github.com/nkosilomusa-hue/afrisyl-tokenizer
Project-URL: Documentation, https://github.com/nkosilomusa-hue/afrisyl-tokenizer#readme
Project-URL: Issues, https://github.com/nkosilomusa-hue/afrisyl-tokenizer/issues
Keywords: nlp,tokenizer,african languages,shona,ndebele,bantu,syllable,low-resource
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Natural Language :: English
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: regex>=2023.0.0
Requires-Dist: tqdm>=4.65.0
Requires-Dist: tokenizers>=0.13.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: black; extra == "dev"
Requires-Dist: flake8; extra == "dev"
Requires-Dist: transformers>=4.30.0; extra == "dev"
Dynamic: license-file

# Afrisyl-tokenizer

**A syllable-aware tokenizer for African languages**

Afrisyl-tokenizer is built to fix a core problem: BPE and WordPiece tokenizers break African languages.
Instead of splitting "ndinoda" into ['n', 'din', 'oda'], we respect the natural CV syllable structure of Bantu languages.

Built as part of MSc research by Nkosilomusa Ncube, Alumni WeThinkCode.

## Key Features
- **Syllable-based**: Uses CV, CVC, V patterns found in Shona, Ndebele, Zulu, Swahili, etc
- **Low fertility**: 30% fewer tokens than BPE on Shona text
- **Fast**: Pure Python + regex. No dependencies for inference
- **HF Compatible**: Drop-in replacement for HuggingFace tokenizers
- **Trainable**: Train your own tokenizer on any African language corpus
- **Small vocab**: 8k-16k vocab covers 95%+ of tokens
- **Open Source**: MIT Licensed for research and social impact

## Installation
```bash
pip install afrisyl-tokenizer
```

## Quick Usage

### 1. Load Tokenizer
```python
from afrisyl_tokenizer import AfriSylTokenizer

# Load default shona vocab from package
tok = AfriSylTokenizer(language="shona")

# Or load custom vocab
# tok = AfriSylTokenizer(vocab_path="path/to/ndebele_vocab.json")

# Batch processing
texts = ["ndinoda rubatsiro", "mhoroi shamwari"]

batch = tok.batch_encode(
    texts, 
    add_bos=True, 
    add_eos=True, 
    max_len=32, 
    padding=True
)
# batch['input_ids'] -> [[1, 12, 45, ...], [1, 8, 22, ...]]
# batch['attention_mask'] -> [[1, 1, 1, ...], [1, 1, 1, ...]]

decoded_batch = tok.batch_decode(batch["input_ids"])
print(decoded_batch)
# ['ndinodarubatsiro', 'mhoroshamwari']
```
