Metadata-Version: 2.1
Name: amtokenizers
Version: 0.0.6
Summary: Amharic language tokenizers
Home-page: UNKNOWN
Author: Mitiku Yohannes
Author-email: se.mitiku.yohannes@gmail.com
License: MIT
Platform: UNKNOWN
Requires-Dist: transformers

<h1>Amharic Language Tokenizers</h1>

This package contains set of Classes which can be used to encode Amharic language sentences into tokens that could be used by language models. The tokenizers are trained using [Contemporary Amharic Corpus (CACO)](https://www.findke.ovgu.de/findke/en/Research/Data+Sets/Contemporary+Amharic+Corpus+%28CACO%29-p-1142.html) dataset

<h2>Installing using pip</h2>
The package can be installed from test.pypi using the following command.
<code>
pip install -i https://test.pypi.org/simple/ amtokenizers==0.0.1
</code>

<h2>Example usage</h2>

<code>
from amtokenizers import AmTokenizer

a  = AmTokenizer(10000, 5 , "byte_bpe")
encoded = a.encode("አበበ በሶ በላ።", return_tokens=False)
print("encoded", encoded.tokens)
print("decoded:", a.decode(encoded.ids))
</code>

<h2>Disclaimer</h2>

This package is highly inspired by Hugging Face's [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train) tutorial.





# Amharic Langugae Tokenizers

This package contains set of Classes which can be used to encode Amharic language sentences into tokens that could be used by language models. The tokenizers are trained using [Contemporary Amharic Corpus (CACO)](https://www.findke.ovgu.de/findke/en/Research/Data+Sets/Contemporary+Amharic+Corpus+%28CACO%29-p-1142.html) dataset


## Installing


#### Pip installation

```bash
pip install -i https://test.pypi.org/simple/ amtokenizers==0.0.5
```

## Sample Code

```python
# toolchain (g++, make) is very likely installed along Xcode, just install cmake and pkg-config using homebrew
print("Hello world")
```


