Metadata-Version: 2.4
Name: eztokenizer
Version: 0.1.4
Summary: A tiny tokenizer library for tiny Generative Pretrained Transformers.
Author: im-lemon
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: torch

# EzTokenizer

---

EzTokenizer is a simple and user-friendly tokenizer for small language models and/or Generative Pretrained Transformers (GPT's).

It's main purpose is to automate and make tokenization easy, so you don't have to program one from scratch.

 - - - 
# Inspirations

EzTokenizer was inspired by [Transformers's](https://github.com/huggingface/transformers) AutoTokenizer feature.

What makes EzTokenizer stand out is that it automates tokenization, is easy to use, and it doesn't rely on a pre-trained vocabulary.

# About the usage of AI

AI was used for instructions on how to get started, all of EzTokenizer's code is hand-written.


---
Thank you for reading! - *im-lemon*
