Metadata-Version: 2.1
Name: bpetokenizer
Version: 1.0.0
Summary: Byte Pair Encoding Tokenizer with special tokens and regex pattern
Home-page: https://github.com/Hk669/bpetokenizer
Author: Hrushikesh Dokala
Author-email: hrushi669@gmail.com
License: MIT
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9,<3.13
Description-Content-Type: text/markdown
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: twine; extra == "dev"

# bpetokenizer

