Metadata-Version: 2.4
Name: lexibyte
Version: 0.3.0
Summary: A production-grade, from-scratch implementation of a Byte-Pair Encoding (BPE) tokenizer.
Author-email: Bibek Dhakal <imbibek8366@gmail.com>
Project-URL: Homepage, https://github.com/Bibek-Dhakal/lexibyte
Project-URL: Bug Tracker, https://github.com/Bibek-Dhakal/lexibyte/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: regex
Dynamic: license-file

# 🔤 LexiByte Engine ("Lexi")

**LexiByte** is a production-grade, algorithmically optimized **Byte-Pair Encoding (BPE)** tokenizer, inspired by the
architecture used in modern Large Language Models (LLMs) like GPT-2, GPT-4, and Llama.

Built entirely in Python, LexiByte bridges the gap between raw text and Neural Networks by safely compressing UTF-8 byte
streams into integer token sequences.

## ✨ Production-Grade Features

- **Sennrich Frequency Optimization:** Escapes the $O(N^2)$ training bottleneck by compressing the corpus into a
  frequency dictionary. It trains on millions of words in seconds by mathematically weighting unique pairs.
- **Inference Memoization Cache:** Bypasses BPE loops during inference by caching tokenized sequences, delivering $O(1)$
  constant time lookups for previously seen words.
- **Regex Pre-Splitting Guardrails:** Implements GPT-2/GPT-4 style regex boundaries to prevent unnatural merges (e.g.,
  merging punctuation with words, or trailing spaces with letters).
- **UTF-8 Byte Level Base:** Starts with a base vocabulary of 256 standard UTF-8 bytes, meaning it can theoretically
  encode *any* string (including emojis and non-English scripts) without out-of-vocabulary (OOV) errors.
- **Special Token Support:** Safely handles control tokens (e.g., `<|endoftext|>`).

## 📚 Documentation

Detailed documentation has been separated into the following guides:

- [Architecture & Internal Mechanics](https://github.com/Bibek-Dhakal/lexibyte/blob/main/docs/architecture.md) -
  Understand how BPE works, why regex guardrails matter, and view visual pipeline diagrams.
- [Training & Inference Deep Dive](https://github.com/Bibek-Dhakal/lexibyte/blob/main/docs/training_and_inference_deep_dive.md) -
  A concrete, step-by-step look at how the `merges` and `vocab` maps are built and how production optimization works.
- [Usage Guide](https://github.com/Bibek-Dhakal/lexibyte/blob/main/docs/usage.md) - Step-by-step instructions on
  training, encoding, decoding, and saving your tokenizer.
- [Developer & Publishing Guide](https://github.com/Bibek-Dhakal/lexibyte/blob/main/docs/developer_guide.md) -
  Instructions for maintainers on how to build and publish the package to PyPI.

## 🚀 Quick Start

### Installation (For Users)

You can install LexiByte directly from PyPI using pip.

```bash
pip install lexibyte
```

*(If you are developing locally from the source repository, run `pip install -r requirements.min.txt` instead).*

### Basic Example

```python
from lexibyte import LexiByteTokenizer

# Initialize tokenizer
tokenizer = LexiByteTokenizer()

# Sample corpus
text = "hello world! 👋 This is the LexiByte engine."

# Train the tokenizer to reach a vocabulary of 276 (256 base bytes + 20 learned merges)
tokenizer.train(text, vocab_size=276, verbose=True)

# Encode text to token IDs
encoded = tokenizer.encode("hello world!")
print("Encoded:", encoded)

# Decode token IDs back to text
decoded = tokenizer.decode(encoded)
print("Decoded:", decoded)
```

## 🧠 Context

This project was developed as an advanced Systems Engineering project to bridge the gap between educational tokenizers
and scalable production logic.

---
