Metadata-Version: 2.5
Name: tokencraft-gpt
Version: 0.1.1
Summary: An educational GPT language model implemented from scratch with PyTorch.
Author: Ranjeet Epili
License-Expression: MIT
Requires-Python: >=3.10
Requires-Dist: torch>=2.0
Description-Content-Type: text/markdown

# TokenCraft

### Learn how GPT-style language models work — from tokens to text generation.

**TokenCraft** is an educational, decoder-only GPT-style language model implemented with PyTorch. It helps learners explore the building blocks behind transformer language models: tokenization, embeddings, causal self-attention, transformer blocks, training, and autoregressive text generation.

> **Install TokenCraft**
>
> ```bash
> pip install tokencraft-gpt
> ```

**Author:** Ranjeet Epili

TokenCraft is an educational project—not a pretrained general-purpose chatbot. It is designed to help you understand how a GPT-style model works and experiment with training on your own text.

---

## Table of Contents

- [What is TokenCraft?](#what-is-tokencraft)
- [The big picture](#the-big-picture)
- [How the transformer works](#how-the-transformer-works)
  - [1. Text and tokenization](#1-text-and-tokenization)
  - [2. Token and positional embeddings](#2-token-and-positional-embeddings)
  - [3. Causal self-attention](#3-causal-self-attention)
  - [4. Query, Key, and Value](#4-query-key-and-value)
  - [5. Multi-head attention](#5-multi-head-attention)
  - [6. Feed-forward network and transformer block](#6-feed-forward-network-and-transformer-block)
  - [7. Next-token training](#7-next-token-training)
  - [8. Autoregressive generation](#8-autoregressive-generation)
- [Installation](#installation)
- [How to use TokenCraft](#how-to-use-tokencraft)
- [Project structure](#project-structure)
- [Limitations](#limitations)
- [Inspiration and acknowledgements](#inspiration-and-acknowledgements)
- [Author and license](#author-and-license)

## What is TokenCraft?

A language model learns statistical patterns in sequences. Given a context such as `the model learns`, it estimates which token could come next. TokenCraft demonstrates this process through a compact GPT-style architecture.

It includes educational implementations of:

- Tokenization and token IDs
- Token and positional embeddings
- Causal self-attention
- Query, Key, and Value projections
- Multi-head attention
- Feed-forward networks and transformer blocks
- Next-token training
- Autoregressive text generation

## The big picture

![TokenCraft decoder-only GPT architecture](images/tokencraft-architecture.svg)

The model converts text into token IDs, maps those IDs into vectors, processes the vectors through transformer blocks, and produces scores for possible next tokens. During generation, a selected token is appended to the context and the model runs again.

## How the transformer works

### 1. Text and tokenization

![Text to tokens to token IDs](images/tokenization.svg)

A tokenizer turns text into a sequence of integer token IDs. Depending on the tokenizer, a token can represent a word, a word fragment, punctuation, or another text unit.

For a small illustrative vocabulary:

| Token | Example ID |
|---|---:|
| `the` | 12 |
| `model` | 37 |
| `learns` | 21 |

Token IDs are not numerical measurements of meaning. They are indices used to look up vectors in an embedding table. The IDs above are only an example; your tokenizer's vocabulary will determine the actual IDs.

### 2. Token and positional embeddings

![Token plus positional embeddings](images/embeddings.svg)

The embedding table maps every token ID to a learned vector. But a sequence also has order: “the model” and “model the” contain the same words in different positions.

A positional embedding provides information about where a token appears. Token and positional embeddings are combined to form the initial representation:

**Input representation = token embedding + positional embedding**

This gives the transformer information about both the token and its position in the sequence.

### 3. Causal self-attention

Self-attention lets each token build a context-aware representation by gathering information from other tokens. A GPT-style model uses **causal attention**: a position may attend to itself and earlier positions, but not future tokens.

![Causal attention mask](images/causal-mask.svg)

In this diagram, green cells are allowed and gray cells are blocked. The third token can use tokens one, two, and three—but cannot see token four. This prevents the model from “cheating” by looking at the answer while learning to predict the next token.

### 4. Query, Key, and Value

Each input vector is transformed into three learned representations:

- **Query (Q):** what the current position is looking for
- **Key (K):** what each position can be matched against
- **Value (V):** the information each position contributes

The scaled dot-product attention operation is:

\[
\operatorname{Attention}(Q,K,V)
=\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V
\]

Here, \(d_k\) is the key dimension and \(M\) is the causal mask. Query-key similarity produces attention scores. The mask blocks future positions, softmax converts the scores into weights, and those weights combine the value vectors.

![Query Key Value attention flow](images/attention-qkv.svg)

### 5. Multi-head attention

![Multi-head attention](images/multi-head-attention.svg)

One attention head learns one set of contextual relationships. **Multi-head attention** runs several attention heads in parallel, each with its own learned projections.

The heads' outputs are concatenated and projected into a shared representation. Different heads can learn different patterns—for example, relationships between nearby tokens or dependencies across a longer context.

### 6. Feed-forward network and transformer block

![Transformer block](images/transformer-block.svg)

After attention, a feed-forward network applies learned transformations to each position's representation. A transformer block combines attention and feed-forward processing, along with residual connections and normalization where implemented.

Stacking transformer blocks lets the model refine its contextual representations layer by layer.

### 7. Next-token training

![Next-token training loop](images/next-token-training.svg)

GPT-style language modeling is trained as a next-token prediction task. The target sequence is shifted by one token relative to the input:

| Context provided to the model | Target token |
|---|---|
| `the` | `model` |
| `the model` | `learns` |
| `the model learns` | `from` |

The model produces logits—scores for tokens in its vocabulary. Cross-entropy loss compares the predictions with the correct next tokens. An optimizer such as AdamW updates the model parameters to reduce this loss.

Training text, model size, and training duration influence what the model learns. A small corpus is useful for demonstrating the process, but does not give a model broad language knowledge.

### 8. Autoregressive generation

After training, the model can generate text from a prompt. It predicts the next token, appends the selected token to the context, and predicts again.

![Autoregressive generation](images/autoregressive-generation.svg)

For example, a prompt might be:

```text
The model
```

The model may continue with a sequence such as:

```text
The model learns from text
```

This is an illustrative example, not a guaranteed output. The actual continuation depends on learned weights, prompt, and sampling settings.

**Temperature** adjusts the sampling distribution:

- Lower temperature generally makes token selection more focused.
- Higher temperature generally makes token selection more varied.

Generation is iterative: each generated token becomes part of the context for the next prediction.

## Installation

Install TokenCraft from PyPI:

```bash
pip install tokencraft-gpt
```

The **distribution name** used by pip is `tokencraft-gpt`. The **Python import name** is `tokencraft`.

## How to use TokenCraft

### 1. Import the package

```python
import tokencraft
from tokencraft import GPT, Tokenizer

print(tokencraft.__version__)
```

### 2. Inspect the available API

The model and tokenizer classes are exposed by the package. Use Python's built-in help to inspect the constructor arguments and methods available in your installed version:

```python
help(GPT)
help(Tokenizer)
```

### 3. Prepare a text dataset

Create a plain-text file containing the material you want the model to learn from. For example, save this as `data.txt`:

```text
A language model learns patterns from text.
A transformer uses attention to process sequences.
TokenCraft demonstrates next-token prediction.
```

For meaningful experiments, use a larger and appropriately prepared dataset. Keep track of the data you train on and ensure you have permission to use it.

### 4. Train the model

A typical training workflow is:

1. Load the training text.
2. Build the tokenizer vocabulary and encode the text into token IDs.
3. Create input sequences and next-token targets.
4. Initialize the GPT model using the model configuration.
5. Run the training loop with a next-token loss such as cross-entropy.
6. Update the parameters with an optimizer such as AdamW.
7. Save the trained model parameters for later use.

TokenCraft's exact constructor arguments and training options are version-specific. Inspect `GPT` and `Tokenizer` with `help()` and refer to the project's training example or scripts for the precise commands.

### 5. Generate text from a trained model

The generation workflow is:

1. Load the trained model parameters.
2. Encode a prompt with the same tokenizer used during training.
3. Pass the prompt tokens to the model.
4. Select a next token from the model's output.
5. Append the token and repeat until the desired length or stopping condition is reached.
6. Decode the generated token IDs back into text.

The model must be trained first—or loaded from a compatible checkpoint—before its output can reflect learned patterns. The package does not automatically download pretrained weights.

### Quick API check

Use this small snippet to confirm that the package installed and its main classes are importable:

```python
import tokencraft

print("TokenCraft version:", tokencraft.__version__)
print("GPT class:", tokencraft.GPT)
print("Tokenizer class:", tokencraft.Tokenizer)
```

## Project structure

The package is organized around the main GPT components:

```text
tokencraft/
├── __init__.py
├── tokenizer.py
└── model/
    ├── __init__.py
    ├── gpt.py
    ├── embedding.py
    ├── positional_embedding.py
    ├── input_embedding.py
    ├── self_attention.py
    ├── multi_head_attention.py
    ├── feed_forward.py
    └── transformer_block.py
```

The source repository may also contain training scripts, generation examples, and sample data.

## Limitations

TokenCraft is intended for education and experimentation:

- It is not a pretrained production chatbot.
- Generated text depends on the training data and model configuration.
- Small models and small datasets may generate repetitive or incoherent text.
- It is not designed to match the capabilities of large-scale language models.

## Inspiration and acknowledgements

TokenCraft's educational approach is inspired by transformer literature and visual explanations such as Jay Alammar's *The Illustrated Transformer*. This README uses original diagrams and explanations tailored to a decoder-only GPT-style model; it does not reproduce that article's illustrations.

## Author and license

**Author:** Ranjeet Epili

**License:** MIT
