A language model learns patterns from text.
A language model predicts the next token.
A token can be a word or a piece of a word.
The tokenizer converts text into token IDs.
The model converts token IDs into embeddings.
Embeddings represent tokens as vectors.
A transformer uses attention to process context.
Self attention helps the model understand relationships between tokens.
Causal attention prevents the model from seeing future tokens.
The feed forward network processes token representations.
The model learns by comparing predictions with target tokens.
The loss measures how incorrect the predictions are.
Backpropagation calculates gradients.
The optimizer updates the model parameters.
Training helps the model improve its next token predictions.
A GPT model generates text one token at a time.
The generated token becomes part of the next input.