Current take

The softmax over every pair of tokens sets the cost.

Basics

• Softmax Read first.
• Self-attention
• FlashAttention tiling
• Why attention is quadratic
• Attention, from scratch

Reading

• Attention Is All You Need
• Which attention methods are exact
• Tiled attention
• Attention in five minutes