Softmax
The softmax of a vector of scores is the exponential of each score divided by the sum of the exponentials.
The trap
Large scores overflow the exponential. Self-attention subtracts the row maximum first.
Softmax
The softmax of a vector of scores is the exponential of each score divided by the sum of the exponentials.
Large scores overflow the exponential. Self-attention subtracts the row maximum first.