Every token scores against every other token, so the cost grows with the square of the length.
Where the scores come from
In self-attention each query meets every key.
Where the weights come from
Each row of scores goes through a softmax.
Why does a longer context cost more than it adds?
Every new token scores against all the others.