What it claims

• Self-attention links any two positions in a constant number of steps, where a recurrent layer needs as many steps as the sequence is long.

My take

It holds for short sequences. It agrees with our softmax page.