Why Attention Divides by Square Root of d

Why does attention divide by the square root of d? Skip it, and your transformer quietly breaks.

• Attention's mysterious scaling factor

Attention scores are dot products between a query and a key. Each one measures how well two vectors match.

• Query dot key
• Higher means better match

But watch what happens as dimension grows. Add more components, and the dot products get bigger and bigger.

• More dimensions, wider spread
• Spread grows with √d

Feed those huge scores into softmax, and it collapses into a spike. One token gets almost all the weight.

• One token takes everything
• Others get near zero

A spike is flat almost everywhere, so gradients vanish. The model stops learning which tokens matter.

• Saturated softmax is flat
• Learning stalls

Here is the math. Each component product has variance one, so a sum of d of them has variance d.

• Each term has variance 1
• Sum of d terms gives d

So divide by the square root of d. Variance shrinks back to one, no matter the dimension.

• Variance returns to 1
• Works for any dimension

Now softmax stays soft. Attention spreads across tokens, and gradients keep flowing.

• Smooth weights
• Gradients keep flowing

We derive the full thing with real numbers in lesson fifty-five. Follow, so you don't miss it.

• Attention derived with numbers