How does a model know which word in a sentence to focus on? Attention turns word similarity into weights.
Introduction
Which Word Matters?
How does an AI know which word actually matters?
One Sentence
Take one sentence. The cat sat because it was tired.
- Split into tokens
Tokens Become Vectors
First, each word becomes a token. Each token becomes a vector, a list of numbers.
- Token
- Vector of numbers
The Query
Now every token asks a question. We call it the query.
- Each token asks a question
Query Meets Key
Every other token offers a key. We compare query and key to measure similarity.
- Compare by dot product
- Aligned means similar
Softmax Makes Weights
Raw scores can be any size. Softmax squeezes them into weights that add up to one.
- Scores to probabilities
- Weights sum to one
The Attention Heatmap
Stack them all and you get a heatmap. Bright cells show where each word is looking.
- Bright means more focus
Focus Where It Counts
So it looks at cat, not mat. Attention is just similarity, turned into focus.
- Similarity becomes focus
Follow for More
Follow for the next step in machine learning.