StudyBrixFree worksheets, study guides and courses

Artificial intelligence · lesson 3 of 4

Attention, softmax and transformer structure

Scaled dot-product attention forms weights softmax(QKᵀ/sqrt(dk)) and uses them to combine value vectors. Dimensions and masks are part of the definition.

Advanced university8 practice questionsNotes + worked examples
Take the quizStudy guideRevision sheet

Learn the ideas

Prerequisites: Linear algebra, probability and neural layers.

Attention

Scaled dot-product attention forms weights softmax(QKᵀ/sqrt(dk)) and uses them to combine value vectors. Dimensions and masks are part of the definition.

Softmax

Softmax converts a finite score vector to nonnegative weights summing to one. A common additive shift leaves the result unchanged.

Position

Pure self-attention without position information is permutation-equivariant. Position representations and masks supply relevant ordering structure.

Interpretation

Attention weights are internal computational quantities. They should not automatically be treated as complete causal explanations of a model output.

Worked examples

Example 1

Two attention scores are both zero. Find either softmax weight.

exp(0)/(exp(0)+exp(0))=1/2.

0.5

Example 2

Attention weights are 0.25 and 0.75, and scalar values are 2 and 6. Find the weighted value.

0.25×2+0.75×6=5.

5

Check your understanding

8 questions

Decimals or fractions are accepted for numeric answers. Proofs and written responses use sample answers and require human marking. This is practice, not a protected examination.

1 Two attention scores are both zero. Find either softmax weight.

Show answer and explanation

0.5

exp(0)/(exp(0)+exp(0))=1/2.

2 Attention weights are 0.25 and 0.75, and scalar values are 2 and 6. Find the weighted value.

Show answer and explanation

5

0.25×2+0.75×6=5.

3 What is the purpose of a causal attention mask in an autoregressive training setup?

Show answer and explanation

Prevent a position from attending to future tokens

The mask respects the next-token prediction ordering.

4 What happens to softmax when the same constant is added to every finite score?

Show answer and explanation

It stays the same

A common exponential factor cancels from numerator and denominator.

5 If dk=64, find the usual dot-product scaling denominator sqrt(dk).

Show answer and explanation

8

sqrt(64)=8.

6 Why is positional information useful in a sequence model built from self-attention?

Sample answer and marking guidance

Without it, the operation does not by itself encode token order. Position information helps distinguish different sequences with the same tokens.

Credit ordering rather than claiming attention cannot process sets.

7 Why should attention weights not be presented automatically as a causal explanation?

Sample answer and marking guidance

Other layers, value representations and interactions also shape the result. Causal explanations require appropriate interventions or analysis beyond the weight display.

Credit computational context and evidence.

8 Which vector collection is combined using attention weights?

Show answer and explanation

Value vectors

Weights aggregate the V representations in scaled dot-product attention.

Further reading and review references

These are reference links, not copied exercises, institutional endorsement or proof of independent review.