Artificial intelligence · Advanced university
Attention, softmax and transformer structure · revision notes
Scaled dot-product attention forms weights softmax(QKᵀ/sqrt(dk)) and uses them to combine value vectors. Dimensions and masks are part of the definition.
Key ideas and conditions
Attention
Scaled dot-product attention forms weights softmax(QKᵀ/sqrt(dk)) and uses them to combine value vectors. Dimensions and masks are part of the definition.
Softmax
Softmax converts a finite score vector to nonnegative weights summing to one. A common additive shift leaves the result unchanged.
Position
Pure self-attention without position information is permutation-equivariant. Position representations and masks supply relevant ordering structure.
Interpretation
Attention weights are internal computational quantities. They should not automatically be treated as complete causal explanations of a model output.
Explain without looking
- Why should attention weights not be presented automatically as a causal explanation?
- Which vector collection is combined using attention weights?
Further reading and review references
These are reference links, not copied exercises, institutional endorsement or proof of independent review.
AI-assisted preview; independent educational review required. Level labels describe prerequisites, not accreditation or official curriculum alignment.