StudyBrixFree worksheets, study guides and courses

Artificial intelligence · Advanced university

Attention, softmax and transformer structure

Scaled dot-product attention forms weights softmax(QKᵀ/sqrt(dk)) and uses them to combine value vectors. Dimensions and masks are part of the definition.

Make a worksheet

Prerequisites

Linear algebra, probability and neural layers.

Study notes

Attention

Scaled dot-product attention forms weights softmax(QKᵀ/sqrt(dk)) and uses them to combine value vectors. Dimensions and masks are part of the definition.

Softmax

Softmax converts a finite score vector to nonnegative weights summing to one. A common additive shift leaves the result unchanged.

Position

Pure self-attention without position information is permutation-equivariant. Position representations and masks supply relevant ordering structure.

Interpretation

Attention weights are internal computational quantities. They should not automatically be treated as complete causal explanations of a model output.

Worked examples

Two attention scores are both zero. Find either softmax weight.

exp(0)/(exp(0)+exp(0))=1/2.

0.5

Attention weights are 0.25 and 0.75, and scalar values are 2 and 6. Find the weighted value.

0.25×2+0.75×6=5.

5

Check your understanding

  1. What is the purpose of a causal attention mask in an autoregressive training setup?

    1. Remove all earlier tokens
    2. Guarantee truthfulness
    3. Prevent a position from attending to future tokens
    4. Eliminate every parameter

  2. What happens to softmax when the same constant is added to every finite score?

    1. It always reverses order
    2. It becomes all zeros
    3. It stays the same
    4. It always doubles

  3. If dk=64, find the usual dot-product scaling denominator sqrt(dk).

  4. Why is positional information useful in a sequence model built from self-attention?

Answers and marking guidance

Answers and marking notes
  1. Prevent a position from attending to future tokens

    The mask respects the next-token prediction ordering.

  2. It stays the same

    A common exponential factor cancels from numerator and denominator.

  3. 8

    sqrt(64)=8.

  4. Without it, the operation does not by itself encode token order. Position information helps distinguish different sequences with the same tokens.

    Credit ordering rather than claiming attention cannot process sets.

Further reading and review references

These are reference links, not copied exercises, institutional endorsement or proof of independent review.

AI-assisted preview; independent educational review required. Level labels describe prerequisites, not accreditation or official curriculum alignment.