# Attention, softmax and transformer structure > Scaled dot-product attention forms weights softmax(QKᵀ/sqrt(dk)) and uses them to combine value vectors. Dimensions and masks are part of the definition. Course: [Neural networks, attention and embeddings](https://studybrix.com/en/courses/deep-learning/index.md) Level: advanced Prerequisites: Linear algebra, probability and neural layers. ## Notes ### Attention Scaled dot-product attention forms weights softmax(QKᵀ/sqrt(dk)) and uses them to combine value vectors. Dimensions and masks are part of the definition. ### Softmax Softmax converts a finite score vector to nonnegative weights summing to one. A common additive shift leaves the result unchanged. ### Position Pure self-attention without position information is permutation-equivariant. Position representations and masks supply relevant ordering structure. ### Interpretation Attention weights are internal computational quantities. They should not automatically be treated as complete causal explanations of a model output. ## Practice questions ### 1. Two attention scores are both zero. Find either softmax weight. Answer: 0.5 Explanation: exp(0)/(exp(0)+exp(0))=1/2. ### 2. Attention weights are 0.25 and 0.75, and scalar values are 2 and 6. Find the weighted value. Answer: 5 Explanation: 0.25×2+0.75×6=5. ### 3. What is the purpose of a causal attention mask in an autoregressive training setup? A. Remove all earlier tokens B. Guarantee truthfulness C. Prevent a position from attending to future tokens D. Eliminate every parameter Answer: Prevent a position from attending to future tokens Explanation: The mask respects the next-token prediction ordering. ### 4. What happens to softmax when the same constant is added to every finite score? A. It always reverses order B. It becomes all zeros C. It stays the same D. It always doubles Answer: It stays the same Explanation: A common exponential factor cancels from numerator and denominator. ### 5. If dk=64, find the usual dot-product scaling denominator sqrt(dk). Answer: 8 Explanation: sqrt(64)=8. ### 6. Why is positional information useful in a sequence model built from self-attention? Sample answer / marking guidance: Without it, the operation does not by itself encode token order. Position information helps distinguish different sequences with the same tokens. Explanation: Credit ordering rather than claiming attention cannot process sets. ### 7. Why should attention weights not be presented automatically as a causal explanation? Sample answer / marking guidance: Other layers, value representations and interactions also shape the result. Causal explanations require appropriate interventions or analysis beyond the weight display. Explanation: Credit computational context and evidence. ### 8. Which vector collection is combined using attention weights? A. Only filenames B. Only class labels in every use C. No vectors D. Value vectors Answer: Value vectors Explanation: Weights aggregate the V representations in scaled dot-product attention. ## Further reading - https://developers.google.com/machine-learning/crash-course - https://cs229.stanford.edu/ Original AI-assisted educational draft; not independently teacher-reviewed. References support further study, not copied exercises, affiliation or curriculum accreditation.