StudyBrixFree worksheets, study guides and courses

Mathematics · lesson 3 of 3

Orthogonality and least-squares fitting

The closest point in a subspace is found by an orthogonal residual.

University8 practice questionsNotes + worked examples
Take the quizStudy guideRevision sheet

Learn the ideas

Prerequisites: Dot products, matrices and Euclidean norms.

Projection

The closest point in a subspace is found by an orthogonal residual.

Normal equations

AᵀAx=Aᵀb characterizes least-squares solutions.

Uniqueness

Full column rank makes AᵀA invertible; without it, fitted values may be unique while coefficients are not.

Computation

QR factorization avoids explicitly forming AᵀA and often offers better numerical stability.

Worked examples

Example 1

Project v=(3,4) onto the x-axis. Enter the x-coordinate.

The closest point is (3,0), leaving residual (0,4) perpendicular to the axis.

3

Example 2

Fit a constant c by minimizing (1−c)²+(3−c)²+(8−c)². Find c.

The minimizer is the sample mean (1+3+8)/3=4.

4

Check your understanding

8 questions

Decimals or fractions are accepted for numeric answers. Proofs and written responses use sample answers and require human marking. This is practice, not a protected examination.

1 Project v=(3,4) onto the x-axis. Enter the x-coordinate.

Show answer and explanation

3

The closest point is (3,0), leaving residual (0,4) perpendicular to the axis.

2 Fit a constant c by minimizing (1−c)²+(3−c)²+(8−c)². Find c.

Show answer and explanation

4

The minimizer is the sample mean (1+3+8)/3=4.

3 At a least-squares solution, the residual b−Ax is orthogonal to what?

Show answer and explanation

The column space of A

Aᵀ(b−Ax)=0 means the residual is orthogonal to every column.

4 Find the scalar k such that the projection of (2,1) onto span(1,1) equals k(1,1).

Show answer and explanation

1.5

k=((2,1)·(1,1))/2=3/2.

5 Derive the normal equations by differentiating the squared residual norm.

Sample answer and marking guidance

The gradient of (Ax−b)ᵀ(Ax−b) is 2Aᵀ(Ax−b). Setting it to zero gives AᵀAx=Aᵀb.

Credit expansion or a correct vector derivative and stationary condition.

6 Which condition guarantees a unique least-squares coefficient vector?

Show answer and explanation

A has full column rank

Full column rank makes the quadratic strictly convex in x.

7 For data with Sxx=20 and Sxy=−30, find the least-squares slope with an intercept.

Show answer and explanation

-1.5

The slope is Sxy/Sxx=−30/20.

8 Why is a small least-squares residual not by itself proof of a causal relationship?

Sample answer and marking guidance

It measures fit within the chosen model, not identification of a causal mechanism. Confounding, sampling bias and model selection may still explain the association.

Credit separation of predictive fit from causality and a concrete limitation.

Further reading and review references

These are reference links, not copied exercises, institutional endorsement or proof of independent review.