StudyBrixFree worksheets, study guides and courses

Mathematics · University

Orthogonality and least-squares fitting

The closest point in a subspace is found by an orthogonal residual.

Make a worksheet

Prerequisites

Dot products, matrices and Euclidean norms.

Study notes

Projection

The closest point in a subspace is found by an orthogonal residual.

Normal equations

AᵀAx=Aᵀb characterizes least-squares solutions.

Uniqueness

Full column rank makes AᵀA invertible; without it, fitted values may be unique while coefficients are not.

Computation

QR factorization avoids explicitly forming AᵀA and often offers better numerical stability.

Worked examples

Project v=(3,4) onto the x-axis. Enter the x-coordinate.

The closest point is (3,0), leaving residual (0,4) perpendicular to the axis.

3

Fit a constant c by minimizing (1−c)²+(3−c)²+(8−c)². Find c.

The minimizer is the sample mean (1+3+8)/3=4.

4

Check your understanding

  1. At a least-squares solution, the residual b−Ax is orthogonal to what?

    1. Only the zero vector
    2. The row space in all dimensions
    3. Every possible b
    4. The column space of A

  2. Find the scalar k such that the projection of (2,1) onto span(1,1) equals k(1,1).

  3. Derive the normal equations by differentiating the squared residual norm.

  4. Which condition guarantees a unique least-squares coefficient vector?

    1. A has a zero column
    2. b has only positive entries
    3. A has full column rank
    4. There are more columns than rows

Answers and marking guidance

Answers and marking notes
  1. The column space of A

    Aᵀ(b−Ax)=0 means the residual is orthogonal to every column.

  2. 1.5

    k=((2,1)·(1,1))/2=3/2.

  3. The gradient of (Ax−b)ᵀ(Ax−b) is 2Aᵀ(Ax−b). Setting it to zero gives AᵀAx=Aᵀb.

    Credit expansion or a correct vector derivative and stationary condition.

  4. A has full column rank

    Full column rank makes the quadratic strictly convex in x.

Further reading and review references

These are reference links, not copied exercises, institutional endorsement or proof of independent review.

AI-assisted preview; independent educational review required. Level labels describe prerequisites, not accreditation or official curriculum alignment.