# Orthogonality and least-squares fitting

> The closest point in a subspace is found by an orthogonal residual.

Course: [Linear algebra: spaces, eigenvectors and fitting](https://studybrix.com/en/courses/linear-algebra/index.md)
Level: university
Prerequisites: Dot products, matrices and Euclidean norms.

## Notes

### Projection

The closest point in a subspace is found by an orthogonal residual.

### Normal equations

AᵀAx=Aᵀb characterizes least-squares solutions.

### Uniqueness

Full column rank makes AᵀA invertible; without it, fitted values may be unique while coefficients are not.

### Computation

QR factorization avoids explicitly forming AᵀA and often offers better numerical stability.

## Practice questions

### 1. Project v=(3,4) onto the x-axis. Enter the x-coordinate.

Answer: 3

Explanation: The closest point is (3,0), leaving residual (0,4) perpendicular to the axis.

### 2. Fit a constant c by minimizing (1−c)²+(3−c)²+(8−c)². Find c.

Answer: 4

Explanation: The minimizer is the sample mean (1+3+8)/3=4.

### 3. At a least-squares solution, the residual b−Ax is orthogonal to what?

A. Only the zero vector
B. The row space in all dimensions
C. Every possible b
D. The column space of A

Answer: The column space of A

Explanation: Aᵀ(b−Ax)=0 means the residual is orthogonal to every column.

### 4. Find the scalar k such that the projection of (2,1) onto span(1,1) equals k(1,1).

Answer: 1.5

Explanation: k=((2,1)·(1,1))/2=3/2.

### 5. Derive the normal equations by differentiating the squared residual norm.

Sample answer / marking guidance: The gradient of (Ax−b)ᵀ(Ax−b) is 2Aᵀ(Ax−b). Setting it to zero gives AᵀAx=Aᵀb.

Explanation: Credit expansion or a correct vector derivative and stationary condition.

### 6. Which condition guarantees a unique least-squares coefficient vector?

A. A has a zero column
B. b has only positive entries
C. A has full column rank
D. There are more columns than rows

Answer: A has full column rank

Explanation: Full column rank makes the quadratic strictly convex in x.

### 7. For data with Sxx=20 and Sxy=−30, find the least-squares slope with an intercept.

Answer: -1.5

Explanation: The slope is Sxy/Sxx=−30/20.

### 8. Why is a small least-squares residual not by itself proof of a causal relationship?

Sample answer / marking guidance: It measures fit within the chosen model, not identification of a causal mechanism. Confounding, sampling bias and model selection may still explain the association.

Explanation: Credit separation of predictive fit from causality and a concrete limitation.

## Further reading

- https://ocw.mit.edu/courses/18-100a-real-analysis-fall-2020/
- https://ocw.mit.edu/courses/18-06-linear-algebra-spring-2010/

Original AI-assisted educational draft; not independently teacher-reviewed. References support further study, not copied exercises, affiliation or curriculum accreditation.