StudyBrixFree worksheets, study guides and courses

Artificial intelligence · lesson 3 of 4

PCA, variance and dimensionality reduction

Principal component analysis finds orthogonal directions of high variance in centred data under its standard formulation.

University8 practice questionsNotes + worked examples
Take the quizStudy guideRevision sheet

Learn the ideas

Prerequisites: Linear algebra, covariance and orthogonal projections.

Purpose

Principal component analysis finds orthogonal directions of high variance in centred data under its standard formulation.

Computation

Principal directions can be obtained from covariance eigenvectors or a singular-value decomposition of centred data.

Variance

Explained-variance ratios summarize variance captured, not predictive accuracy or importance to every task.

Scale

PCA is sensitive to units and scaling. High variance can reflect nuisance variation, and low-variance directions can still be useful for prediction.

Worked examples

Example 1

Covariance eigenvalues are 6,3,1. Find the explained-variance ratio of the first component.

6/(6+3+1)=0.6.

0.6

Example 2

For eigenvalues 6,3,1, find the first two components' cumulative ratio.

(6+3)/10=0.9.

0.9

Check your understanding

8 questions

Decimals or fractions are accepted for numeric answers. Proofs and written responses use sample answers and require human marking. This is practice, not a protected examination.

1 Covariance eigenvalues are 6,3,1. Find the explained-variance ratio of the first component.

Show answer and explanation

0.6

6/(6+3+1)=0.6.

2 For eigenvalues 6,3,1, find the first two components' cumulative ratio.

Show answer and explanation

0.9

(6+3)/10=0.9.

3 What does standard PCA do before computing covariance-based directions?

Show answer and explanation

Centre each feature

Subtracting feature means is part of the standard centred formulation.

4 Are explained-variance ratios identical to classification accuracy?

Show answer and explanation

No

They measure retained variance, not supervised predictive success.

5 A feature has values 2,4,6. Find the centred value corresponding to 6.

Show answer and explanation

2

The mean is 4, so 6−4=2.

6 Why can an important predictive signal lie in a low-variance direction?

Sample answer and marking guidance

Class separation can depend on a subtle feature even when other unrelated variation is much larger. PCA does not use labels in its basic form.

Credit unsupervised variance versus predictive relevance.

7 How are distinct principal directions chosen in the standard Euclidean formulation?

Show answer and explanation

Orthogonally

PCA components form orthogonal directions under the standard metric.

8 Why should PCA be fit on training data rather than the combined train and test data in a strict evaluation?

Sample answer and marking guidance

Means, scales and directions are fitted preprocessing parameters. Using test inputs can leak information about the evaluation distribution into training.

Credit fitted preprocessing and split discipline.

Further reading and review references

These are reference links, not copied exercises, institutional endorsement or proof of independent review.