Artificial intelligence · University

PCA, variance and dimensionality reduction

Principal component analysis finds orthogonal directions of high variance in centred data under its standard formulation.

Prerequisites

Linear algebra, covariance and orthogonal projections.

Principal component analysis finds orthogonal directions of high variance in centred data under its standard formulation.

Principal directions can be obtained from covariance eigenvectors or a singular-value decomposition of centred data.

Explained-variance ratios summarize variance captured, not predictive accuracy or importance to every task.

PCA is sensitive to units and scaling. High variance can reflect nuisance variation, and low-variance directions can still be useful for prediction.

6/(6+3+1)=0.6.

0.6

(6+3)/10=0.9.

0.9

What does standard PCA do before computing covariance-based directions?
1. Always convert all values to zero
2. Use test labels to orient axes
3. Centre each feature
4. Sort features alphabetically as the only step
Are explained-variance ratios identical to classification accuracy?
1. Only if ratios sum to one
2. Yes, exactly
3. Only for every two-dimensional dataset
4. No
A feature has values 2,4,6. Find the centred value corresponding to 6.
Why can an important predictive signal lie in a low-variance direction?

Answers and marking notes

Centre each feature
Subtracting feature means is part of the standard centred formulation.
No
They measure retained variance, not supervised predictive success.
2
The mean is 4, so 6−4=2.
Class separation can depend on a subtle feature even when other unrelated variation is much larger. PCA does not use labels in its basic form.
Credit unsupervised variance versus predictive relevance.