# Overfitting, regularization and data distribution

> A model can fit training-specific noise or patterns that do not generalize. Compare training and appropriately held-out performance.

Course: [Mathematics of machine learning](https://studybrix.com/en/courses/machine-learning-mathematics/index.md)
Level: university
Prerequisites: Training losses, validation and algebra.

## Notes

### Overfitting

A model can fit training-specific noise or patterns that do not generalize. Compare training and appropriately held-out performance.

### Regularization

Penalties or constraints can prefer simpler parameter solutions. Their effect depends on the model, scaling and chosen strength.

### Bias-variance

Model flexibility, sample size and noise influence prediction error. The tradeoff is a framework, not a universal curve guaranteed in all modern settings.

### Distribution shift

New data may differ from training data. Validation should reflect the intended use, and monitoring may reveal later drift.

## Practice questions

### 1. Which pattern often suggests overfitting?

A. A white interface
B. A short model name
C. Equal file sizes
D. Very low training error but much worse held-out error

Answer: Very low training error but much worse held-out error

Explanation: A generalization gap can indicate training-specific fit, though other problems should also be checked.

### 2. For w=(3,4), calculate squared L2 norm.

Answer: 25

Explanation: 3²+4²=25.

### 3. If data loss is 2 and λ times squared L2 norm is 0.5, find the penalized objective.

Answer: 2.5

Explanation: Add loss and penalty.

### 4. Where should regularization strength normally be selected?

A. Through a suitable validation process
B. By file extension
C. By repeatedly inspecting final test labels
D. By training accuracy alone in every case

Answer: Through a suitable validation process

Explanation: Regularization is a hyperparameter choice.

### 5. Why can feature scaling affect an L2 penalty's practical meaning?

Sample answer / marking guidance: The same prediction can require differently sized coefficients when units change. A coefficient-size penalty therefore interacts with feature scales.

Explanation: Credit units and coefficient magnitudes.

### 6. Which is an example of distribution shift?

A. Deployment inputs differ systematically from training inputs
B. Changing only a chart title
C. Renaming the same unchanged file
D. Keeping exactly the same data assumptions

Answer: Deployment inputs differ systematically from training inputs

Explanation: Shift concerns the data-generating setting.

### 7. Why is collecting more data not a guaranteed fix for incorrect labels or leakage?

Sample answer / marking guidance: It can reproduce the same systematic defects at larger scale. The measurement or evaluation process must be corrected.

Explanation: Credit systematic error versus sample size.

### 8. Training error is 0.05 and validation error is 0.12 in the same metric. Find validation minus training error.

Answer: 0.07

Explanation: 0.12−0.05=0.07.

## Further reading

- https://developers.google.com/machine-learning/crash-course
- https://cs229.stanford.edu/

Original AI-assisted educational draft; not independently teacher-reviewed. References support further study, not copied exercises, affiliation or curriculum accreditation.