StudyBrixFree worksheets, study guides and courses

Artificial intelligence · University

Overfitting, regularization and data distribution

A model can fit training-specific noise or patterns that do not generalize. Compare training and appropriately held-out performance.

Make a worksheet

Prerequisites

Training losses, validation and algebra.

Study notes

Overfitting

A model can fit training-specific noise or patterns that do not generalize. Compare training and appropriately held-out performance.

Regularization

Penalties or constraints can prefer simpler parameter solutions. Their effect depends on the model, scaling and chosen strength.

Bias-variance

Model flexibility, sample size and noise influence prediction error. The tradeoff is a framework, not a universal curve guaranteed in all modern settings.

Distribution shift

New data may differ from training data. Validation should reflect the intended use, and monitoring may reveal later drift.

Worked examples

Which pattern often suggests overfitting?

A generalization gap can indicate training-specific fit, though other problems should also be checked.

Very low training error but much worse held-out error

For w=(3,4), calculate squared L2 norm.

3²+4²=25.

25

Check your understanding

  1. If data loss is 2 and λ times squared L2 norm is 0.5, find the penalized objective.

  2. Where should regularization strength normally be selected?

    1. Through a suitable validation process
    2. By file extension
    3. By repeatedly inspecting final test labels
    4. By training accuracy alone in every case

  3. Why can feature scaling affect an L2 penalty's practical meaning?

  4. Which is an example of distribution shift?

    1. Deployment inputs differ systematically from training inputs
    2. Changing only a chart title
    3. Renaming the same unchanged file
    4. Keeping exactly the same data assumptions

Answers and marking guidance

Answers and marking notes
  1. 2.5

    Add loss and penalty.

  2. Through a suitable validation process

    Regularization is a hyperparameter choice.

  3. The same prediction can require differently sized coefficients when units change. A coefficient-size penalty therefore interacts with feature scales.

    Credit units and coefficient magnitudes.

  4. Deployment inputs differ systematically from training inputs

    Shift concerns the data-generating setting.

Further reading and review references

These are reference links, not copied exercises, institutional endorsement or proof of independent review.

AI-assisted preview; independent educational review required. Level labels describe prerequisites, not accreditation or official curriculum alignment.