StudyBrixFree worksheets, study guides and courses

Artificial intelligence · lesson 3 of 4

Training, validation and honest testing

The model fits its parameters on training data. Success there does not alone demonstrate generalization to new cases.

Grades 9–128 practice questionsNotes + worked examples
Take the quizStudy guideRevision sheet

Learn the ideas

Prerequisites: Fractions, tables and basic prediction tasks.

Training

The model fits its parameters on training data. Success there does not alone demonstrate generalization to new cases.

Validation

Validation data supports choices such as model type, thresholds and hyperparameters. Repeated choices can overfit the validation process.

Testing

A held-out test set assesses the selected procedure. Do not repeatedly tune against it while continuing to call it untouched.

Metrics

Choose metrics that reflect important mistakes. Accuracy alone can hide failures on rare classes or specific groups.

Worked examples

Example 1

Which dataset primarily fits model parameters?

Parameter fitting uses the training examples.

Training data

Example 2

What is a usual purpose of validation data?

Validation supports model selection, not independent final assessment after repeated tuning.

Choose settings and compare candidate models

Check your understanding

8 questions

Decimals or fractions are accepted for numeric answers. Proofs and written responses use sample answers and require human marking. This is practice, not a protected examination.

1 Which dataset primarily fits model parameters?

Show answer and explanation

Training data

Parameter fitting uses the training examples.

2 What is a usual purpose of validation data?

Show answer and explanation

Choose settings and compare candidate models

Validation supports model selection, not independent final assessment after repeated tuning.

3 A classifier gets 90 of 120 test examples correct. Find accuracy as a fraction.

Show answer and explanation

0.75

90/120=0.75.

4 What happens if test performance repeatedly guides tuning?

Show answer and explanation

The test is no longer an untouched evaluation

Decisions based on test results leak information into selection.

5 Why can 99% accuracy be unhelpful when only 1% of cases are positive?

Sample answer and marking guidance

Predicting every case as negative would get 99% accuracy while missing all positives. Additional metrics and error costs matter.

Credit the majority-class baseline and missed positives.

6 A baseline makes 20 errors and a new model 15 on the same 100 cases. How many fewer errors are observed?

Show answer and explanation

5

20−15=5; statistical confidence requires further analysis.

7 Why does one good test result not guarantee performance in every future setting?

Sample answer and marking guidance

Data distribution, users and conditions may change. A test samples a particular setting with finite uncertainty.

Credit domain shift and finite sample scope.

8 Which is appropriate when reporting a model result?

Show answer and explanation

State dataset, split and metric

Evaluation needs reproducible context.

Further reading and review references

These are reference links, not copied exercises, institutional endorsement or proof of independent review.