StudyBrixFree worksheets, study guides and courses

Artificial intelligence · lesson 4 of 4

Cross-validation, grouped data and time-aware splits

A k-fold procedure rotates held-out folds to estimate performance for a specified training process. Splits and all fitted preprocessing must be controlled.

University8 practice questionsNotes + worked examples
Take the quizStudy guideRevision sheet

Learn the ideas

Prerequisites: Training/validation concepts and averages.

Cross-validation

A k-fold procedure rotates held-out folds to estimate performance for a specified training process. Splits and all fitted preprocessing must be controlled.

Grouped records

When multiple records share a person, device or source, the split should match whether deployment encounters new or known groups.

Time series

Forecast evaluation must respect time order. Training on future information while predicting earlier records creates a misleading task.

Nested choices

Using the same folds for extensive tuning and final performance reporting can produce optimistic estimates. Separate outer evaluation or a final untouched test can help.

Worked examples

Example 1

A 100-record dataset is divided equally into 5 folds. How many records are in one held-out fold?

100/5=20.

20

Example 2

Three fold scores are 0.7,0.8,0.9. Find their arithmetic mean.

(0.7+0.8+0.9)/3=0.8.

0.8

Check your understanding

8 questions

Decimals or fractions are accepted for numeric answers. Proofs and written responses use sample answers and require human marking. This is practice, not a protected examination.

1 A 100-record dataset is divided equally into 5 folds. How many records are in one held-out fold?

Show answer and explanation

20

100/5=20.

2 Three fold scores are 0.7,0.8,0.9. Find their arithmetic mean.

Show answer and explanation

0.8

(0.7+0.8+0.9)/3=0.8.

3 Which split is suitable when evaluating prediction on new devices?

Show answer and explanation

Keep records from each device within a single side of a split

Group-aware splits avoid sharing device-specific signals across sides.

4 For a genuine forecasting task, what should training data generally precede?

Show answer and explanation

The forecast evaluation time

Use past information to predict later outcomes.

5 Why must imputation or scaling be fitted inside each training fold?

Sample answer and marking guidance

Otherwise held-out data influence fitted preprocessing, producing leakage and an overly favourable assessment.

Credit preprocessing as part of the learned procedure.

6 What does nested cross-validation separate?

Show answer and explanation

Hyperparameter selection from outer evaluation

Inner selection and outer evaluation serve different roles.

7 A forecast cutoff is day 50. How many of days 51 through 60 are future evaluation days?

Show answer and explanation

10

Count the ten subsequent days inclusively.

8 Why should fold-score variability be reported or considered?

Sample answer and marking guidance

It can reveal sensitivity to splits, small samples or heterogeneity. A single mean hides that uncertainty and possible instability.

Credit variability and its interpretation.

Further reading and review references

These are reference links, not copied exercises, institutional endorsement or proof of independent review.