StudyBrixFree worksheets, study guides and courses

Artificial intelligence · University

Cross-validation, grouped data and time-aware splits

A k-fold procedure rotates held-out folds to estimate performance for a specified training process. Splits and all fitted preprocessing must be controlled.

Make a worksheet

Prerequisites

Training/validation concepts and averages.

Study notes

Cross-validation

A k-fold procedure rotates held-out folds to estimate performance for a specified training process. Splits and all fitted preprocessing must be controlled.

Grouped records

When multiple records share a person, device or source, the split should match whether deployment encounters new or known groups.

Time series

Forecast evaluation must respect time order. Training on future information while predicting earlier records creates a misleading task.

Nested choices

Using the same folds for extensive tuning and final performance reporting can produce optimistic estimates. Separate outer evaluation or a final untouched test can help.

Worked examples

A 100-record dataset is divided equally into 5 folds. How many records are in one held-out fold?

100/5=20.

20

Three fold scores are 0.7,0.8,0.9. Find their arithmetic mean.

(0.7+0.8+0.9)/3=0.8.

0.8

Check your understanding

  1. Which split is suitable when evaluating prediction on new devices?

    1. Keep records from each device within a single side of a split
    2. Randomly mix near-duplicate device records everywhere
    3. Always put labels into inputs
    4. Split only by filename length

  2. For a genuine forecasting task, what should training data generally precede?

    1. No date restriction ever
    2. The alphabet's last letter
    3. The forecast evaluation time
    4. Only the output filename

  3. Why must imputation or scaling be fitted inside each training fold?

  4. What does nested cross-validation separate?

    1. Hyperparameter selection from outer evaluation
    2. Model names from file names
    3. All data from every label permanently
    4. Every numeric feature from computation

Answers and marking guidance

Answers and marking notes
  1. Keep records from each device within a single side of a split

    Group-aware splits avoid sharing device-specific signals across sides.

  2. The forecast evaluation time

    Use past information to predict later outcomes.

  3. Otherwise held-out data influence fitted preprocessing, producing leakage and an overly favourable assessment.

    Credit preprocessing as part of the learned procedure.

  4. Hyperparameter selection from outer evaluation

    Inner selection and outer evaluation serve different roles.

Further reading and review references

These are reference links, not copied exercises, institutional endorsement or proof of independent review.

AI-assisted preview; independent educational review required. Level labels describe prerequisites, not accreditation or official curriculum alignment.