# Training, validation and honest testing > The model fits its parameters on training data. Success there does not alone demonstrate generalization to new cases. Course: [AI foundations and responsible use](https://studybrix.com/en/courses/ai-foundations/index.md) Level: school Prerequisites: Fractions, tables and basic prediction tasks. ## Notes ### Training The model fits its parameters on training data. Success there does not alone demonstrate generalization to new cases. ### Validation Validation data supports choices such as model type, thresholds and hyperparameters. Repeated choices can overfit the validation process. ### Testing A held-out test set assesses the selected procedure. Do not repeatedly tune against it while continuing to call it untouched. ### Metrics Choose metrics that reflect important mistakes. Accuracy alone can hide failures on rare classes or specific groups. ## Practice questions ### 1. Which dataset primarily fits model parameters? A. A blank table B. Training data C. A random website title D. An untouched final test set only Answer: Training data Explanation: Parameter fitting uses the training examples. ### 2. What is a usual purpose of validation data? A. Secretly add test labels to features B. Replace all training data with metadata C. Guarantee no future failures D. Choose settings and compare candidate models Answer: Choose settings and compare candidate models Explanation: Validation supports model selection, not independent final assessment after repeated tuning. ### 3. A classifier gets 90 of 120 test examples correct. Find accuracy as a fraction. Answer: 0.75 Explanation: 90/120=0.75. ### 4. What happens if test performance repeatedly guides tuning? A. The test is no longer an untouched evaluation B. Its size doubles automatically C. The test gets more independent D. All leakage disappears Answer: The test is no longer an untouched evaluation Explanation: Decisions based on test results leak information into selection. ### 5. Why can 99% accuracy be unhelpful when only 1% of cases are positive? Sample answer / marking guidance: Predicting every case as negative would get 99% accuracy while missing all positives. Additional metrics and error costs matter. Explanation: Credit the majority-class baseline and missed positives. ### 6. A baseline makes 20 errors and a new model 15 on the same 100 cases. How many fewer errors are observed? Answer: 5 Explanation: 20−15=5; statistical confidence requires further analysis. ### 7. Why does one good test result not guarantee performance in every future setting? Sample answer / marking guidance: Data distribution, users and conditions may change. A test samples a particular setting with finite uncertainty. Explanation: Credit domain shift and finite sample scope. ### 8. Which is appropriate when reporting a model result? A. Omit label definitions B. State dataset, split and metric C. Report only the best number after hiding failures D. Claim universal accuracy Answer: State dataset, split and metric Explanation: Evaluation needs reproducible context. ## Further reading - https://developers.google.com/machine-learning/crash-course - https://cs229.stanford.edu/ Original AI-assisted educational draft; not independently teacher-reviewed. References support further study, not copied exercises, affiliation or curriculum accreditation.