Artificial intelligence · University
Evaluating generated answers and retrieval systems · revision notes
Use representative tasks with clear expected behaviour and deliberately include failure cases. Keep an evaluation set separate from prompt tuning.
Key ideas and conditions
Test set
Use representative tasks with clear expected behaviour and deliberately include failure cases. Keep an evaluation set separate from prompt tuning.
Rubric
Separate factual support, completeness, instruction following and usability. A single overall score can hide distinct failure modes.
Judging
Human and model-assisted judging can both be inconsistent. Use explicit criteria, calibration examples and agreement checks; do not assume an AI judge is an oracle.
Regression
Retest after changing prompts, models, data or retrieval settings. Keep versioned results so an apparent improvement can be checked.
Explain without looking
- Does passing a small benchmark establish universal safety or correctness?
- Old and new systems pass 30 and 34 of the same 40 cases. Find the increase in percentage points.
Further reading and review references
These are reference links, not copied exercises, institutional endorsement or proof of independent review.
AI-assisted preview; independent educational review required. Level labels describe prerequisites, not accreditation or official curriculum alignment.