StudyBrixFree worksheets, study guides and courses

Artificial intelligence · University

Evaluating generated answers and retrieval systems

Use representative tasks with clear expected behaviour and deliberately include failure cases. Keep an evaluation set separate from prompt tuning.

Make a worksheet

Prerequisites

Metrics, annotation and reliable source checking.

Study notes

Test set

Use representative tasks with clear expected behaviour and deliberately include failure cases. Keep an evaluation set separate from prompt tuning.

Rubric

Separate factual support, completeness, instruction following and usability. A single overall score can hide distinct failure modes.

Judging

Human and model-assisted judging can both be inconsistent. Use explicit criteria, calibration examples and agreement checks; do not assume an AI judge is an oracle.

Regression

Retest after changing prompts, models, data or retrieval settings. Keep versioned results so an apparent improvement can be checked.

Worked examples

Which is a useful dimension for an evidence-grounded answer rubric?

Support measures grounding rather than appearance alone.

Whether claims are supported by the supplied evidence

A toy test has 40 tasks and 6 fail a stated requirement. Find pass rate.

34/40=0.85.

0.85

Check your understanding

  1. Why keep an evaluation set separate from prompt tuning?

    1. To eliminate all future errors
    2. To make the set inaccessible to reviewers
    3. To reduce selection overfitting
    4. To remove all hard tasks

  2. Two judges agree on 18 of 20 toy binary ratings. Find raw agreement.

  3. Why might a model judge prefer a verbose answer even when it is less accurate?

  4. Describe a regression test for a changed RAG prompt.

Answers and marking guidance

Answers and marking notes
  1. To reduce selection overfitting

    Repeated tuning to the same tests can overstate generalization.

  2. 0.9

    18/20=0.9; this is raw agreement, not a chance-corrected statistic.

  3. Its learned preferences or prompt can overweight style or detail. Explicit evidence checks and calibrated rubrics help expose such bias.

    Credit possible judge bias and mitigation.

  4. Reuse a held-out set with known source support and missing-evidence cases, compare groundedness and task success, and inspect any new failures with versions recorded.

    Credit stable comparison, edge cases and versioning.

Further reading and review references

These are reference links, not copied exercises, institutional endorsement or proof of independent review.

AI-assisted preview; independent educational review required. Level labels describe prerequisites, not accreditation or official curriculum alignment.