StudyBrixFree worksheets, study guides and courses

Artificial intelligence · lesson 3 of 4

Evaluating generated answers and retrieval systems

Use representative tasks with clear expected behaviour and deliberately include failure cases. Keep an evaluation set separate from prompt tuning.

University8 practice questionsNotes + worked examples
Take the quizStudy guideRevision sheet

Learn the ideas

Prerequisites: Metrics, annotation and reliable source checking.

Test set

Use representative tasks with clear expected behaviour and deliberately include failure cases. Keep an evaluation set separate from prompt tuning.

Rubric

Separate factual support, completeness, instruction following and usability. A single overall score can hide distinct failure modes.

Judging

Human and model-assisted judging can both be inconsistent. Use explicit criteria, calibration examples and agreement checks; do not assume an AI judge is an oracle.

Regression

Retest after changing prompts, models, data or retrieval settings. Keep versioned results so an apparent improvement can be checked.

Worked examples

Example 1

Which is a useful dimension for an evidence-grounded answer rubric?

Support measures grounding rather than appearance alone.

Whether claims are supported by the supplied evidence

Example 2

A toy test has 40 tasks and 6 fail a stated requirement. Find pass rate.

34/40=0.85.

0.85

Check your understanding

8 questions

Decimals or fractions are accepted for numeric answers. Proofs and written responses use sample answers and require human marking. This is practice, not a protected examination.

1 Which is a useful dimension for an evidence-grounded answer rubric?

Show answer and explanation

Whether claims are supported by the supplied evidence

Support measures grounding rather than appearance alone.

2 A toy test has 40 tasks and 6 fail a stated requirement. Find pass rate.

Show answer and explanation

0.85

34/40=0.85.

3 Why keep an evaluation set separate from prompt tuning?

Show answer and explanation

To reduce selection overfitting

Repeated tuning to the same tests can overstate generalization.

4 Two judges agree on 18 of 20 toy binary ratings. Find raw agreement.

Show answer and explanation

0.9

18/20=0.9; this is raw agreement, not a chance-corrected statistic.

5 Why might a model judge prefer a verbose answer even when it is less accurate?

Sample answer and marking guidance

Its learned preferences or prompt can overweight style or detail. Explicit evidence checks and calibrated rubrics help expose such bias.

Credit possible judge bias and mitigation.

6 Describe a regression test for a changed RAG prompt.

Sample answer and marking guidance

Reuse a held-out set with known source support and missing-evidence cases, compare groundedness and task success, and inspect any new failures with versions recorded.

Credit stable comparison, edge cases and versioning.

7 Does passing a small benchmark establish universal safety or correctness?

Show answer and explanation

No

A finite benchmark covers a limited set of conditions.

8 Old and new systems pass 30 and 34 of the same 40 cases. Find the increase in percentage points.percentage points

Show answer and explanation

10 percentage points

75% to 85% is a 10-percentage-point increase.

Further reading and review references

These are reference links, not copied exercises, institutional endorsement or proof of independent review.