# Evaluating generated answers and retrieval systems

> Use representative tasks with clear expected behaviour and deliberately include failure cases. Keep an evaluation set separate from prompt tuning.

Course: [Generative AI, RAG and practical evaluation](https://studybrix.com/en/courses/generative-ai-and-rag/index.md)
Level: university
Prerequisites: Metrics, annotation and reliable source checking.

## Notes

### Test set

Use representative tasks with clear expected behaviour and deliberately include failure cases. Keep an evaluation set separate from prompt tuning.

### Rubric

Separate factual support, completeness, instruction following and usability. A single overall score can hide distinct failure modes.

### Judging

Human and model-assisted judging can both be inconsistent. Use explicit criteria, calibration examples and agreement checks; do not assume an AI judge is an oracle.

### Regression

Retest after changing prompts, models, data or retrieval settings. Keep versioned results so an apparent improvement can be checked.

## Practice questions

### 1. Which is a useful dimension for an evidence-grounded answer rubric?

A. Only word count
B. Whether claims are supported by the supplied evidence
C. Only confident tone
D. Only formatting colour

Answer: Whether claims are supported by the supplied evidence

Explanation: Support measures grounding rather than appearance alone.

### 2. A toy test has 40 tasks and 6 fail a stated requirement. Find pass rate.

Answer: 0.85

Explanation: 34/40=0.85.

### 3. Why keep an evaluation set separate from prompt tuning?

A. To eliminate all future errors
B. To make the set inaccessible to reviewers
C. To reduce selection overfitting
D. To remove all hard tasks

Answer: To reduce selection overfitting

Explanation: Repeated tuning to the same tests can overstate generalization.

### 4. Two judges agree on 18 of 20 toy binary ratings. Find raw agreement.

Answer: 0.9

Explanation: 18/20=0.9; this is raw agreement, not a chance-corrected statistic.

### 5. Why might a model judge prefer a verbose answer even when it is less accurate?

Sample answer / marking guidance: Its learned preferences or prompt can overweight style or detail. Explicit evidence checks and calibrated rubrics help expose such bias.

Explanation: Credit possible judge bias and mitigation.

### 6. Describe a regression test for a changed RAG prompt.

Sample answer / marking guidance: Reuse a held-out set with known source support and missing-evidence cases, compare groundedness and task success, and inspect any new failures with versions recorded.

Explanation: Credit stable comparison, edge cases and versioning.

### 7. Does passing a small benchmark establish universal safety or correctness?

A. Yes, universally
B. No
C. Only when all answers are long
D. Only when scores are percentages

Answer: No

Explanation: A finite benchmark covers a limited set of conditions.

### 8. Old and new systems pass 30 and 34 of the same 40 cases. Find the increase in percentage points.

Answer: 10 percentage points

Explanation: 75% to 85% is a 10-percentage-point increase.

## Further reading

- https://developers.google.com/machine-learning/crash-course
- https://cs229.stanford.edu/

Original AI-assisted educational draft; not independently teacher-reviewed. References support further study, not copied exercises, affiliation or curriculum accreditation.