Session 7. Retrieval and grounding metrics — Tue 22 Sep

Quick quiz (ungraded)

Hit rate is the more accurate metric
It is the coarser one. Hit rate @3 says the document was somewhere in the top three, not that it ranked first or that the other two were useful.
It needs one labeled doc id per query, so the set exists by lunchtime
Correct. Recall and precision need every relevant passage judged for every query; hit rate needs one pair, and five cases beat zero cases by infinity.
Recall and precision do not apply to retrieval
They are the standard retrieval pair. They are just expensive to label, which is the whole reason hit rate is the day-one metric.

100%, because that is the measured number
It is measured on the five cases the rule was written to fix. On four probe queries it never saw, hit rate @1 fell from 75% to 0%.
Both: 80% to 100% on the tuning set, and 75% to 0% at k=1 on queries it never saw
Correct. One change, measured twice — once where it was designed to help and once where it was not.
100% with a note that results may vary
That is a mood, not a regression. A regression sentence names what fires, on what, and the number it moved.

None; it earned all four
Three of the eight cases retrieve nothing, so the agent refuses before any model call. The fake's answer never ran on them.
The three refusal cases — retrieval was empty, so the model was never asked
Correct. Those cases measure the retriever and the refusal path. Split refusal from grounded and the fake scores 1/5 on the half that is about grounding.
The four grounded failures
Those are failures, not passes. They fail because the expected doc id is missing from the citations.

Nothing; the citation is genuinely correct
The citation is correct and the sentence is supported by nothing. A correct doc id next to an empty claim is exactly the false positive to hunt.
It asks which document was cited, never whether the answer follows from it
Correct. That is claim support, and it is missing. The capstone grader adds it as the claim_support gate.
It should have rejected the citation as fabricated
It is not fabricated. Retrieval did return rag-basics for that question, so the agent's fabricated-citation strip has nothing to remove.

Record the result and move on
A perfect score is evidence about the system and about the evaluator at the same time, and you cannot yet tell which.
Add more cases until something fails
Useful later, and it does not answer the question. More cases graded by the same weak pass condition inherit the same blind spot.
Build something that deserves zero, run it, and see what it scores
Correct. Until you have attacked the evaluator on purpose, the pass rate is a guess about what your pass rate means.