Session 6. A retrieval baseline — Mon 21 Sep

Quick quiz (ungraded)

Skip it and return the other five documents
A corpus that is quietly one document short produces retrieval scores that look completely normal. Nothing downstream can detect it.
Load it with an empty title
A document with no title is not a document with a blank title. The empty string travels on and fails somewhere with no connection to the cause.
Raise CorpusError naming the file and the missing header
Correct. The whole directory or nothing, and the message names the file so the fix takes seconds.

Lower the score threshold until three chunks come back
That converts a visible failure into an invisible one: the same nothing, now dressed as an answer with a score above it.
Nothing to fix here — record it as a miss and let the agent refuse
Correct. No query token appears in any chunk, so the empty list is honest, and the agent refuses before spending a model call.
Rewrite the prompt so the model handles the missing context
The model was never shown the passage. No instruction recovers a passage that never reached the prompt.

The index is stale — it was built before that file existed
Correct. The index is a value computed once, and it is right about the world it was built from. Nothing raises, which is what makes it the dangerous failure.
Its score is below the cutoff
A low score still appears when you print the scored chunks. This document appears nowhere, at any k.
The loader rejected it
Then the load would have raised CorpusError and named the file. A rejection is loud; this failure is silent.

That chunk is the closest thing the corpus has to an answer
It is about parsing failures and retries. Closeness in meaning is exactly what this retriever cannot measure.
chunk and size appear nowhere in the corpus, so the only shared token is good
Correct. rag-basics writes chunking and chunks, never chunk, and there is no stemming. One shared everyday word carries the whole score.
The score is low, so the result is already flagged
3.18 sits in the same range as scores from queries that work. The number cannot tell you it is wrong.

Did the right passage reach the prompt?
Correct. No means retrieval — chunking, index, query. Yes means generation — instructions, schema, model. One print answers it.
Is the system prompt strict enough?
That assumes generation before checking retrieval. Never prompt-engineer a retrieval failure; a better instruction cannot recover a passage the model never saw.
Would a bigger model get this right?
It would answer the same question from the same context. If the passage was missing, the bigger model invents more fluently.