Session 9. Trace and evaluate an agent — Thu 24 Sep

Quick quiz (ungraded)

retrieval — five grounded cases failed, so retrieval is the suspect
The report alone leaves both buckets open. The trace closes one of them: the passage reached the prompt.
instruction_following — the context arrived and the model ignored it
Correct. Retrieval did its job, one llm_call happened, and the answer cited nothing. The fix lives in the prompt or the model, not the index.
formatting — the reply must have broken the schema
A schema break shows up as a parse failure and a retry in the trace. There is neither here.

Nothing; the safest thing to do with a secret is remove it
The secret is gone either way. What is also gone is the fact that there was one.
A reader can no longer tell a removed credential from one that was never there
Correct. Those two facts lead to different next steps, and a missing field cannot express either. Replace with a placeholder that says a value was removed.
Only readability; the eval still passes
The eval never looked at the log. The cost lands on the person reading it during an incident.

Nothing — the placeholder is consistent, which is what consistency means
Consistent across different secrets is exactly the problem: it makes them indistinguishable.
That attempt 3 used a different key from attempts 1 and 2
Correct. Derive the placeholder from the secret — a short digest — and the log shows two distinct values without printing either.
The status codes, which the redactor overwrote
The status codes survive; a redactor replaces the secret and keeps every byte around it.

The trace is incomplete, so the next reader will treat it with caution
There is nothing in the file that says it is incomplete. That is the danger.
The trace reads as a run where no model call happened, and sends the reader to the wrong bucket
Correct. A gap does not look like a gap; it looks like a different run. Hold the trace to a shape — at least one retrieve, exactly one decision — so a broken logger is detectable.
The evaluation will fail, which surfaces the problem
The evaluation reads the answer, not the trace. It stays green while the log tells a false story.

A code check; deterministic beats probabilistic every time
Prefer code where code can express the property. Tone is not one a regex can express.
A model as judge, spot-checked against human labels on a sample
Correct. Reach for a judge only where code cannot express the property, and measure the judge itself — an unchecked judge is an instrument with an error bar nobody reports.
A model as judge, and trust it — it is the same class of model that wrote the answer
That is the reason for suspicion, not for trust. A judge shares the generator's biases and will forgive its own failure modes.