Bonus 6. Improve the coach

Improve the coach

Optional, never counted, and free. Not required for completion or a certificate, and excluded from every total. It runs offline on your own machine: no key, no account, no billing. What it ends in is not a submission but a pull request against a repository this course actually runs.

Outcome

Make the coach better for everyone in the cohort, prove it with a number, and open a pull request.

You do not need to know how retrieval works to start. The smallest contribution takes ten minutes and no code: find a question the coach gets wrong, and write it down.

Where the coach lives

There are two copies, and each has a job:

	Where	What you do there
In your notebooks	`bootcamp_agent.coach`	use it: `coach("your question")`
Its own repository	gecko-ai-coach	improve it: change, measure, open a pull request

Both rank pages the same way. Contributions go to gecko-ai-coach. The course repository is published fresh every week, so a pull request against your clone of it has nowhere to land.

Before you change anything

Run demo 5 — the coach, up close. In ten minutes you ask the coach a question, see why one page beat another, watch it refuse, and find a question it gets wrong and the reason why. That reason is where every improvement starts.

Pick your size

Each step is a complete contribution. Start at the top.

Step	Time	Code?	What you hand in
1 — Add a question the coach gets wrong	10 min	no	one file, made by one command
2 — Explain a miss	30 min	no	an issue, or a corrected label
Translate the README	30 min	no	a reviewed Portuguese or Spanish draft, or a new language
3 — Change one knob, and measure it	1–2 h	a few lines	before and after, on named sets

Step 1 is one command, and it tells you whether your question is really a miss before it saves anything:

uv run ai-coach propose "how do I hand in a session" --page unit0/how-to-submit \
  --pages ../dev3pack-cohort-2026-09/units/en --write

CONTRIBUTING.md has every git step, from the first clone to the pull request. The good first issues are real misses, each with the command that reproduces it.

The number

Every change is judged by one number: on a set of labelled questions, how often is the right page in the top three?

In a notebook:

from bootcamp_agent.coach_eval import measure
measure()

hit rate 21/25 = 84% (top-k)

missed:
  ...

In gecko-ai-coach, on three sets:

uv run ai-coach measure --pages ../dev3pack-cohort-2026-09/units/en --cases data/dev3pack.jsonl

Your number depends on which weeks are in your clone: a page you do not have yet cannot be found. Measure before and after on the same clone, or the difference means nothing.

A hit means the right page was reached. That metric is weak on purpose: it says nothing about whether the passage answers the question. A better metric is itself a contribution.

Two lessons the number teaches

A bigger number is not always a better coach

The coach returns three passages. Is three right? Measure it:

measure()            # 20/25 = 80%   as shipped
measure(top_k=5)     # 22/25 = 88%
measure(top_k=8)     # 22/25 = 88%   and no further: the curve flattened

Eight points for one argument. Take the win?

No. Read what you bought. The metric asks "was the right page anywhere in what came back". Of course more passages win: at top_k=25 it would score 100% and be useless. Retrieval did not get better. The learner now reads more, and the metric cannot see that cost.

A metric you can game is a metric you will game by accident.

A win on one snapshot can vanish on the next

Chunk size looked like a real win on an earlier corpus:

max_chars=800   18/25 = 72%     as shipped
max_chars=400   19/25 = 76%     smaller chunks concentrate the signal
max_chars=300   17/25 = 68%     too small; the curve is not monotonic

Then two pages were added to the course, and the gain disappeared. Nobody changed the retriever. The corpus grew, the new pages competed, and a tuned constant stopped being tuned.

That is not a failure. It is what measuring is for: an improvement that holds for one snapshot of the data is a fact about that snapshot. You only find out by running the number again. This is why gecko-ai-coach keeps a held-out set that nobody tunes on.

Evidence artifact

Step 1: the pull request that adds your question, with what the coach returned
Step 2: the issue or label fix, naming the pages and the cause
Step 3: the before and after numbers on every set, one thing that got worse (or an honest statement that nothing did), and the pull request

Failure clinic

A number with no before. "76%" alone says nothing.
Tuning on the held-out set. Run it once, at the end. If you change the code after you see it, it is no longer held out. Session 7 is exactly this.
Reporting only the win. The regression you hide is the one that bites somebody else.
A change you cannot explain. If you do not know why it helped, you do not know when it will stop helping.
A pull request against the course clone. It is regenerated every week. Contribute in gecko-ai-coach instead.

Reference

Demo 5 — the coach, up close: use it, read it, find a miss
gecko-ai-coach CONTRIBUTING.md: every step, with git commands
Good first issues
src/bootcamp_agent/coach.py and coach_eval.py: the copy in your notebooks
Give the course to your assistant: the same coach, as an MCP server