Session 8. Loops and graphs — Wed 23 Sep

Follow along: today's class

Keep this page open during the session. Every step says what to open and what to run, in the order we run it in class.

Part What Time
0 Before we start: pull, and one extra to install 5 min
1 Where we are going: the autonomy ladder 10 min
2 Project 03, together: four roles over yesterday's index 45 min
3 Your exercise: three counted numbers and a table with no fall-through 40 min
4 Break it on purpose: a retry storm, a half-failed tool, an edge nobody drew 10 min
5 Hand it in 10 min

0. Before we start

From your course folder:

git stash push -m "my work before today's update"
git pull
uv sync --extra projects --extra agents
uv run jupyter lab

Why the stash first. git pull stops when you have typed into a notebook it wants to update, and it says "your local changes would be overwritten". git stash puts your changes on a shelf and deletes nothing. To take one notebook back as you left it, see when git pull stops.

Check that the pull worked before you sync:

grep agents pyproject.toml     # one line back means today's update is here

If uv says Extra 'agents' is not defined, your pull did not land. The error names the extra, so it reads like a broken course. It is not: it is the old pyproject.toml still on your disk. Run git pull again and read what it says. The usual cause is the stash step above being skipped, so the pull stopped on a notebook you had typed into and you never saw the message.

Name both extras, or you lose ChromaDB. uv sync makes the environment match exactly what you asked for. Ask for agents on its own and it uninstalls the projects extra, ChromaDB goes with it, and project 02's index stops building. Name both, every time.

What --extra agents adds. LangGraph, a DuckDuckGo search tool, and grandalf so the graph can be drawn without the network. All three are for the project only. Today's exercise needs nothing new: it runs offline, on fakes, in milliseconds. The project runs without any of them, and says so when it falls back.

Ollama is optional today. The project replays a recorded run for the cells that need a model, and prints [recorded] where it does. If you have Ollama and did yesterday's project, you already have what you need.

Open two files:

No setup on your laptop? Open them in Colab. The first cell of each fetches the course for you.

1. Where we are going

Yesterday you measured retrieval. Today you count calls.

Row What it adds The new way it fails
A model call nothing but the prompt it answers from memory, fluently
RAG passages retrieved first, cited back the wrong passages, cited with confidence
An agent a loop: decide, call a tool, read the result, go again it never stops, unless you make it
Agents in a graph several narrow roles, one shared state, declared edges more calls, and a transition nobody declared

Every row costs more than the one above it. Nobody can tell you from taste which row your problem is on. You count, and then you decide.

The projects are practice, not the capstone. Project 03 runs on the real filings index. The capstone stays on the six documents in data/corpus/, offline, on the fake model. Nothing you build today in the project is required for the capstone; everything you learn is.

2. Project 03, together

Open the project notebook and run it from the top. Twelve steps now, which is more than one session holds, so run 1 to 8 together and leave 9 to 12 for after. The moments to watch for:

Step What happens
1. Build the index Yesterday's eight filings, chunked again: 1,927 passages, no word lost, the shortest 6 characters and the longest 799.
2. One agent, one loop The number to beat. One model call per question, and on 5 of the 20 labelled questions it reads the wrong company's filing and answers fluently, with a real citation to the wrong document.
3. The coordinator A router that picks the company from the question with no model call at all. Measured on its own against the 20 labels, because routing and answering fail for different reasons and a single score hides which one broke.
4. The output contract ANSWER_JSON_INSTRUCTIONS, printed and taken apart a line at a time. Four malformed replies, four refusals that name the field. Session 3's rule in a team: the application validates, never the model.
5. Every prompt in the open Both system prompts and both user messages, whole. Then the word the code actually reads: APPROVE. "Looks good to me" approves in English and rejects in code, and the run pays two more calls for it.
6. Build the team by hand Four node functions, one transition dict, a nine-line walker. Then build_team beside it, field by field, all ==. They wrote it, so they can say what it does.
7. The team The imported team over one TeamState. Every exit names why it stopped, and every citation is one retrieval returned.
8. Fail first The critic as a conversation. Uncapped, writer and critic argue until the budget stops them, and the run ends with stopped_because="budget" instead of an answer. The cap is one line.
9 to 11. Tools A tool package written from a cell and imported, a researcher that picks between tools, an optional DuckDuckGo search that is honest about not being reproducible, and which role may hold which tool.
12. The framework seam The same four nodes as a LangGraph StateGraph and as plain Python. The graph gets drawn and one run gets streamed, node by node. The graph is yours; the framework is a way to run it.
Measure The team against the single loop on the same questions. Loop: 1 call. Team: 2, or 4 when the critic sends it back.

If the room wants the full LangGraph build, it is a tutorial of its own: projects/tutorials/03-build-a-team-with-langgraph.ipynb walks the state and its reducer, the team as data, each node under the prompt it sends, the wiring, the drawn graph and the streamed run. Not graded, and it runs with no model and no LangGraph.

The point of the project: the second, third and fourth role are not free, and the table is how you find out what they bought. Right company in the retrieved passages: the loop 15 of 20, the team 20 of 20, for 3 model calls against 8 over the same three questions. Then the last cell undercuts it honestly: the same loop with k=8 reaches 17 of 20 at one call per question. Decide out loud whether the rest of that gap is worth the calls.

3. Your exercise

In the session notebook. 300 marks, three exercises, in notebook order:

Challenge What you write Marks
ch08-e1 the comparison table: what the chain, the loop and the capped reflection each cost, in counted calls 100
ch08-e2 the graph appendix: what its trace showed, or why you did not run it 100
ch08-e3 step(state, event): five states, six declared edges, and a refusal that does not raise 100 
uv run bootcamp check ch08

As shipped it prints 0/3 and says what is wrong in each one. That is where you start.

The three that catch people:

4. Break it on purpose

Three moments, and nothing raises in any of them.

Where The failure
Project, step 8 The critic as a conversation: no cap, so the run ends on budget with no answer.
Session notebook, section 10 A retry storm: one edge back to the state it came from, with nothing counting. visited only grows, and that is the counter that caps it.
Session notebook, section 11 A partial tool failure: two sources, one down, half the passages back. Which event is that? It is a decision, made in the open, and the flag travels with the answer.

The question for the room: in your own step table, which pair did you leave out because it felt impossible? That pair is the edge nobody drew, and it is the one that reaches production.

5. Hand it in

When the check is green:

uv run bootcamp submit ch08 --github <your-github-name> --push

Save the notebook first. submit reads the file on disk.

It goes to dev3pack-submissions, never to the course repository. No gh on your machine? The command prints the browser route. Type the path exactly as it shows: submissions/<your-github-name>/ch08/, with no spaces.

Tomorrow

Session 9 stops asking what the graph is and starts asking whether it is the right one: a trace with events you can read back, a golden set, and a pass rate.

← The RAG family, and the index underneathQuick quiz (ungraded) →