AI collaboration

Lab setup

First, make sure you have completed the initial setup.

If you are part of a course

  1. Open Terminal. Run the update command to make sure you have the latest code.
    $ mwc update
  2. Move to this lab's directory.
    $ cd ~/Desktop/making_with_code/pilot/labs/lab_ai_collaboration
    

If you are working on your own

  1. Move to your MWC directory.
    $ cd ~/Desktop/making_with_code
    
  2. Get a copy of this lab's materials.
    git clone https://git.makingwithcode.org/founders-studio/lab_ai_collaboration.git

Ask an AI model almost anything and you will get a fluent, confident answer. But where does that answer come from? In this lab, you will build a working model of how an AI agent "knows" things, and use it to take on a task that no model can do in one shot: analyzing the themes of a whole library of books, with real quotations to back up every claim.

By the end of this lab, you will be able to:

An impossible question

๐Ÿ’ป Start Claude Code in this lab's folder, and ask:

Summarize the main themes of Western literature, with an explanatory quotation for each theme.

๐Ÿ‘ Read the answer. It probably sounds impressive: a handful of grand themes (the hero's journey, love and loss, the individual and society...) each with a famous-sounding quotation.

Now let's check it. The corpus folder contains the full text of the books in our library, downloaded from Project Gutenberg.

๐Ÿ’ป Copy one of the quotations into quotes.txt, one quotation per line, followed by | and the book it supposedly comes from. Then run:

$ python verify_quotes.py quotes.txt
"To be, or not to be, that is the question" ... FOUND in corpus/hamlet.md
"All happy families are alike..."           ... NOT FOUND

verify_quotes.py searches the actual texts for each quotation (ignoring small differences in punctuation and capitalization).

๐Ÿ’ป Check all of the model's quotations. How many are exact? How many are paraphrased, misattributed, or not in the text at all?

The context window

The alternative to baked-in knowledge is to give the model the actual text. Anything the model reads in a conversation (your messages, files it opens, the output of commands it runs) goes into its context window. The model can quote what is in its context exactly, because it is right there.

๐Ÿ’ป In Claude Code, run /context. You will see how much of the context window is already used, and by what: instructions from the harness, tool definitions, and your conversation so far.

๐Ÿ’ป Ask Claude Code to read corpus/hamlet.md and find three quotations about death. Then run /context again. How much space did the book take up?

๐Ÿ’ป Now verify the new quotations with verify_quotes.py. What happened this time?

The context window is large, but not unlimited. It is measured in tokens, the chunks of text a model reads (roughly three-quarters of a word each).

๐Ÿ’ป Count the tokens in the whole corpus.

$ python count_tokens.py corpus
corpus/anna_karenina.md                       481,252
...
corpus/hamlet.md                               48,509
...
corpus/middlemarch.md                         430,920
...
TOTAL                                       5,180,147

Compare the total with the size of the context window shown by /context.

Usage and model choice

Every token a model reads or writes costs computation, which costs money and energy. That's why AI services have usage limits, and why choosing the right model for a task matters.

๐Ÿ’ป Run /usage to see how much of your usage allowance you have spent.

๐Ÿ’ป Run /model to see which models are available. Larger models are generally more capable, but slower and more expensive; smaller models are fast and cheap.

๐Ÿ’ป Give the same task to two different models: "Read corpus/frankenstein.md and list the chapters in which the creature speaks." Compare speed, the share of your usage spent, and the quality of the answers.

Data density

Most of the scholarly writing you will want to read arrives as PDF. PDFs are designed for printing, not reading by machines: along with the text, they carry page layouts, running headers and footers, page numbers, and images. When a model reads a PDF, all of that costs tokens and adds noise.

Converting a source to Markdown keeps the text and its structure (headings, lists, quotations) and drops the rest. More of what matters fits in context, and the model reads it more reliably. We call this data density: how much useful content you get per token.

๐Ÿ’ป The theory folder contains PDFs of essays about theme in literature. Convert one to Markdown with MarkItDown.

$ markitdown theory/pdf/aristotle_poetics.pdf > theory/aristotle_poetics.md

๐Ÿ‘ Open the Markdown file. Is anything missing or garbled? Fix any problems by hand. Conversion isn't perfect, especially for footnotes and tables.

๐Ÿ’ป Compare the token counts.

$ python count_tokens.py theory/pdf/aristotle_poetics.pdf theory/aristotle_poetics.md

๐Ÿ’ป Convert the rest of the theory PDFs. Commit the Markdown files.

Decomposing the problem

Now we can return to the impossible question. The corpus is far too large to fit in one context window, so no single conversation can read it all. But we can design a process (a pipeline) in which the model never needs to hold more than fits in context at once.

Our pipeline has four stages:

  1. Read the theory (in parallel). Each theory text is read into context and summarized on its own. These summaries don't depend on each other, so they can run at the same time, in separate sessions.
  2. Compose a framework. The summaries are small enough to fit in context together. From them, write a theoretical framework: a set of themes, each with a definition and a description of what evidence for it looks like in a text. This is your literature review.
  3. Apply the framework (in sequence). Read each target text into context, one at a time, and record which themes appear, with quotations. Working in sequence lets you refine the framework as you go, when a text doesn't fit it well.
  4. Synthesize. Combine the results for each text into an answer to the original question.

๐Ÿ‘ Open pipeline.md. It lays out the four stages, with a place to record who (or which agent session) does each step and where its output is saved.

๐Ÿ’ป As a team, divide the theory texts among yourselves. For each one, start a fresh Claude Code session (so the context holds only that text), have it read the Markdown file, and write a summary to summaries/. Read each summary yourself, and check its quotations with verify_quotes.py.

Literature review

๐Ÿ’ป In a new session, have the agent read all of your summaries together and help you draft framework.md: four to six themes, each with a definition, the theorists it draws on, and what evidence for it looks like.

The literature review is your writing. Use the agent to organize ideas, suggest structure, and give feedback on drafts, but you decide what the framework says, and every claim must be tied to a source you actually read (a summary in summaries/ and the Markdown text behind it).

Applying the framework

๐Ÿ’ป Divide the target texts in corpus/ among your team. For each text, in a fresh session, have the agent read framework.md and the text, and record the themes that appear, with quotations, in analysis/. Use a smaller model if it does the job well.

๐Ÿ’ป When a text doesn't fit the framework well, discuss it as a team. Should the framework change? If you change it, note what changed and why at the bottom of framework.md, and decide whether earlier texts need another look.

๐Ÿ’ป Check every quotation in analysis/ with verify_quotes.py.

Synthesis

๐Ÿ’ป Finally, have the agent read your analysis/ files together and help you write synthesis.md: your team's answer to the original question, the main themes of this corpus of Western literature, with verified quotations.

๐Ÿ‘ Compare synthesis.md with the answer the model gave at the very start of the lab. What's different? Which do you trust more, and why?

Deliverables

๐Ÿ’ป Answer these questions in questions.md:

  1. In your own words, what is the difference between what a model has baked in and what it reads into context? When would you trust each?
  2. How much smaller was a theory text as Markdown than as a PDF? Why does that matter?
  3. Which steps of your pipeline ran in parallel, and which in sequence? Why?
  4. Which parts of the decomposition were your thinking, and which were the agent's?
  5. Describe one way the agent contributed to your literature review, and one decision you made yourself.