Why AI Coding Agents Need a Memory Pipeline

An episode of the “AI that Works” show where Dex (CEO of HumanLayer) and Aaron (CTO of Boundary/BAML) walk through a pipeline they built together that mines AI coding agent sessions for durable knowledge. The pipeline analyzes agent traces to produce proposed snippets for CLAUDE.md/AGENTS.md, reports on what agents get stuck on, and structured “context shards” — all for about two cents per session by using cheap classifier models for bulk filtering and strong models only where they matter.

Key Points

• The problem: Learning is lost between sessions. When a user corrects an agent or the agent figures something out after hundreds of tool calls, that knowledge disappears with the next fresh context window.
• Naive first attempt: Strip tool calls from session transcripts, run them through a single Haiku structured-output call, and emit “context shards” with citations (session ID, message ID, user ID). Directionally correct but low quality.
• Three classes of memory emerged:
  • Facts — things agents discovered that are true but hard to find in the repo (e.g., BAML language semantics).
  • Rules — behavioral corrections a human explicitly gave an agent (“don’t do X that way”).
  • Recurring research — questions agents keep re-investigating, like how the BAML standard library is organized.
• Start with maximum intelligence, then distill: The first version was just a Claude Code agent digging through the data (~$100 of API usage for one report). Watching which scripts and filtering patterns it used revealed the structure to encode into a fixed pipeline that costs $0.50 total for a week of sessions (~2 cents per session).
• Jev (cheap classifier model) was the cost breakthrough: Classification steps like “is this message a discovery attempt?” or “is this a human correction?” run on Jev, which handles 97% of the calls at ~5% of the bill. A weak-model/strong-model alternation runs throughout the pipeline.
• Fast iteration mattered more than clever prompts: Hashing inputs and caching responses as JSON on disk lets any step be replayed without rerunning the whole workflow. Print out the actual prompts at every step to see what the pipeline is really doing (this caught a bug where truncating messages from the end destroyed valuable data — truncating the middle instead kept 100% of valuable content in only 60% of the text).
• Golden-dataset evals: The initial expensive Claude Code run served as a golden dataset; every prompt change was checked against it for recall (“did the filter keep all the messages that mattered?”).
• Results: A week of the BAML team’s sessions produced three region-tied MD files (e.g., for baml_compiler_to_..., baml_builtins), a CLAUDE.md draft with architecture pointers, rules, language facts, and a report of recurring agent questions — including the insight that agents weren’t reading an existing architecture.md because nothing pointed them to it.
• BAML Query: Observability comes for free — every LLM call is stored to a program database on disk (SQLite-style, per Jared Sumner’s Bun tweet), and agents can query it with SQL to build bespoke inspection UIs, which generic observability dashboards can’t replicate.
• Read the prompts yourself: Models are bad at writing prompts/skills because anything they add is by definition already in the weights — generic slop. Hand-written prompts carry the actual new information.

Detailed Summary

The setup: why memory matters

Agent sessions are hill-climbing: the user gives feedback, the agent fixes things, and either party learns something. But when the session ends, that learning is lost. Some of it is tribal knowledge (“please don’t run that command again”), and some took 500 tool calls to discover — information you’d rather capture as a snippet, diagram, or architecture note than re-derive every session. Related approaches exist (compounding engineering, Matt Biozo’s “retro skill”), but the open question is how to decide what earns a spot in your CLAUDE.md or memory bank, and how to keep it from going stale.

The naive v1

Take user + assistant messages only (drop tool calls for cost), send to Haiku with structured output, get back “context shards”: the memory, reasoning for why it matters, and citations back to session/message/user IDs so feedback could be tracked across a whole org (HumanLayer is multiplayer, so two users hitting the same problem is visible). It worked directionally but required constant prompt fiddling.

Building the real pipeline

Aaron’s process, in order:

1. Build lots of tooling — about ten dashboard iterations to browse sessions, filter to human messages, and trace what flowed where (including Sankey diagrams of extraction).
2. Prototype with Claude Code — let a top model roam the data freely and produce the best possible report, then study its behavior to find the reusable patterns (which messages it filtered, what signals it used).
3. Encode the workflow in BAML — LLM functions with typed inputs/outputs, enum classifications, and float scores that come back as true probabilities.

The final pipeline has three branches, mapped to the three memory classes:

• What agents found out: Extract small facts from sessions, build a “code map” from every path agents read (a heat map of the codebase), and prioritize facts that span many files. A grouping step coalesces the same fact found across separate sessions (e.g., a BAML fact that surfaced in 3 sessions from 2 people) into a single memory with all its citations attached.
• Human corrections: Classify which human messages are actual corrections versus just “do phase six” style instructions, group the rules (with Opus), and feed them to a draft writer that produces a report (“fix this first… asked for in 6 of 7 tool calls… these four questions reoccur in 4–5 sessions each”).
• What agents kept researching: Take subagent intro prompts (the parent’s request to the subagent), extract paths and directory names via regexes from tool calls, group recurring questions, then have Opus read the actual contents of every file the subagent touched and pick which docs best answer each question. Jev then checks whether the subagent’s final report actually answered the question.

Post-filtering, memories are judged for quality: is it tribal knowledge (not obvious from reading the code — “did it take 10 files to understand this?”), is it readable, is it a duplicate? Then everything is sorted into the rules/facts/research categories and drafts and reports are written.

Iteration discipline

• Disk caching of every step (hashed inputs → JSON responses) so mid-pipeline changes replay instantly instead of rerunning everything.
• Print all prompts at every step to actually see what’s happening — this surfaced the middle-truncation fix and other silent data loss.
• Vibe-val against a golden set: the original Claude Code run defined what “good” looked like; filters were tuned to keep 100% of valuable messages while dropping 40% of the text.
• Human feedback: team members rated whether they’d add a shard to their code; specificity (per-region CLAUDE.md files) is what moved them from lukewarm to interested.

BAML Query

Rather than sprinkling span/trace code through the pipeline, BAML auto-captures spans for LLM functions into a local program database. bamlq gives a CLI/SQL interface over the run’s data, which let Aaron vibe-code custom dashboards on top of the live program — the only API needed is the query layer. The argument: generic observability UIs can’t anticipate what you need to see; give the model keys to the database instead.

Open problems discussed

• Memory health/staleness: How do you detect that a memory is now wrong or “hobbles” the model by telling it things it already knows? Ideas: tie each memory to a code region via the code map, ask the agent in-session whether context is outdated, or analyze after the fact for moments where the agent discovers a memory was wrong.
• Storage and retrieval: Naively stuffing 200 skills into context breaks attention; skills don’t stack well. A public “big skill eval” benchmark doesn’t exist yet.
• Comparison with Cognition’s Devon memory graph: Devon’s “dreaming across sessions” covers storage, organization, and metadata well, but leaves what makes it into a memory to the agent in-session — which loses cross-session correlation, the hardest part.
• Making the pipeline incremental: Add a fourth input — already-approved memories, the user’s current CLAUDE.md, and dismissed memories with feedback — so reruns don’t regenerate the same things, and the report can also recommend removals. Over time, rejection feedback can tune earlier steps (fact-finding criteria, correction classification).
• Staleness testing is hard: Truly testing whether a memory is still needed would mean running some sessions without it, which users won’t accept; backtesting new models on old traces is unreliable because changing early messages invalidates the rest of the trajectory.