How Distill Eats Its Own Tail

Today was mostly distill eating its own tail. I spent the day running the full pipeline, end to end, on real data for the first time in a while, and the session log is almost comically self-referential. Twenty-eight coding sessions, nearly all of them distill prompting Claude to write things about what I built with Claude.

Let me be specific. The longest sessions were thematic blog synthesis: "The Self-Referential AI Tooling Loop" (0.8 min), "What Your Coordination System Can't See" (1.1 min), "Why Branch Merges Keep Failing" (1.0 min), and two separate runs on automation versus autonomy in AI agents (1.1 min each). These are the meaty calls where BlogSynthesizer shells out to claude -p with a few thousand words of journal entries and gets back a full essay draft. Sandwiched between them: a dozen quick entity extraction and classification calls, each under 10 seconds, the intake intelligence pipeline doing its batch NLP work.

What strikes me looking at this session log is how much of the compute is structural. Out of 28 sessions, maybe 5 are the "real" creative work (journal entries, blog essays, the daily digest). The other 23 are classification, entity extraction, key point extraction, blog metadata parsing. The ratio is roughly 80% scaffolding, 20% substance. That feels about right for any content pipeline, honestly. Most of the work in publishing has always been the boring parts: tagging, categorizing, linking, formatting. LLMs just made it possible to automate the boring parts without hand-coding rules for each one.

When Your Tool Knows Too Much

The structural work matters because without it, the creative parts wouldn't know what to synthesize. The session that caught my eye was the daily digest synthesis at the end, 0.6 minutes. That's the get_unified_intake_prompt() call that blends sessions, reading, and seed ideas into one narrative. The prompt literally says "write a personal daily digest that blends what you built, what you consumed, and what you're thinking about." Which is exactly what I'm doing right now, writing this essay. The pipeline is trying to do my job.

Here's what I think about that: it's fine, actually. The digest prompt produces a decent first draft that captures what happened. But it can't do what I'm doing in this paragraph, which is noticing the irony and deciding that's interesting enough to write about. The pipeline captures events. The essay captures judgment.

Three journal entry sessions ran today (0.5 min each), and I can tell from the prompt fragments that they're asking Claude to "write in a reflective, personal voice as if the developer is recounting their day." The emphasis on "as if" is doing a lot of work there. The journal synthesizer produces something that reads like a developer's journal. The thing you're reading now is a developer's journal. The difference is that I'm choosing what to emphasize and what to skip, and those choices are the whole point.

The Blog Assembly Line

That choice layer shows up most clearly in the blog generation runs. The weekly essay sessions are interesting because they ran three times with slightly different weeks of data (0.8, 0.9, 1.0 min). That's the generate_blog_posts() flow in core.py doing its thing: load config, read journal entries, build WeeklyBlogContext, detect themes, synthesize. Each run also triggered a format_index and metadata extraction pass. The Ghost newsletter adaptation ran once (0.6 min), reformatting a blog post into email-friendly markdown with a subject line and preview text.

I'm watching the thematic post generator try to find arguments in my journal data. "The difference between automation and autonomy in AI agents" ran twice, which means the theme detector flagged it, the synthesizer produced a draft, and then something (probably the blog state tracker in state.py) decided it needed a second pass. The automation/autonomy distinction is one I keep coming back to: automation replaces steps, autonomy replaces judgment. My pipeline automates the steps of content production. It doesn't have autonomy over what's worth saying.

Which brings me back to the unsexy infrastructure that makes the synthesis possible.

The 80/20 Nobody Talks About

Fifteen of today's sessions were pure classification and entity extraction, the intelligence.py batch calls. Each one under 12 seconds. They're processing content items from the intake pipeline, tagging them as tutorial/opinion/news/reference, pulling out project names and people. This is the invisible infrastructure. Nobody reads a blog post and thinks "wow, great entity extraction." But without it, the blog synthesizer wouldn't know which content items relate to which projects, and the thematic detector couldn't cluster journal entries by topic.

Building distill has taught me that content pipelines are mostly plumbing. The glamorous part, the essay generation, the thematic synthesis, sits on top of a very unglamorous foundation of parsing, classifying, and linking. Today's session log proves it. Five creative sessions. Twenty-three janitorial ones. And the janitorial ones are what make the creative ones possible.

Tomorrow I need to look at why the automation/autonomy theme triggered two synthesis runs. Probably a state tracking bug in blog/state.py. The kind of fix that takes ten minutes but saves duplicate API calls forever. Plumbing, again. The same kind of plumbing that let me see today's session log clearly enough to notice the irony in the first place.