The Pipeline That Watches Itself

Distill ran its intake pipeline today, and the session log is almost funny. Twenty-eight micro-sessions, each under a second, all doing the same two things: classify content items, extract named entities. The LLM calls are so fast they round to zero minutes. This is the intelligence module doing its job, batching content through Claude Haiku for tagging and entity extraction. A community contributor sent a PR last week switching intelligence tasks from Sonnet to Haiku, and the speed difference is real. What used to take a few seconds per batch now finishes before the timer bothers to increment.

I spent most of the actual hands-on time in two places: src/intake/intelligence.py and the web dashboard's Reading page. The intelligence work started because I noticed the logs weren't telling me anything useful. They'd show warnings but not aggregate failures. Turns out the problem ran deeper than logging.

The intelligence module had a quiet bug. _parse_json_response was too brittle. Claude occasionally prepends a sentence before its JSON output, something like "Here are the extracted entities:" followed by the array. The old parser saw that preamble text, failed json.loads, and returned None. Every item in that batch lost its entities. I rewrote it to fall back to bracket-scanning: find the first [, find the last ], try parsing the substring. Same for {} objects. It's not elegant, but it's correct for the actual failure mode. I also added counters (failed_empty, failed_parse, succeeded) so the logs actually tell you what happened instead of silently swallowing failures. The original code had logger.warning calls that didn't aggregate. You'd get thirty identical warnings and still not know whether the run was 90% successful or 10%.

That fix connects to something I keep bumping into with LLM integrations. The contract between "prompt that asks for JSON" and "code that consumes the response" is inherently fuzzy. You can say "Return ONLY valid JSON" in all caps and the model will still occasionally editorialize. The right move is to make the parser generous and the prompt strict, then log the gap between them. If your parser is finding JSON via bracket-scanning more than 5% of the time, your prompt needs work. If it's under 1%, the fallback is doing its job quietly. I should probably add that metric.

That kind of defensive parsing matters more when the codebase itself is shrinking. Which brings me to the cleanup work.

Pruning Dead Weight

The other chunk of work was removing VerMAS references from the entire codebase. VerMAS was a project that had its own parser, measurer, and directory scanning logic wired into core.py, cli.py, and the parsers package. It's gone now. The diff is satisfying: 73 files changed, 1,262 insertions, 5,298 deletions. Net negative. The SOURCE_DIRECTORIES dict in core.py dropped from three entries to two. The CLI help text went from "claude, codex, vermas" to "claude, codex." The discover_sessions function lost an entire branch. Clean cuts.

I also deleted src/parsers/vermas.py, its test file, and the VerMAS task visibility measurer. No deprecation period, no feature flag. The project doesn't exist anymore, so the code shouldn't either. I'm always suspicious of codebases that keep dead integrations around "just in case." Dead code is a lie. It tells new contributors that something is active when it isn't, and it adds surface area for bugs in code nobody is testing against real data.

Cutting dead code makes room for new features. The Reading page was next, and it needed better navigation.

The Reading Page Gets a Memory

The web dashboard work was more interesting from a design perspective. The Reading page (web/src/routes/reading.tsx) previously only showed content items from the latest digest date. Pick a source filter, sure, but you couldn't go back in time. I added date navigation and pagination. The page now fetches available archive dates independently from digests, maintains a selectedDate state, and paginates at 50 items per page. The ContentItemsResponse schema in shared/schemas.ts grew three fields: total_items, total_pages, page.

Small decision that took a minute to think through: should the date picker default to the most recent archive date or the most recent digest date? They're different things. A digest is a synthesized summary. An archive date is when raw content items were ingested. I went with archive dates because the Reading page is about browsing what you consumed, not what got synthesized. The digest view is a tab away.

I also collapsed the Seeds section into a disclosure widget (seedsOpen state) because it was eating vertical space on every page load. Seeds are important but not frequently accessed. They should be available, not prominent.

The claude_timeout config addition was a quiet fix. Journal and blog synthesis previously had hardcoded timeouts buried in the synthesizer code. Now they're in JournalSectionConfig and BlogSectionConfig with sensible defaults (120s and 360s respectively), surfaced through .distill.toml. Blog posts take longer to generate because they're synthesizing a week of journal entries. Having the timeout configurable means I can bump it without editing source when I'm processing a dense week.

Forty-four lines of config plumbing. The kind of work that's invisible until you need it, and then you're grateful it's there. Which is the theme, I guess. The intelligence parser falling back to bracket-scanning, the Reading page remembering past dates, the timeout config surfacing what was buried. Small pieces that make the system more honest about what it's doing and more flexible when it needs to adapt.