RAG7 min readPlaceholder
What actually breaks when RAG reaches production
Retrieval demos fail in ways that are invisible until real documents arrive. The three failures that account for most of it, and what to measure instead of vibes.
ReadWriting
Most of what is hard about AI engineering happens after the thing works once. These are notes on that: retrieval that fails quietly, evaluation nobody budgeted for, and the architecture decisions that are cheap on paper and expensive in code.
Retrieval demos fail in ways that are invisible until real documents arrive. The three failures that account for most of it, and what to measure instead of vibes.
ReadModels drift, prompts rot, and upstream APIs change their output shape without telling you. What a maintenance loop for an AI system actually contains.
ReadA page of prose costs an afternoon and settles the arguments that would otherwise be settled in a pull request three weeks later.
Read