Что на самом деле ломается, когда RAG доходит до продакшена
Демо поиска отказывают так, что этого не видно, пока не придут настоящие документы. Три сбоя, на которые приходится большая часть, и что стоит измерять вместо ощущений.
Заготовка. Эта статья идёт вместе с разделом как макетный каркас и исключена из поиска и из ассистента сайта, пока в ней не появится настоящий текст.
Эта статья опубликована на английском. Остальной сайт переведён, а статьи нет, потому что технический разбор в машинном переводе стоит меньше, чем тот же разбор на языке, на котором он написан.
Every retrieval system demos well. That is the problem: the demo is run against twenty clean documents by the person who chose them, and production is four thousand documents chosen by nobody, half of which are scanned, and a user who phrases the question differently than you did.
Chunking is a retrieval decision, not a preprocessing one
Splitting on a fixed token count is the default because it is the easiest thing to write, and it is wrong for most corpora. A contract clause split across two chunks retrieves as two half-answers, both of which score badly, neither of which is returned.
Embeddings do not understand your acronyms
The failure looks like the model hallucinating. It is not. The retriever returned nothing relevant and the model did what it was asked to do with what it was given.
Nobody is measuring retrieval
The single most common gap. Teams evaluate the generated answer, which is the end of a chain, and conclude the model is weak. Measure recall at the retrieval step first: if the right passage was never in the context window, no amount of prompt work will fix the answer.
If you cannot say what fraction of questions retrieved the correct source document, you do not have a RAG system. You have a search box with a language model attached to it.
This post is a placeholder shipped with the writing section. Replace the body with the real piece; the frontmatter above is the only thing the site reads.