RAG Architecture — Chunking, Retrieval, and Why the Model Is Rarely at Fault

Most RAG debugging goes wrong in the first five minutes, when someone decides the model gave a bad answer and starts changing the model. The overwhelming majority of bad RAG answers are retrieval failures, and retrieval failures are cheap to diagnose if you have the one piece of instrumentation most systems are missing.

The pipeline, and where it breaks

document ──▶ chunk ──▶ embed ──▶ index
                                   │
query ──▶ embed ──▶ search ──▶ rank ──▶ truncate ──▶ prompt ──▶ generate
             │         │         │          │
             │         │         │          └── the right chunk falls off the end
             │         │         └── the right chunk ranks below noise
             │         └── the right chunk is not similar enough to be found
             └── the query embeds into a different region than the document did

Four distinct failure points before generation is even reached. A model change addresses none of them.

Log the retrieved chunk IDs

This is the single highest-value thing in this guide.

On every request, log which chunks were retrieved, their scores, and which of them survived truncation into the final prompt. Without it, the first question in every investigation — “was the correct chunk even retrieved?” — is unanswerable, and you are reduced to guessing.

With it, diagnosis is mechanical:

1. Correct chunk not retrieved at all → chunking, embedding, or query problem
2. Retrieved but ranked below the cut-off → ranking or context budget problem
3. In the prompt and ignored → now, and only now, a prompt or model problem

Chunking

Chunking is where most retrieval quality is won or lost, and the default of “split every 512 tokens” is usually the worst option available.

Split on structure, not on length. Documents have boundaries — sections, headings, function definitions, table rows. Splitting on those produces chunks that are about one thing. Splitting on a token count produces chunks that end mid-sentence and mix two topics, which embeds into a vector that represents neither well.

Size to the question, not to the document. If your users ask questions answerable by a paragraph, chunk to paragraphs. If they ask questions requiring a whole procedure, chunk to procedures. Chunks much larger than the answer dilute the embedding; chunks much smaller fragment it.

Overlap sparingly. Overlap helps when the answer straddles a boundary, and hurts by inflating the index and returning near-duplicate results that consume context budget. Ten to fifteen percent is a reasonable starting point; large overlaps usually indicate the split strategy is wrong.

Carry context into the chunk. A chunk that says “it must be restarted after this change” is useless in isolation. Prefixing each chunk with its document title and section heading is a small change that measurably improves both retrieval and generation, because the chunk now carries enough context to be interpreted alone.

Embeddings

Choose an embedding model on your own data, not on a leaderboard. A general benchmark such as MTEB is a filter, not an answer — the models near the top are close enough that domain fit matters more than rank.

The practical evaluation is: take fifty real queries, label which chunks should be returned, and measure recall at k for each candidate model. It takes a day and it produces an answer specific to your corpus.

Retrieval and ranking

Pure vector search has a known weakness: it is bad at exact matches. A query containing an error code, a product name, or an identifier often ranks a semantically related chunk above the one containing the literal string.

Hybrid search — combining vector similarity with keyword search such as BM25 — fixes most of this and is usually the highest-value retrieval improvement after chunking. Technical corpora benefit the most, because they are full of identifiers.

Reranking applies a slower, more accurate model to the top 50 candidates and reorders them. It costs latency and buys precision. It is worth adding when recall is good but the right chunk lands at position 20 — and worth nothing when the right chunk was never retrieved.

Metadata filtering before the vector search is cheap and frequently forgotten. If a query is scoped to one product, one version, or one tenant, filtering first shrinks the search space and removes an entire class of confidently wrong answers drawn from the wrong document.

Context budget

The prompt has a limit, and retrieval will happily return more than fits. What gets dropped is a design decision, and if you do not make it, truncation makes it for you — usually by cutting the end of the list, which is where a reranker would have put the useful long-tail results.

Decide explicitly: how many chunks, in what order, and what happens when they do not fit. Placing the highest-ranked chunks nearest to the question rather than at the top of a long block is a small change that helps in practice.

Track how often truncation is dropping chunks. If it is common, the answer is usually better chunking rather than a larger context window — a larger window makes the same problem more expensive.

Evaluate retrieval separately

Two separate evaluation suites, because they answer different questions and have different fixes.

Retrieval evaluation needs queries with labelled relevant chunks. Measure recall at k — of the chunks that should have been found, how many were? This is answerable without a model in the loop and it is where most improvement comes from.

Generation evaluation takes the retrieved context as given and asks whether the answer is correct, grounded, and complete. Grounding — whether every claim in the answer is supported by the provided context — is the property that matters most and is the one users notice when it fails.

Running these together hides which half regressed. Run them separately and you will know within minutes whether last week’s chunking change or last week’s prompt change caused the drop.

When RAG is the wrong tool

• The corpus is small enough to fit in context. Retrieval adds a failure mode for no benefit.
• The answer requires aggregation across many documents. “How many customers reported this?” is a database query, not a retrieval problem. Retrieval finds relevant chunks; it does not count things.
• The data is structured. If the answer lives in a table, query the table. Embedding rows and hoping is strictly worse than SQL.
• Freshness is measured in seconds. Indexing lag becomes the bottleneck, and a direct lookup is both simpler and more current.

Checklist

• Retrieved chunk IDs and scores logged on every request
• Chunks split on document structure, sized to the expected answer
• Document title and section heading prefixed into each chunk
• Hybrid search where the corpus contains identifiers or error codes
• Metadata filters applied before vector search where the query is scoped
• Explicit context budget policy, with truncation frequency monitored
• Separate retrieval and generation evaluation suites
• Index records which embedding model produced it
• Index rebuildable from source