Two Ways RAG Fails
When a RAG system gives a wrong answer, the instinct is to blame the model. But most of the time, the model did exactly what it was told. The problem is what it was given: the retrieval step handed it the wrong evidence. Wrong evidence in, confident wrong answer out.
The author tested two new questions against a RAG system built from 60 lines of Python and six onboarding documents. Both failed, for different reasons.
Failure 1: Chunks Too Small
Question: "Can I expense my home office setup during onboarding?"
The system retrieved five chunks. A cross-reference from the expense policy: "See Benefits Guide Section 5.3 for home office stipend details." A section heading with just its intro line: "5.3 Home Office Setup, for hybrid and remote employees." A mention of the ergonomic assessment. The scores were decent, around 0.47. The system found the right documents.
But the actual answer—the $750 stipend on hire, the $250 annual refresh, the $1,200 ergonomic budget—none of it made it into the retrieved chunks. Those numbers sat in a chunk that never cracked the top five. The model was honest: it said it didn't have specific information and that the details sit in Benefits Guide Section 5.3, "which isn't fully shown in the context provided."
This is not hallucination. It's incomplete response because the paragraph-based split cut the section too short.
Failure 2: Ambiguous Match
Question: "What's the 90-day rule?"
Five chunks came back from four different documents, all with low, similar scores in the 0.27 to 0.39 range. "90 days" shows up in completely different contexts: probation period, RRSP matching (starts after 90 days), expense claims (submit within 90 days), security access (expires after 90 days). The embeddings all carry the same phrase. Semantic search can't tell them apart.
The model handled it reasonably—it surfaced two rules and asked for clarification. But with 6,000 documents, every vague question floods the model with irrelevant context. Costs go up, latency goes up, quality goes down.
Fix 1: Heading-based Chunking
The home office problem was a chunking problem. The paragraph split cut sections too short. But these onboarding docs have structure: headings, sections. Heading-based chunking keeps a section together: each heading plus everything under it until the next heading becomes one chunk.
import re
HEADING_RE = re.compile(r"^#{2,6}\s") # split on ## and deeper, keep the # title attached
def chunk_by_heading(text: str, source: str) -> list[dict]: """Each heading + everything beneath it, until the next heading, = one chunk.""" chunks, current = [], [] for line in text.split("\n"): if HEADING_RE.match(line) and current: block = "\n".join(current).strip() if block: chunks.append({"text": block, "source": source}) current = [line] else: current.append(line) block = "\n".join(current).strip() if block: chunks.append({"text": block, "source": source}) return chunks
Same question again: "Can I expense my home office setup during onboarding?" This time the retrieved chunk contains the full section: $750 on hire, $250/year refresh, up to $1,200 for the ergonomic assessment. The model has complete evidence and gives a complete answer.
### The Chunking Toolkit
- **Fixed-size splitting**: Cut every N tokens. Works for uniform text like transcripts, but splits mid-sentence and mid-paragraph.
- **Recursive splitting**: Try headings first, then paragraphs, then sentences. A good default when you don't know your data shape.
- **Semantic chunking**: Split on meaning by embedding each sentence and cutting where similarity drops. Costs more upfront but works for dense material without clean headings.
- **Parent-child chunking**: Store small chunks for precise search, but return their parent chunks (larger) to the model for context. Search small, feed big.
A Chroma study found that chunking strategy can matter as much as which embedding model you pick. A reasonable starting point: recursive splitting, 256-512 token chunks, 10-20% overlap. But the only way to know is to measure: build 20-30 test questions with expected answers, change one thing, rerun, compare.
## Fix 2: Metadata Filtering for Retrieval
The "90-day rule" problem is about disambiguation. The embeddings all contain the same phrase, so semantic search alone can't separate them. But we know something useful: each chunk came from a specific document. The fix is to tag each chunk with its source at ingestion, then filter to one source before the similarity search runs. This is called pre-filtering.
```python
# At ingestion, stamp every chunk with a source label
{"text": "...", "source_label": "employee-handbook"}
{"text": "...", "source_label": "expense-policy"}
{"text": "...", "source_label": "benefits-guide"}
# At query time, shrink the candidate set BEFORE scoring
def retrieve(question, chunks, embeddings, top_k=5, source_label=None):
q_vec = np.array(embed_text(question))
if source_label: # the whole fix is this pre-filter
candidates = [i for i, c in enumerate(chunks) if c.get("source_label") == source_label]
else:
candidates = range(len(chunks))
scored = [
(i, np.dot(q_vec, embeddings[i]) / (np.linalg.norm(q_vec) * np.linalg.norm(embeddings[i])))
for i in candidates
]
scored.sort(key=lambda x: x[1], reverse=True)
return [{**chunks[i], "score": s} for i, s in scored[:top_k]]
Same question, "What's the 90-day rule?", filtered to the employee handbook. Now it searches 26 of the 185 chunks instead of all of them. The answer is clean: it's the probationary period. Switch the filter to the expense policy? Same question, different clean answer: submit expense claims within 90 days.
Who Picks the Filter?
The obvious catch: someone has to pick the filter. If a user just asks "What's the 90-day rule?", how does the system know which document to search? There's a pattern called a self-querying retriever: pass the question to a small, fast model first that extracts the filter (e.g., "What's the 90-day rule for expenses?" gives source = expense-policy). If ambiguous, it can ask the user to clarify. This doesn't need a big model—picking the filter is just a classification task. The big model only steps in for the final answer.
Frameworks like LangChain provide this as a building block. Managed services like Amazon Bedrock Knowledge Bases run the filter step as part of the pipeline. You still choose which model does what.
Key Insight
Neither failure is a model problem. Both are in retrieval. And retrieval is tunable. The two failures live at different layers: chunking and search. Understanding this difference is important because the fix is different for each. Start by measuring your retrieval quality with a small test set, then apply the appropriate fix.
