The 7 concepts of advanced RAG

Tap any concept to jump to its 5-card cluster.

• 1What Basic RAG Gets WrongFour silent failure modes
• 2Chunking StrategiesFixed, sentence, semantic, recursive
• 3Hybrid Search: BM25 + DenseLexical + semantic, fused by RRF
• 4Re-rankingCross-encoder precision on top-N
• 5Multi-Query ExpansionMany phrasings, one dedup pool
• 6Contextual CompressionKeep only the relevant excerpt
• 7Parent-Child ChunkingMatch small, return big

Basic RAG fails silently

Basic RAG embeds your query, grabs the nearest chunks by cosine distance, and stuffs them into context. When the query is precise and chunks are even, it works beautifully.

But real users ask messy questions. When the answer is spread across five chunks, or hinges on an exact product ID, or lives under a word you never typed, the system fails silently — and the model hallucinates with confidence. There are four repeatable failure modes, and each has a specific fix that this module builds.

The rigid card catalogue

BEFORE: Picture finding a library book using only a card catalogue that matches the exact title the cataloguer typed on the index card — one-dimensional lookup, nothing more.

PAIN: Search "car" when the card says "automobile" and you find nothing, even though the book is right there on the shelf. Ask "drug interactions for metformin" and you get efficacy studies, because "interaction" and "effect" sit close in embedding space — the real answer is filed under "contraindications," a word you never used.

MAPPING: Basic RAG is that card catalogue. Advanced RAG adds more ways to find the same shelf: exact-word search, better cutting, a second opinion on relevance, and multiple phrasings of your question.

The four failure modes

1. Semantic drift"Side effects of drug X?" pulls efficacy chunks — nearby in meaning, but wrong. Fix: hybrid BM25 + dense.
2. Exact-match miss"FDA approval #123-456" fails; dense vectors can't encode rare IDs. Fix: BM25 lexical search.
3. Context split"Compare Q1 vs Q2" spans a page boundary; fixed chunks cut it apart. Fix: parent-child chunking.
4. Query ambiguity"Python memory issues" retrieves one facet, misses related framings. Fix: multi-query expansion.

Diagnose then fix

IF retrieval grabs the wrong topic:
  → add hybrid (BM25 + dense)

IF exact IDs / codes never match:
  → add BM25 lexical search

IF the answer spans chunk edges:
  → use parent-child chunking

IF the query is broad / ambiguous:
  → expand into multiple queries

# each fix targets ONE failure —
# stack them, don't guess

Hybrid search alone lifts recall@5 by 12–18%; adding a re-ranker lifts precision@5 by a further 15–22%.

Misconceptions

Where you cut decides what you find

Chunking splits a long document into pieces that get embedded and indexed. The goal: chunks small enough to retrieve precisely but large enough to hold one complete thought.

Naive fixed-size chunking optimizes for neither — it cuts every N characters whether or not a sentence, paragraph, or section boundary falls there. Everything downstream (search, re-ranking, generation) is capped by how well you chunked. Four strategies trade speed for coherence.

The newspaper scissors problem

BEFORE: Imagine clipping articles from a newspaper with scissors that cut every 200 characters regardless of where you are on the page.

PAIN: You slice mid-sentence, through a photo, chopping the headline off the body. When you try to file the clippings, "Drug Recall" ends up filed apart from the drug's actual name — and search on words can't reassemble your mangled fragments.

MAPPING: Chunking strategy is where you put the scissors down. Sentence-aware cuts at periods; semantic cuts when the topic shifts; recursive respects the document's own structure. The right strategy depends on the document type.

Four strategies

1. Fixed + overlapCut every N chars with an M-char overlap. Fast, no NLP, but cuts mid-word.
2. Sentence-awareSplit at sentence boundaries, group until a target size. Never cuts mid-sentence.
3. SemanticEmbed consecutive sentences, cut where cosine distance spikes (a topic shift). Most coherent, slowest.
4. RecursiveTry paragraph breaks first, then newlines, then sentences, then spaces. Respects structure hierarchy.

Match strategy to document

IF docs are uniform (DB rows)
   OR speed > quality:
  → fixed-size + overlap

IF prose / tickets / descriptions:
  → sentence-aware

IF dense technical text,
   topics shift within a page:
  → semantic (accept slower + variable size)

IF structured docs (markdown, legal):
  → recursive (respects \n\n > \n > .)

Semantic chunking is 10–50x slower than fixed-size on large corpora — reach for it only when coherence really matters.

Misconceptions

Two searches, fused into one

Dense embeddings find meaning but blur exact tokens like product codes into neighborhood similarity. BM25 (Best Match 25) finds exact words, weighting rare terms heavily, but is blind to synonyms.

Hybrid search runs both, then merges their ranked lists with Reciprocal Rank Fusion (RRF). Each retriever "votes," and a document scoring 1/(k+rank) in each list gets its votes summed. The result covers both semantic and exact-match queries with one pipeline.

Ranked-choice voting

BEFORE: Suppose you trust only one search engine to pick the best result — whatever it ranks #1 wins, full stop.

PAIN: A single engine has blind spots. Dense search crowns a semantically-close-but-wrong doc; lexical search crowns an exact-word match that misses the intent. One big vote from one voter is fragile.

MAPPING: RRF is ranked-choice voting. Each retriever casts a ballot (its ranked list), and a vote is worth more the higher the doc placed. The payoff: a doc landing mid-pack in both lists (two modest votes) beats a doc that's #1 in only one list. Agreement across two methods is a stronger signal than a high score from either alone.

The fusion pipeline

1. Run BM25Score docs by term frequency × rarity; take the top fetch_k as a ranked list.
2. Run denseEmbed the query, query the vector store; take its top fetch_k ranked list.
3. Score with RRFFor each doc in each list add 1/(k+rank), with k=60 smoothing.
4. Sort & take top_kDocs appearing in both lists rise; return the merged top_k to re-ranking.

Pseudocode

FUNCTION hybrid_search(query, k=60):
  bm25_ranked  = bm25.top(query, fetch_k)
  dense_ranked = vectors.top(query, fetch_k)

  scores = {}
  FOR list IN [bm25_ranked, dense_ranked]:
    FOR rank, doc IN enumerate(list, 1):
      scores[doc] += 1 / (k + rank)

  RETURN top_k(sort_desc(scores))

# a doc in BOTH lists collects two
# votes and outranks a lone #1.

Misconceptions

A precise second opinion

Hybrid search hands you ~20 good candidates using a fast bi-encoder (query and doc embedded separately, compared by dot product). Great for recall, mediocre for fine-grained relevance.

A cross-encoder re-ranker reads the query and each document together and outputs one relevance score per pair. It's ~10x more accurate but can't pre-index, so you only run it on the small candidate set — recall wide, then re-rank narrow.

Resume screen, then interview

BEFORE: Imagine hiring by reading 10,000 resumes in full, one by one, giving each a deep interview. Accurate, but you'd never finish.

PAIN: Deep evaluation doesn't scale to the whole corpus. But a fast keyword skim of every resume (the bi-encoder) is cheap yet shallow — it lets weak candidates through and ranks them crudely.

MAPPING: So you do both. The fast skim (hybrid retrieval) narrows 10,000 to a shortlist of 20. Then the cross-encoder gives those 20 a real interview — reading query and document side by side — and picks the true top 5. Cheap recall first, expensive precision last.

The re-rank step

1. Take the candidatesFeed the ~20 docs from hybrid retrieval into the re-ranker.
2. Pair with the queryBuild (query, doc) pairs so the model attends across both texts at once.
3. Score jointlyThe cross-encoder emits one relevance logit per pair in a single batch.
4. Sort, keep top 5Order by score, drop the rest. Only these reach the prompt.

Which re-ranker?

IF production, latency budget ~120ms:
  → cross-encoder (ms-marco-MiniLM)
     67MB, CPU, near-SOTA quality

IF domain-specific relevance rules
   (clinical, legal) + offline batch:
  → LLM re-ranker via Mistral
     (~2–8s for 20 docs, custom logic)

IF latency is critical (< ~10ms):
  → skip re-ranking, use bi-encoder
     order (baseline quality)

Cross-encoder scores are unbounded logits — use them only for ordering, never as probabilities.

Misconceptions

Ask the question five ways

Multi-query expansion asks the LLM to generate N alternative phrasings of the query, retrieves for each independently, then deduplicates the combined pool before re-ranking.

If the query is "memory leak in Python," useful docs might say "garbage collection," "reference counting," or "high RSS" — none containing your phrase. Variant queries surface all of them into the candidate pool, so retrieval casts a wider net before re-ranking narrows it back down.

Asking around the office

BEFORE: Imagine looking for a document by asking exactly one coworker, using exactly the words that first came to mind.

PAIN: If you ask for "the churn report" but the team files it as "retention analysis," that one coworker shrugs — and you conclude it doesn't exist, even though it does.

MAPPING: Multi-query is asking several coworkers, each in different words: "churn," "retention," "cancellations," "attrition." Between them, someone recognizes the request. You then pool everything they hand back and drop the duplicates — one wide search built from many phrasings.

Expand, retrieve, dedup

1. Generate variantsAsk Mistral for 3–5 rephrasings using different vocabulary and framings.
2. Retrieve eachRun the full search for every variant, plus the original query.
3. Pool resultsCombine all hits into one candidate list, tagging their source variant.
4. Dedup by embeddingDrop near-duplicates by cosine similarity, not text — it catches paraphrases.

Pseudocode

FUNCTION multi_query(query, n=3):
  variants = llm_expand(query, n)   # temp 0.5
  variants = [query] + variants     # keep original

  pool = []
  FOR v IN variants:
    pool += search(v)

  # dedup by EMBEDDING, not text —
  # catches paraphrase duplicates
  RETURN dedup_by_similarity(pool, 0.95)

Temperature 0.4–0.7 is the sweet spot: too low gives near-identical variants, too high drifts off-topic.

Misconceptions

Keep the sentence, drop the page

A retrieved chunk might be 500 tokens where only 40 answer the query. Feeding all 500 into context wastes the model's attention and your token budget.

Contextual compression asks the LLM to extract only the portion of each chunk that directly answers the query, discarding the rest. A 10-chunk context of tight excerpts can outperform a 3-chunk context of full passages — less noise, more room, better answers.

The highlighter, not the whole book

BEFORE: Imagine answering a colleague's question by photocopying and handing them ten entire book chapters that each mention the topic somewhere.

PAIN: They now have to re-read hundreds of pages to find the two sentences that matter. The signal is buried in noise, and they may latch onto an irrelevant paragraph instead.

MAPPING: Compression is running a highlighter over each chapter first and handing over only the highlighted lines. Same source material, a fraction of the reading — and far less chance of getting distracted by the wrong passage.

Compress each chunk

1. Skip short chunksIf a chunk is already under ~150 chars, return it as-is — nothing to gain.
2. Extract the excerptAsk the LLM for the verbatim portion relevant to the query.
3. Handle "not relevant"If the model finds nothing useful, it returns a marker and the chunk is dropped.
4. Fall back safelyIf compression returns empty or errors, keep the full chunk rather than lose it.

Compress or not?

USE compression WHEN:
  - chunks are long (> 400 tokens)
  - questions are specific
  - context window is tight

SKIP compression WHEN:
  - chunks already short (< 150 tokens)
  - question is broad, needs full context
  - real-time latency is critical

# heuristic: if avg compression < 50%,
# the extra LLM call isn't worth it

Compression adds one LLM call per chunk — that latency multiplies fast across a candidate set.

Misconceptions

Match small, return big

Small chunks retrieve precisely; large chunks give the model enough context to answer coherently. You can't have both from one index — unless you keep two.

Parent-child (small-to-big) chunking indexes small child chunks (50–150 tokens) for precise matching, each linked to a larger parent chunk (300–600 tokens). When a child matches the query, you return its parent to the LLM. Precision on retrieval, richness on generation.

The book index vs. the chapter

BEFORE: Suppose you had to find and understand a fact using only a book's back-of-book index — single precise entries, no surrounding text.

PAIN: The index entry "lactic acidosis, p.212" pinpoints the location perfectly, but reading just that line tells you nothing — no cause, no severity, no context. Precision without context is useless.

MAPPING: Parent-child search uses the precise index entry (the child chunk) to find the spot, then hands you the whole page (the parent chunk) to actually read. You match on the needle but return the paragraph around it.

Two collections, one link

1. Build parentsSplit the doc into large 300–600 token parent chunks; index them for lookup.
2. Build childrenSplit each parent into small children; store the parent_id in each child's metadata.
3. Search childrenQuery the small-chunk index for precise matches, fetch a few extra.
4. Return parentsFollow each match's parent_id, dedupe, and return the parent text to the model.

Pseudocode

# INDEX TIME
FOR parent IN split(doc, ~500):
  store_parent(parent)
  FOR child IN split(parent, ~120):
    index_child(child, parent_id=parent.id)

# QUERY TIME
FUNCTION search(query):
  hits    = child_index.top(query)
  parents = unique(h.parent_id FOR h IN hits)
  RETURN fetch_parents(parents)

Misconceptions

Test the core insight

One question per concept. Tap to reveal the answer.