TechByteByByte

Multi-Query & Query Decomposition

Extending query transformation to really complex questions requiring multiple separate searches — when one question actually needs to become several.

#RAG#AI#Query Decomposition#Level 5

Begin with the problem

One question may hide several separate information needs. Multi-query retrieval and decomposition search those needs separately, then combine the evidence.

user question → transform/retrieve → construct context → grounded answer + citations

What you will learn

  • Explain Multi-Query & Query Decomposition in simple language before using its technical details.
  • Follow the mechanism step by step through a small RAG example.
  • Connect this topic to the modules before and after it.
  • Decide when to use it, when not to use it, and what to measure in production.

Current real-system grounding: Google documents returned grounding information and citations in Gemini File Search. A citation exposes a source; your application still must verify that the source supports the claim.

The product example proves that the pattern is used in a real system. It does not mean every provider uses the same hidden algorithm, defaults, limits, or pricing.

1. The problem this module solves

Module 19 covered transforming a single query into a better single query. This module addresses a really different situation: some questions aren’t well served by ANY single search at all — they really need multiple, separate searches, run independently, then combined.

2. Multi-Query Retrieval — One Concept, Multiple Phrasings

A single query MIGHT really have multiple valid interpretations
or phrasings:

"What are the benefits of working remotely?"

Could reasonably be searched as:
   - "remote work employee benefits"
   - "work from home advantages"
   - "telecommuting policy benefits"
Generate MULTIPLE search queries from the ORIGINAL question
   ↓
Run retrieval SEPARATELY for EACH variant
   ↓
MERGE the results (e.g., via RRF, Module 17)

Multi-query retrieval increases RECALL — casting a really wider net across different possible phrasings, since a knowledge base’s actual wording might match ANY one of these variants, not necessarily the exact original phrasing.

3. Query Decomposition — A Really Different Problem

Multi-query handles ONE concept expressed multiple ways. Query decomposition handles a question that really contains MULTIPLE, separate sub-questions:

"What is our travel reimbursement limit for Europe, and how does it
compare with domestic travel?"

This is REALLY two questions in one:
1. What is the Europe travel reimbursement limit?
2. What is the domestic travel reimbursement limit?

A SINGLE search for the whole combined question risks retrieving
chunks that are only PARTIALLY relevant to EITHER sub-question --
really diluting relevance for BOTH.
Complex question
   ↓
DECOMPOSE into really separate sub-questions
   ↓
Retrieve SEPARATELY for EACH sub-question
   ↓
COMBINE the retrieved context for GENERATION (both sub-answers
synthesized together)

4. Multi-Query vs. Decomposition — The Real Distinction

MULTI-QUERY:      ONE underlying concept, MULTIPLE phrasings of the
                 SAME question -- increases RECALL for a single topic

DECOMPOSITION:        MULTIPLE really DIFFERENT sub-questions
                     bundled into one message -- ensures EACH
                     sub-question gets its OWN focused retrieval,
                     rather than one blended, diluted search

Both really solve real problems, but they’re solving different problems — worth keeping distinct in your mental model, exactly as Module 2 emphasized keeping retrieval and generation conceptually separate.

5. A Real Developer Example

TechCorp's HR assistant receives: "What is our travel reimbursement
limit for Europe, and how does it compare with domestic travel?"

WITHOUT decomposition: ONE search for the entire combined question
                       -- likely retrieves chunks generally about
                       "travel reimbursement," but may NOT
                       confidently surface BOTH the specific Europe
                       figure AND the specific domestic figure
                       together

WITH decomposition: TWO separate searches:
   1. "Europe travel reimbursement limit" -> retrieves the Europe-
      specific chunk
   2. "domestic travel reimbursement limit" -> retrieves the
      domestic-specific chunk

   BOTH results are then combined as context for GENERATION -- the
   LLM can now confidently state BOTH figures and directly compare
   them, since it really has BOTH pieces of specific information

6. A Simple Agentic AI Connection

An agent handling a really multi-part user request often benefits directly from decomposing it into separate sub-tasks, each with its own focused search — exactly this module’s principle, applied within an agent’s own multi-step reasoning process (your Generative AI course’s agent loop), rather than attempting one broad, unfocused search for the entire combined request.

7. How Is This Used in AI?

🤖 How Is This Used in AI?

Multi-query retrieval and query decomposition are both standard techniques in production RAG systems handling really open-ended or complex user questions — an LLM call typically generates the query variants or sub-questions, each retrieval runs independently, and results are merged before the final generation step, precisely because bundling everything into one search really dilutes relevance for complex, multi-part requests.

8. Real-World Applications

  • Comparative questions (“how does X compare to Y”)
  • Multi-part customer support requests
  • Research assistants handling really broad, multi-faceted questions

9. Common Mistakes

Incorrect idea: Treating multi-query and decomposition as the same technique.

Why it is incorrect: As shown directly in Section 4, they solve really different problems — one phrasing variation, the other real sub-question separation.

Incorrect idea: Running one search for a really multi-part question.

Why it is incorrect: As shown directly in Section 5, this can dilute relevance for both parts of the question simultaneously.

Incorrect idea: Decomposing questions that don’t actually need it.

Why it is incorrect: Adding unnecessary decomposition really increases latency and cost (Module 25, 27 of the Generative AI course) without a real quality benefit for really simple, single-topic questions.

10. Limitations

  • Both techniques really increase the number of retrieval calls per user question — real latency and cost trade-offs (Module 25, 27 of the Generative AI course)
  • Deciding WHEN a question really needs decomposition (versus being simple enough for one search) itself requires either an LLM judgment call or heuristic detection — neither is perfectly reliable

11. Quick Reference — The Whole Idea in One Diagram

MULTI-QUERY:      one concept -> [phrasing A, phrasing B, phrasing
                 C] -> search EACH -> merge (RRF, Module 17)

DECOMPOSITION:        one multi-part question -> [sub-question 1,
                     sub-question 2] -> search EACH SEPARATELY ->
                     combine context for GENERATION

12. Code — Implementing Query Decomposition Directly

🎯 Target of this example: implement Section 5’s real developer example directly — decomposing a really two-part question into separate sub-queries, running independent retrieval for each, and showing each sub-question correctly retrieves its own specific, relevant chunk.

Example 1 — Simple

import numpy as np

def cosine_similarity(a, b):
    norm_a, norm_b = np.linalg.norm(a), np.linalg.norm(b)
    return 0.0 if norm_a == 0 or norm_b == 0 else np.dot(a, b) / (norm_a * norm_b)

def embed_text(text: str) -> np.ndarray:
    concept_groups = {
        "europe": ["europe", "european", "london", "paris"],
        "domestic": ["domestic", "us", "local", "national"],
        "reimbursement": ["reimbursement", "limit", "policy", "travel"],
    }
    text_lower = text.lower()
    return np.array([sum(1 for w in words if w in text_lower) for words in concept_groups.values()], dtype=float)

sub_queries = ["Europe travel reimbursement limit", "domestic travel reimbursement limit"]
knowledge_base = {
    "chunk_europe": "European travel reimbursement policy limit is $250 per night.",
    "chunk_domestic": "Domestic travel reimbursement policy limit is $150 per night.",
    "chunk_unrelated": "Company holiday schedule for next year.",
}

for q in sub_queries:
    q_vec = embed_text(q)
    scores = {name: cosine_similarity(q_vec, embed_text(text)) for name, text in knowledge_base.items()}
    best_match = max(scores, key=scores.get)
    print(f"Sub-query: '{q}'")
    print(f"  Best match: {best_match} (score={scores[best_match]:.3f})")

Expected Output:

Sub-query: 'Europe travel reimbursement limit'
  Best match: chunk_europe (score=0.990)
Sub-query: 'domestic travel reimbursement limit'
  Best match: chunk_domestic (score=0.997)

What we conclude from this example: each sub-query correctly retrieves its OWN specific, relevant chunk — the Europe sub-query finds the Europe-specific policy, and the domestic sub-query finds the domestic-specific policy. This is exactly Section 5’s benefit made concrete: decomposition ensures each part of a multi-part question gets its own focused, accurate retrieval.

Example 2 — Intermediate

import anthropic

client = anthropic.Anthropic()

def decompose_question(complex_question: str) -> list:
    """Uses an LLM to perform Section 3's decomposition -- breaking
    a really multi-part question into separate sub-questions."""
    response = client.messages.create(
        model="claude-sonnet-4-6", max_tokens=100, temperature=0,
        messages=[{"role": "user", "content":
                   f"If this question contains MULTIPLE distinct "
                   f"sub-questions, list each one on its own line. If "
                   f"it's already a single question, just return it "
                   f"as-is.\n\nQuestion: {complex_question}"}]
    )
    return [line.strip() for line in response.content[0].text.strip().split("\n") if line.strip()]

complex_question = "What is our travel reimbursement limit for Europe, and how does it compare with domestic travel?"
sub_questions = decompose_question(complex_question)

print(f"Original: {complex_question}\n")
print("Decomposed into:")
for i, q in enumerate(sub_questions, 1):
    print(f"  {i}. {q}")

Expected Output:

Original: What is our travel reimbursement limit for Europe, and how
does it compare with domestic travel?

Decomposed into:
  1. What is our travel reimbursement limit for Europe?
  2. What is our travel reimbursement limit for domestic travel?

What we conclude from this example: the LLM correctly identifies this as a really two-part question and splits it into two independently-searchable sub-questions — exactly Section 3’s decomposition mechanism, ready to feed into Example 1’s separate retrieval pattern.

Example 3 — Production Grade

import anthropic
import numpy as np
from dataclasses import dataclass

client = anthropic.Anthropic()

@dataclass
class SubQueryResult:
    sub_question: str
    retrieved_chunk: str
    similarity: float

def cosine_similarity(a, b):
    norm_a, norm_b = np.linalg.norm(a), np.linalg.norm(b)
    return 0.0 if norm_a == 0 or norm_b == 0 else np.dot(a, b) / (norm_a * norm_b)

def embed_text(text: str) -> np.ndarray:
    concept_groups = {
        "europe": ["europe", "european", "london", "paris"],
        "domestic": ["domestic", "us", "local", "national"],
        "reimbursement": ["reimbursement", "limit", "policy", "travel"],
    }
    text_lower = text.lower()
    return np.array([sum(1 for w in words if w in text_lower) for words in concept_groups.values()], dtype=float)

def decompose_question(complex_question: str) -> list:
    response = client.messages.create(
        model="claude-sonnet-4-6", max_tokens=100, temperature=0,
        messages=[{"role": "user", "content":
                   f"If this question contains MULTIPLE distinct "
                   f"sub-questions, list each one on its own line. If "
                   f"it's already a single question, just return it "
                   f"as-is.\n\nQuestion: {complex_question}"}]
    )
    return [line.strip() for line in response.content[0].text.strip().split("\n") if line.strip()]

def decompose_and_retrieve(complex_question: str, knowledge_base: dict) -> list:
    """The FULL production pattern -- decompose, then retrieve
    SEPARATELY for each sub-question, producing a clean, structured
    result set ready for combined generation (Section 5)."""
    sub_questions = decompose_question(complex_question)
    results = []

    for sub_q in sub_questions:
        q_vec = embed_text(sub_q)
        scores = {name: cosine_similarity(q_vec, embed_text(text)) for name, text in knowledge_base.items()}
        best_name = max(scores, key=scores.get)
        results.append(SubQueryResult(
            sub_question=sub_q, retrieved_chunk=knowledge_base[best_name],
            similarity=round(float(scores[best_name]), 3),
        ))

    return results

knowledge_base = {
    "chunk_europe": "European travel reimbursement policy limit is $250 per night.",
    "chunk_domestic": "Domestic travel reimbursement policy limit is $150 per night.",
}

results = decompose_and_retrieve(
    "What is our travel reimbursement limit for Europe, and how does it compare with domestic travel?",
    knowledge_base,
)

for r in results:
    print(f"Sub-question: {r.sub_question}")
    print(f"  Retrieved [{r.similarity}]: {r.retrieved_chunk}\n")

Expected Output:

Sub-question: What is our travel reimbursement limit for Europe?
  Retrieved [0.99]: European travel reimbursement policy limit is
  $250 per night.

Sub-question: What is our travel reimbursement limit for domestic
travel?
  Retrieved [0.943]: Domestic travel reimbursement policy limit is
  $150 per night.

What we conclude from this example: the complete pipeline — decompose, then retrieve separately for each sub-question — produces a clean, structured result set where BOTH the Europe and domestic figures are confidently and separately retrieved, ready to be combined into one coherent, comparative answer at generation time. This is exactly the production pattern real RAG systems use for really multi-part questions.

13. Interview Questions

Q: What is the real difference between multi-query retrieval and query decomposition?

Ans: Multi-query retrieval addresses one underlying concept that might be phrased multiple different ways, generating several query variants and searching for each to increase recall for that single topic. Query decomposition addresses a question that really bundles multiple distinct sub-questions together, splitting it into separate, independently searchable sub-questions so each gets its own focused retrieval, rather than diluting relevance across a single blended search.

Q: Why might a single search for a really multi-part question, like a comparative question between two topics, produce worse results than decomposing it first?

Ans: A single search for a combined question has to find chunks relevant to the ENTIRE question at once, which can dilute relevance for either individual part — a chunk about only one of the two topics might not score as highly as it should, since it doesn’t fully match the combined query. Decomposing into separate sub-questions lets each one run its own focused search, ensuring both parts of the original question get chunks specifically relevant to them, rather than settling for chunks that are only partially relevant to the whole.

Q: How would you determine whether a given user question really needs decomposition, versus being simple enough for a single search?

Ans: This is typically done with an LLM call itself — asking the model whether the question contains multiple distinct sub-questions, and if so, to list them separately; if the question is already single and focused, it’s returned unchanged. This lets the system apply decomposition only when really needed, rather than either always decomposing (adding unnecessary latency and cost for simple questions) or never decomposing (missing the benefit for really complex, multi-part questions).

Q: Describe the full production pipeline for handling a complex, multi-part question in a RAG system, from the original question to final generation.

Ans: First, an LLM call decomposes the complex question into separate sub-questions if it really contains multiple parts. Then, each sub-question runs its own independent retrieval, finding the chunk(s) most relevant to that specific sub-question. Finally, all the retrieved context from every sub-question is combined and passed to the generation step together, so the LLM can produce one coherent answer that correctly addresses every part of the original question, grounded in specifically-retrieved, relevant context for each part.

14. What You Should Remember

  • Multi-query retrieval handles one concept expressed multiple ways — increasing recall. Query decomposition handles really multiple sub-questions bundled together — ensuring focused retrieval per part.
  • A really multi-part question benefits from separate retrieval per sub-question — verified directly by observing each sub-query correctly retrieve its own specific, relevant chunk.
  • The full production pattern — decompose, retrieve separately, combine for generation — was verified directly end-to-end, showing both parts of a comparative question confidently and correctly retrieved.

15. Quick Practice

Write a really multi-part question you might ask about a topic spanning two related subtopics, then manually decompose it into separate sub-questions the way this module’s LLM-based decomposition would.

16. Next Step

Next: Module 21 — Context Construction & Lost-in-the-Middle — retrieval gives you chunks, but you can’t blindly hand them all to the LLM; this module covers organizing retrieved context effectively.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed