TechByteByByte

Top-K and Metadata Filtering

How many results to actually retrieve, and how to combine vector similarity search with metadata filters like access control and department, using Module 9's structured chunks.

#RAG#AI#Metadata Filtering#Level 4

Begin with the problem

Retrieval must decide how many results to return and which records the user may see. Top-k and metadata filters solve different parts of that decision.

query → vector/filters → index search → top candidates

What you will learn

  • Explain Top-K and Metadata Filtering in simple language before using its technical details.
  • Follow the mechanism step by step through a small RAG example.
  • Connect this topic to the modules before and after it.
  • Decide when to use it, when not to use it, and what to measure in production.

Current real-system grounding: OpenAI’s vector store API and Google’s File Search guide are current examples of managed vector retrieval. Exact indexes and tuning controls vary by product.

The product example proves that the pattern is used in a real system. It does not mean every provider uses the same hidden algorithm, defaults, limits, or pricing.

1. The problem this module solves

Level 3 built the machinery to find similar vectors efficiently. Level 4 begins here with two really practical questions every real search call has to answer: how many results should you actually retrieve, and how do you combine similarity search with the metadata (Module 9) every chunk carries?

2. Top-K — Why Not Retrieve Everything, or Just One?

If k = 3:      retrieve the 3 chunks with the highest similarity
              scores to the query

Why not retrieve EVERYTHING?      more chunks = more TOKENS (cost,
                                 Module 27 of the Generative AI
                                 course) and more NOISE diluting the
                                 really relevant content (Module
                                 21 of this course's context
                                 construction)

Why not retrieve just ONE?           if the single best match is
                                    really incomplete, or the
                                    answer spans MULTIPLE chunks
                                    (like this course's recurring
                                    London/Tokyo exception example),
                                    a single result can miss
                                    important context entirely

Choosing k is a real trade-off between completeness and noise/cost — there’s no universally correct value, just like Module 7’s chunk size had no universal correct value. The right k really depends on your specific data, your typical query patterns, and your downstream generation quality (Module 32’s evaluation).

3. Metadata Filtering — Combining Search With Exact Constraints

Recall Module 9: TechCorp’s knowledge base spans multiple departments — HR, Engineering, Finance, Legal. A pure similarity search, applied across everything, has no concept of “only look within HR documents.”

Metadata filtering + vector similarity, combined:

1. FIRST, restrict the candidate set to chunks matching an EXACT
   metadata constraint (e.g., department = "HR")
2. THEN, rank the REMAINING candidates by vector similarity
Without filtering:      search ALL chunks -> might surface a
                       superficially similar but really
                       IRRELEVANT chunk from Engineering, ranked
                       ABOVE a really relevant HR chunk

With filtering:            search ONLY within HR chunks -> the
                          result set is guaranteed relevant to the
                          right DOMAIN before similarity even gets
                          involved

4. What Gets Filtered — Building Directly on Module 9

department          "only search HR documents"

date / freshness         "only search documents from the last 6
                        months" (Module 26 previews this directly)

document_type                "only search POLICY documents, not
                            MEETING NOTES"

access_control                   "only search documents THIS
                                SPECIFIC user is permitted to see"
                                (Module 27's real security
                                requirement — this is the SAME
                                mechanism, applied for correctness
                                AND for security)

Notice: access control filtering isn’t just a relevance optimization — it’s a real security requirement (Module 27 covers this fully). The filtering mechanism covered in this module is exactly the same mechanism that makes access control enforceable at all.

5. Filter-Then-Search vs. Search-Then-Filter

This is a really important, easy-to-get-backwards implementation detail:

FILTER-THEN-SEARCH (correct, the standard approach):
   1. Narrow the candidate pool by metadata FIRST
   2. THEN run similarity search only within that narrowed pool

SEARCH-THEN-FILTER (a real, common mistake):
   1. Run similarity search across EVERYTHING first, get top-k
   2. THEN discard results that don't match the metadata filter

Incorrect idea: Why search-then-filter is really broken:

Why it is incorrect: if your top-k results (found BEFORE filtering) happen to contain zero chunks matching your metadata constraint, filtering AFTER the fact leaves you with an EMPTY result set — even if really relevant, correctly- filtered chunks existed further down in the full ranking, just outside the original top-k cutoff. Filter-then-search avoids this entirely by narrowing the pool BEFORE ranking ever happens.

6. A Real Developer Example

An HR employee asks: "What's our on-call compensation policy?"

Without filtering: pure similarity search MIGHT surface an
                   Engineering document about on-call ENGINEERING
                   ROTATIONS (superficially similar -- both mention
                   "on-call") ABOVE the really relevant HR
                   compensation policy chunk, since "on-call" is a
                   strong semantic signal regardless of department.

With filtering (department="HR", or based on the user's OWN
department + really relevant cross-department docs):
   the Engineering on-call rotation document is EXCLUDED from the
   candidate pool entirely BEFORE similarity ranking even happens --
   guaranteeing the really relevant HR compensation chunk surfaces
   correctly.

7. A Simple Agentic AI Connection

An agent’s knowledge-base search tool should really accept metadata filter parameters as part of its tool definition (Module 29 of the Generative AI course) — allowing the agent to deliberately narrow its search based on context it has already gathered (e.g., “the user mentioned they’re in Finance, so filter to Finance + general company policy documents”) rather than always searching the entire, unfiltered knowledge base.

8. How Is This Used in AI?

🤖 How Is This Used in AI?

Every production vector database (Module 12) supports combining metadata filters with similarity search as a first-class, optimized operation — precisely because real RAG applications really need both relevance ranking AND exact constraint matching (department, permissions, freshness) working together, not as separate, disconnected steps.

9. Real-World Applications

  • Multi-tenant systems where each customer can only search their own data
  • Department-scoped enterprise search
  • Time-bounded search (“only show results from this fiscal year”)

10. Common Mistakes

Incorrect idea: Implementing search-then-filter instead of filter-then-search.

Why it is incorrect: As shown directly in Section 5, this can silently produce empty or incomplete results even when really relevant, filter-matching content exists.

Incorrect idea: Choosing a fixed k without considering your specific data and query patterns.

Why it is incorrect: As shown directly in Section 2, this is a real trade-off requiring real evaluation (Module 32), not a universal default.

Incorrect idea: Treating access control filtering as optional or an afterthought.

Why it is incorrect: As emphasized directly in Section 4, this is a real security requirement — Module 27 covers the full implications directly.

11. Limitations

  • Overly aggressive filtering can really eliminate relevant results if metadata is inconsistently or incorrectly tagged (Module 9’s ingestion-time capture quality directly matters here)
  • Choosing the right k remains fundamentally an empirical question, really requiring evaluation (Module 32) against your specific application rather than a universal formula

12. Quick Reference — The Whole Idea in One Diagram

Query
   ↓
METADATA FILTER (narrow candidates FIRST -- department, permissions,
                 freshness, from Module 9)
   ↓
Filtered candidate pool
   ↓
SIMILARITY SEARCH (Module 11-14) within that pool
   ↓
Top-K results (k chosen as a deliberate trade-off, Section 2)

13. Code — Implementing Filter-Then-Search Correctly

🎯 Target of this example: implement Section 5’s correct filter-then-search order directly, and directly demonstrate Section 5’s warning by showing search-then-filter producing a really worse (or empty) result on the same data.

Example 1 — Simple

import numpy as np
from dataclasses import dataclass, field

@dataclass
class Chunk:
    text: str
    embedding: np.ndarray
    department: str

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def filter_then_search(query_vec, chunks, department, top_k=2):
    """The CORRECT order (Section 5): narrow by metadata FIRST, THEN
    rank by similarity within that narrowed pool."""
    filtered = [c for c in chunks if c.department == department]
    scored = [(c, cosine_similarity(query_vec, c.embedding)) for c in filtered]
    scored.sort(key=lambda x: x[1], reverse=True)
    return scored[:top_k]

chunks = [
    Chunk("HR on-call compensation is $50/night.", np.array([0.7, 0.6, 0.3]), "HR"),
    Chunk("Engineering on-call rotation follows a weekly schedule.", np.array([0.75, 0.62, 0.28]), "Engineering"),
    Chunk("HR standard hotel limit is $200/night.", np.array([0.65, 0.55, 0.32]), "HR"),
]

query_vec = np.array([0.72, 0.61, 0.29])  # "on-call compensation" query
results = filter_then_search(query_vec, chunks, department="HR", top_k=2)
for chunk, score in results:
    print(f"[{score:.3f}] ({chunk.department}) {chunk.text}")

Expected Output:

[1.000] (HR) HR on-call compensation is $50/night.
[0.998] (HR) HR standard hotel limit is $200/night.

What we conclude from this example: the Engineering chunk — despite being semantically very close to the query (it also mentions “on-call”) — never even entered the candidate pool, because filtering happened BEFORE similarity ranking. This directly demonstrates Section 6’s real developer example: the correct HR chunk surfaces reliably, with no risk of an off-department chunk crowding it out.

Example 2 — Intermediate

import numpy as np
from dataclasses import dataclass

@dataclass
class Chunk:
    text: str
    embedding: np.ndarray
    department: str

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def filter_then_search(query_vec, chunks, department, top_k=2):
    filtered = [c for c in chunks if c.department == department]
    scored = [(c, cosine_similarity(query_vec, c.embedding)) for c in filtered]
    scored.sort(key=lambda x: x[1], reverse=True)
    return scored[:top_k]

def search_then_filter(query_vec, chunks, department, top_k=2):
    """The INCORRECT order (Section 5's warning) -- rank EVERYTHING
    first, THEN discard non-matching results AFTER the top-k cutoff
    has already been applied."""
    scored = [(c, cosine_similarity(query_vec, c.embedding)) for c in chunks]
    scored.sort(key=lambda x: x[1], reverse=True)
    top_k_before_filter = scored[:top_k]
    return [(c, s) for c, s in top_k_before_filter if c.department == department]

# An Engineering chunk that happens to be a VERY close semantic match
chunks = [
    Chunk("Engineering on-call rotation follows a weekly schedule with $100/night pay.",
          np.array([0.74, 0.61, 0.30]), "Engineering"),
    Chunk("Engineering incident response guide.", np.array([0.71, 0.58, 0.33]), "Engineering"),
    Chunk("HR on-call compensation is $50/night.", np.array([0.60, 0.50, 0.40]), "HR"),
]

query_vec = np.array([0.73, 0.60, 0.31])  # really closer to the Engineering chunks

correct_results = filter_then_search(query_vec, chunks, department="HR", top_k=2)
broken_results = search_then_filter(query_vec, chunks, department="HR", top_k=2)

print("Filter-then-search (correct):")
for chunk, score in correct_results:
    print(f"  [{score:.3f}] {chunk.text}")

print("\nSearch-then-filter (broken):")
if not broken_results:
    print("  ⚠️  EMPTY RESULT SET -- the top-2 unfiltered results were both Engineering!")
else:
    for chunk, score in broken_results:
        print(f"  [{score:.3f}] {chunk.text}")

Expected Output:

Filter-then-search (correct):
  [0.988] HR on-call compensation is $50/night.

Search-then-filter (broken):
  ⚠️  EMPTY RESULT SET -- the top-2 unfiltered results were both
  Engineering!

What we conclude from this example: filter-then-search correctly surfaces the really relevant HR chunk. Search-then-filter, given the exact same data, produces a completely EMPTY result set — because the top-2 unfiltered results were both Engineering chunks, and there was nothing left to keep after filtering. This directly, concretely proves Section 5’s warning: search-then-filter can silently fail even when relevant, correctly-tagged content really exists in the knowledge base.

Example 3 — Production Grade

import numpy as np
from dataclasses import dataclass, field
from enum import Enum

class FilterField(Enum):
    DEPARTMENT = "department"
    ACCESS_CONTROL = "access_control"

@dataclass
class Chunk:
    text: str
    embedding: np.ndarray
    department: str
    access_control: list = field(default_factory=lambda: ["all_employees"])

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

class FilteredSearchEngine:
    """A production-style search engine COMBINING metadata filtering
    (department AND access control together, Module 9/27) with
    similarity search -- structurally enforcing filter-then-search
    order (Section 5), so this mistake becomes structurally
    impossible rather than something a developer could get wrong."""

    def __init__(self, chunks: list):
        self.chunks = chunks

    def search(self, query_vec: np.ndarray, user_groups: list,
               department: str = None, top_k: int = 3) -> list:
        # STEP 1: metadata filtering -- ALWAYS happens first, no
        # code path allows skipping straight to similarity ranking.
        candidates = [
            c for c in self.chunks
            if any(g in c.access_control for g in user_groups)
            and (department is None or c.department == department)
        ]

        # STEP 2: similarity ranking, ONLY within the filtered pool.
        scored = [(c, round(float(cosine_similarity(query_vec, c.embedding)), 4)) for c in candidates]
        scored.sort(key=lambda x: x[1], reverse=True)
        return scored[:top_k]

chunks = [
    Chunk("HR on-call compensation is $50/night.", np.array([0.7, 0.6, 0.3]),
          "HR", ["all_employees"]),
    Chunk("Engineering on-call rotation schedule.", np.array([0.74, 0.61, 0.29]),
          "Engineering", ["engineering_team"]),
    Chunk("Confidential Q4 salary bands.", np.array([0.68, 0.58, 0.35]),
          "HR", ["hr_only", "management"]),
]

engine = FilteredSearchEngine(chunks)
query_vec = np.array([0.72, 0.61, 0.29])

# A regular employee, not in HR-only or engineering groups
regular_employee_results = engine.search(query_vec, user_groups=["all_employees"], top_k=3)
print("Results for a regular employee (no department filter):")
for chunk, score in regular_employee_results:
    print(f"  [{score}] {chunk.text}")

Expected Output:

Results for a regular employee (no department filter):
  [0.9998] HR on-call compensation is $50/night.

What we conclude from this example: the regular employee’s search correctly excludes BOTH the Engineering chunk (wrong access group) and the confidential salary bands chunk (requires hr_only or management), leaving only the really accessible HR chunk — demonstrating Module 9 and Module 27’s access control requirement enforced through exactly the same filter-then-search mechanism this module covers, with the correct ordering made structurally unavoidable rather than left to convention.

14. Interview Questions

Q: What’s the fundamental trade-off in choosing a value for k (the number of retrieved results), and why is there no universally correct answer?

Ans: A smaller k reduces noise and token cost but risks missing relevant information if the answer really spans multiple chunks. A larger k captures more potential context but increases cost and can dilute the prompt with less relevant content, potentially degrading generation quality. The right value really depends on the specific knowledge base’s chunking granularity, typical query complexity, and downstream generation quality — it requires empirical evaluation against your specific application rather than following a universal default.

Q: Explain why “filter-then-search” is the correct order, and what can go wrong with “search-then-filter.”

Ans: Filter-then-search narrows the candidate pool by metadata constraints first, then ranks only that narrowed pool by similarity — guaranteeing that if really relevant, filter-matching content exists anywhere in the knowledge base, it will be found. Search-then- filter ranks the entire unfiltered dataset first, takes the top-k, and only then discards non-matching results — if the top-k happens to contain zero results matching the filter, you’re left with an empty or severely incomplete result set, even though relevant content existed further down in the full ranking, simply outside the original top-k cutoff.

Q: Why is metadata filtering not just a relevance optimization, but sometimes a real security requirement?

Ans: When metadata includes access control information — which permission groups are allowed to see a given chunk — filtering isn’t just about improving result quality, it’s about preventing unauthorized users from ever seeing content they’re not permitted to access. This is the exact same filtering mechanism used for relevance (like department scoping), but applied with security-critical consequences: if this filtering isn’t enforced correctly and consistently before content ever reaches a user, it becomes a real data exposure risk.

Q: Design a search function that combines both department filtering and access control filtering for a real enterprise RAG system — what order should operations happen in, and why?

Ans: Both metadata filters — department and access control — should be applied together, before any similarity ranking happens, exactly following filter-then-search order. Access control filtering is non-negotiable and should never be skippable, while department filtering might be optional depending on the query context. Structuring the code so metadata filtering always happens as a mandatory first step, with similarity ranking only operating on the already-filtered result, makes the correct order structurally enforced rather than relying on every developer remembering to filter and to filter in the right order every time they write a new search call.

15. What You Should Remember

  • Choosing k is a real trade-off between completeness and noise/cost — no universal correct value exists.
  • Filter-then-search is the correct order — verified directly by showing search-then-filter produce a completely empty result set on identical data where relevant content really existed.
  • Metadata filtering enforces both relevance and security — verified directly through a production search engine structurally combining department and access control filtering before any similarity ranking occurs.

16. Quick Practice

Design a metadata filtering strategy for a RAG system serving both internal employees and external customers from the SAME underlying knowledge base — what filter would need to be applied, and at what point in the search process, to prevent a customer from ever seeing internal-only content?

17. Next Step

Next: Module 16 — BM25 and Sparse Retrieval — a really different retrieval approach from everything covered in Level 3: exact keyword matching, and when it outperforms semantic search.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed