#Retrieval

After indexing, the online job is find the chunks the LLM should see. Default: dense top-k. If quality stalls, add filters, keyword hybrid, and optional rerank.

flowchart LR
  Q[question] --> E[embed]
  E --> R[retrieve top-k]
  R --> F[metadata filter]
  F --> RR[optional rerank]
  RR --> L[LLM]

Push filters into the store (Chroma where, pgvector WHERE, Qdrant query_filter). Do not fetch 1000 rows and drop them in Python.

#Choosing k

Bigger k is not better: noise fills the window and the model skips the one good paragraph. Freeze k, then change chunking using hit@k in Evaluation.

def format_hits(ids: list[str], docs: list[str], metas: list[dict]) -> str:
    lines = []
    for i, doc, meta in zip(ids, docs, metas):
        src = meta.get("source", "")
        lines.append(f"[{src} #{i}]\n{doc}")
    return "\n\n".join(lines)

#Metadata filters

“How many leave days does HR allow?” must not retrieve the engineering wiki. Write dept, lang, and source at index time; pass them at query time:

• Chroma: where={"dept": "hr"}
• pgvector: WHERE dept = 'hr' ORDER BY embedding <=> %s
• Qdrant: Filter(must=[FieldCondition(...)])

Filters are an access boundary, not a leaderboard trick. Tenant IDs must be enforced server-side; the model must not choose them.

#Hybrid search (vectors + keywords)

Dense vectors paraphrase well (“return” ≈ “refund”). Keywords win on clause IDs, SKUs, and error codes (KH-8842, ECONNRESET). A simple fuse: take top-n from each list, merge with Reciprocal Rank Fusion.

def rrf(rank_lists: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for ranks in rank_lists:
        for r, doc_id in enumerate(ranks, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + r)
    return sorted(scores, key=scores.get, reverse=True)


dense_ids = ["c3", "c1", "c9"]      # from the vector store
bm25_ids = ["c1", "c4", "c3"]       # inverted index or Postgres FTS
print(rrf([dense_ids, bm25_ids])[:4])

Real BM25 can be PostgreSQL full-text search or Qdrant sparse vectors (see the Qdrant teaser). Do not add a fancy hybrid before you measure hit@k.

#Optional rerank

Retrieve 20, score (question, document) pairs, send 4 to the LLM. A cross-encoder runs on CPU (no GPU required):

# pip install sentence-transformers
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-base")  # CPU OK, slower


def rerank(question: str, docs: list[str], top: int = 4) -> list[str]:
    scores = reranker.predict([(question, d) for d in docs])
    order = sorted(range(len(docs)), key=lambda i: float(scores[i]), reverse=True)
    return [docs[i] for i in order[:top]]

If you cannot load a model yet, lexical overlap is a teaching rerank (not a substitute):

def overlap_rerank(question: str, docs: list[str], top: int = 4) -> list[str]:
    q = set(question)
    scored = sorted(docs, key=lambda d: len(q & set(d)), reverse=True)
    return scored[:top]

LangChain / LlamaIndex / Dify rerank components do the same job: LangChain RAG, Dify Knowledge Base.

#Prompt constraints

Accurate retrieval still fails if the prompt invites improvisation. Keep three rules:

1. Answer only from the context below.
2. Do not invent facts missing from context (including numbers, IDs, dates).
3. If evidence is missing, say you do not know and list the source values you used.

Generation quality is measured as faithfulness in the next chapter.

#Next steps

• Evaluation
• Practical Examples
• Embeddings