RAG Hybrid Search: Reranking for Reliable Retrieval

Vector search finds passages that sound relevant and misses the policy number, error code or clause the user typed. Keyword search does the reverse. Running both helps only if the pipeline fuses the two rankings sensibly, caps reranking cost, keeps access filters on both branches, and shows on your own queries that the extra machinery paid for itself.

Diagnose retrieval before changing it

Look at the retrieved passages before the model's answer. A weak answer can come from retrieval, prompt construction, generation or missing source data, and hybrid search only touches the first. Label each failed query with one category:

• Exact identifier: ticket IDs, product codes, names, dates, quoted phrases, error strings.
• Paraphrase: different words from the source, same concept.
• Mixed constraint: a concept plus an exact entity, date or document type.
• Ambiguous: several documents could match.
• Missing corpus: the passage was never indexed or was parsed badly.
• Authorization exclusion: the passage exists but the caller may not see it.

Only the first three are hybrid-search problems. No fusion method recovers a document that is absent, forbidden or unidentifiable. Collect about 50 labeled queries, including a few that should return nothing, and record the expected source passage for each. That set is your fixture for everything below.

Microsoft's hybrid search overview describes the basic shape: full-text and vector queries run in parallel and their result lists are merged.

Build two independent candidate generators

Each branch returns stable chunk and document IDs, its native rank and score, and the metadata needed for authorization and debugging. Give the lexical branch normal BM25 ranking with field boosts, because titles and headings often deserve more weight than body text. Leave fuzzy matching off at first. Exact terms are the reason the branch exists, and broad fuzziness floods the candidate set with near matches.

Use the embedding version from indexing time on the vector branch, and apply the same tenant, department, date and document-state filters to both. A lexical hit that skips a filter is an authorization defect, whatever it does for recall. Start with 30 candidates per branch and log latency, candidate counts and whether the expected passage appeared.

Fusion: reciprocal rank fusion is not the only default

Raw BM25 and cosine scores live on different scales, so fusion has to avoid comparing them directly. Reciprocal Rank Fusion does that by scoring each candidate as the sum of 1 / (k + rank) over the lists it appears in. Elastic's reference documents it.

The most useful comparison is Bruch, Gai and Ingber's analysis of fusion functions for hybrid retrieval. They report that RRF is sensitive to its parameters, that a convex combination of normalized scores outperforms it both in-domain and out-of-domain, and that the single weight in that combination can be tuned from a small set of labeled examples. You already have such a set. Rank-only fusion also throws away how decisive a match was: a BM25 hit that beats every other candidate by a wide margin and one that wins by a hair contribute the same.

The weighted route normalizes each branch over its own candidate pool and then fuses, for example 0.7 vector plus 0.3 lexical. That normalization has a catch. Weaviate's documentation, whose default hybrid fusion has been relative score fusion since v1.24, warns that with a small search limit the result set can be very sensitive to the limit parameter, because the scaling depends on which candidates are in the pool (Weaviate search operators, oversearch). Fetch more candidates than you return before you fuse, and check that the same query with limit 5 and limit 50 agrees on the top five. Stamp a version string for the weights on traces and response metadata, so a bad ranking can be tied to the exact weights that produced it. Hand-set weights are defaults, not tuned numbers, and a reranker is the next step if they stop being enough.

Pick one fusion method, freeze its parameters, and compare it against the other on your fixtures before touching reranking. Whichever you choose, keep a trace per fused item: lexical rank, vector rank, each contribution, final rank and the filters applied. "Search got worse" then becomes a statement about which stage dropped the passage.

Decide how to handle duplicates before you tune. If two chunks of one document fill the top results, either cap chunks per document before reranking or fuse at document level and restore the best passages afterward. Test the rule on multi-section documents.

Rerank a bounded candidate set

A reranker compares the query to each shortlisted passage directly, which is slower than retrieval and can lift a passage that holds the answer above ones that only share vocabulary. The sequence is: retrieve both branches in parallel, fuse and deduplicate, send the first N to the reranker, and give the model the top K.

Elastic measured how deep to go in its reranking depth analysis. On its benchmark mix, reranking one third of the maximum-gain depth captured about 90% of the possible nDCG@10 gain, which works out to roughly 100 BM25 candidates on average, and the authors suggest the top 30 when cost is tight. If your fixtures show the expected passage almost always lands in the top 30 of the fused list, the extra depth buys latency and nothing else. Stronger first-stage retrievers need fewer candidates, so measure rather than copy a number.

Set a rerank deadline. When it expires, return the fused order and record the fallback in the trace. Reranker output must be a subset of the IDs you sent it, and any unknown ID should be rejected. A ranking model decides order only. Authorization belongs to the filters.

Test the stages separately

Run four configurations on the same fixtures: lexical only, vector only, fused, fused plus rerank. Check recall at the candidate boundary first (top 10, top 20, the rerank set), then ranking quality with reciprocal rank or nDCG, then answer quality using only the context each configuration produced. Anthropic's contextual retrieval report follows the same pattern of measuring each retrieval change on its own, though its numbers belong to its corpora, not yours.

Report results per query category. A global average can hide a regression on exact identifiers behind a gain on paraphrases. Also test the negative cases: a forbidden document never reaches either branch, the reranker input, the logs or the model context; a superseded document does not return from a stale index; a query with no support returns nothing or a low-confidence result; a reranker timeout still answers inside the deadline.

Tune one layer at a time

Fix parsing, indexing and authorization first. Then set lexical-only and vector-only baselines, add fusion with fixed parameters, and widen branch depth only if the expected passage misses the fused shortlist. Add the reranker after that, then tune the final context size against answer quality and citation coverage. Later stages cannot restore candidates an earlier stage dropped, so a larger reranker is wasted on a pipeline whose candidate generation misses the answer.

Keep the old retriever behind a flag for the canary. Keep a single branch on any route where the second one adds nothing measurable. Different corpora and query classes can justify different policies.

References

• Microsoft Azure AI Search hybrid-search overview for parallel lexical and vector retrieval with merged results.
• Elastic Reciprocal Rank Fusion reference for rank-based fusion over independent retrievers.
• Bruch, Gai and Ingber, An Analysis of Fusion Functions for Hybrid Retrieval for RRF parameter sensitivity and the convex-combination comparison.
• Elastic, Semantic reranker: selecting optimal reranking depth for the one-third-depth, 90% gain finding.
• Anthropic contextual retrieval for evaluating embeddings, BM25 and reranking as separate steps.
• Weaviate: hybrid search operators: relative score fusion is the default since v1.24 and is sensitive to a small search limit.

---