Picture a support engineer pasting an error code into your product's search box. Three articles come back that are broadly about errors, and the one page with that exact string in its title sits somewhere below the fold. If that sounds plausible for your product, start by reading your last thousand search queries and sorting them by result count. That single export tells you more about your AI search than any model comparison will.
Why Does Vector-Only AI Search Miss Exact Matches?
Vector search retrieves documents whose meaning resembles the query, and it has no built-in idea of an exact string. So error codes, invoice numbers and version strings are precisely where it goes wrong.
Aaditya Chauhan's June 2026 InfoQ article on hybrid retrieval opens with a production failure of this kind. An on-call engineer searched for the runbook to enable a feature flag and was given guidance to disable it, because the two near-identical runbooks were indistinguishable to the embedding model. Chauhan's summary is that embeddings are "excellent at finding semantically similar content, but systematically weak at distinguishing specific entities like version numbers, error codes, and feature flag names."
This is an old result, not a new one. The 2021 BEIR benchmark evaluated retrieval models zero-shot on 18 datasets and found BM25 to be a strong baseline, with dense retrievers often underperforming other approaches. Now list what people type into a B2B SaaS product's search box: error codes copied from logs, invoice numbers, version strings where v3.2 and v3.1 both exist and only one is right, customer names, function names. Each is an exact-match query dressed up as a natural-language one.
I'd treat that as a design constraint. It rarely shows up as an outage. It shows up later as a support ticket that says search never finds anything.
How Do You Combine Keyword and Vector Search?
Run BM25 and vector retrieval in parallel on the same query, then merge the two ranked lists with Reciprocal Rank Fusion, which scores documents by their position in each list instead of by raw relevance numbers.
Rank-based fusion sidesteps the problem that ruins hand-rolled hybrid search: BM25 scores and cosine similarities live on different scales, so a BM25 score of 14.2 and a cosine of 0.83 can't be averaged into anything meaningful. Elastic's RRF documentation gives the formula, adding 1 / (k + rank) for each list a document appears in, with a default rank constant k of 60. It also states that RRF requires no tuning and that the relevance indicators don't have to be related to each other.
You may not need to build the fusion yourself. Qdrant supports hybrid queries that combine sparse and dense vectors, with RRF available since v1.10.0. If you're on Postgres, the pgvector README recommends using it with Postgres full-text search for hybrid retrieval, and it names Reciprocal Rank Fusion and cross-encoders as ways to combine the results.
Fusion is a default, not a law. Chauhan notes in the same article that for a pure exact-match query such as an error-code constant, BM25 alone produces a cleaner result than hybrid with RRF, while hybrid still improves relevance across the wider query mix. If most of your traffic is identifiers, route those queries lexically and keep fusion for the natural-language remainder. That is my recommendation, and your logs should confirm or overturn it.
Consider a product where a fifth of searches are pasted identifiers. Routing those to keyword search alone protects them from dense near-misses, and everything else still gets the fused ranking. The routing rule can be as simple as a regular expression matching your identifier formats, which costs nothing to run and is easy to test against your log.
Does Reranking Justify the Latency?
Reranking usually justifies its cost when you apply it to a shortlist of tens of documents rather than hundreds, because it gives a second, more careful look at candidates the cheap stages already found.
A reranker takes the query and a candidate together, so it can judge how the question and the passage relate, which first-stage retrieval can't do. Cohere describes its Rerank endpoint this way: given a query and a list of documents, it orders the documents from most to least semantically relevant. That is too slow to run over a whole corpus, which is why it belongs at the last stage of the pipeline. I haven't benchmarked latency across vendors, so I'm not quoting millisecond figures. Measure the added latency on your own hosting and shortlist size.
A useful test is to compare the top ten before and after reranking on a small set of labelled queries. If the reranker only reshuffles results that were already right, you're paying for nothing. If it rescues answers from positions eleven to fifty, it earns its place.
| Reranker | Billing unit (verified 30 September 2026) | Context and limits |
|---|---|---|
| Cohere Rerank 3.5 | $5.00 per 1,000 searches on the listed tier; one search is one query with up to 100 documents | Documents over 500 tokens are split into chunks that each count |
| Voyage rerank-3 | $0.05 per million tokens | 32,000-token context, up to 1,000 documents per request |
| Voyage rerank-3-lite | $0.02 per million tokens | 32,000-token context |
The sources are Cohere's pricing page, Voyage's pricing and its reranker documentation, which also says Voyage counts tokens as query tokens times document count plus all document tokens. The two billing shapes behave differently, so compare them on your own chunk sizes. Take a hypothetical shortlist of 50 chunks at 300 tokens each with a 20-token query: that is about 16,000 tokens, or roughly $0.0008 on rerank-3, against $0.005 per search on Cohere's listed rate. Change the chunk length and the ranking of the two flips.
What a whole query costs
Embedding is the line people fear and it is rarely the problem. OpenAI's pricing page lists text-embedding-3-small at $0.02 per million tokens and text-embedding-3-large at $0.13. A hypothetical 50,000-document corpus averaging 800 tokens is 40 million tokens, which is about $0.80 on the small model. A query is perhaps 20 tokens, so query-time embedding is negligible. The recurring cost is the reranker, and it scales with shortlist size instead of corpus size, so halving the shortlist halves that line.
The cost people leave out is re-embedding. Change the embedding model and you re-index the corpus. Change the chunking and you do it again. Budget engineer time for a few full re-indexes in the first year.
In What Order Should You Build AI Search?
Build the lexical leg first, add vectors only when the logs show paraphrase queries going unanswered, and add the reranker last. Each step tells you whether the next one is worth its cost.
- Read your search logs before you write code. Export the last thousand queries and sort by result count, then read the empty ones.
- Get chunking right and then stop touching it. Chunk on document structure, keep headings attached to their body, and remember every change forces a re-index.
- Ship keyword search first. BM25 in Postgres full-text search or your search engine is your quality floor and your fallback when a model provider is down. The PostgreSQL docs describe the tsvector and tsquery types and ranking.
- Add the dense leg, ideally in the same store, and fuse with RRF at the default k of 60.
- Build a set of about fifty labelled query and document pairs from your own logs. Fifty is enough to catch a regression and small enough to maintain.
- Add the reranker on the top fifty candidates and measure the quality gain against the added latency. If the gain is small, look at your fusion first.
- Add query rewriting only if the logs show unanswered semantic queries, because it puts another network call in front of every search.
The counts (fifty, fifty) are my rules of thumb, not published thresholds. Below a few thousand documents with mostly exact-match queries, a well-configured Postgres full-text index may beat a rushed hybrid pipeline, and it ships sooner.
Filter by tenant before you rank
In a multi-tenant product, apply tenant and role filters before ranking, not after. My Callidus case study describes a multi-tenant clinic SaaS built on React, Firebase and Stripe Connect, with data under a tenant path in Firestore and six roles kept consistent through shared role helpers. It doesn't describe an AI search feature, so read this as a design principle, not a result. A search index is one more read path, and access rules that live in one place are far easier to keep correct than rules copied into a ranking layer.
Late interaction is worth knowing about
ColBERT-style late interaction keeps token-level vectors and matches at query time, which is more precise and much heavier to store. Qdrant's multivector tutorial says multivectors are typically used in the reranking stage after a first pass with dense vectors, so you can disable HNSW indexing for them by setting the m parameter to 0. I'd leave it until your precision needs justify the memory.
If you're still choosing infrastructure, my SaaS MVP stack covers what pairs well with what, and which AI features are worth adding to a business product helps you decide whether search is the right first one.
Your next step is small. Export a thousand queries, sort by result count, and read the empty ones yourself. How many would plain keyword search have answered?