A support engineer pastes an error code into your product's search box and gets back three articles that are broadly about errors. The one document with that exact string in its title sits at rank eleven. Nobody scrolls to eleven.
That is not a tuning problem, and it is not a weak embedding model. It is embeddings working exactly as specified. AI search in SaaS goes wrong most often at the moment a team decides "semantic" is a synonym for "better" and quietly retires a keyword index that was doing half the job well.
The version that survives production has three moving parts and one budget line: two retrieval legs running in parallel, a fusion step that merges their rankings, and a reranker that re-sorts the shortlist before anything reaches the user. The embedding model, the choice everyone argues about on the way in, is the least consequential decision on that list.
Embeddings Find Similar, Not Same
Vector search retrieves documents whose meaning resembles the query. It has no concept of an exact string, and that is where it breaks on your own data.
Aaditya Chauhan's InfoQ analysis of hybrid retrieval, published in June 2026, has the cleanest example of this I have read. An on-call engineer searched for the runbook to enable a feature flag called payment_v2_enforce. The system returned the runbook to disable it. Both documents describe the same flag in near-identical language, so in embedding space they sit almost on top of each other. The negation is one token inside a passage of four hundred, and cosine similarity does not weight it the way a human reading the title would.
Now list what people actually type into a B2B SaaS product's search box. Error codes copied out of a log. Invoice numbers. Version strings, where v3.2 and v3.1 both exist and only one is right. Customer names. Function names. Every one of those is an exact-match query wearing the costume of a natural-language one, and every one of those is where a vector-only index returns something plausible instead of something correct.
It rarely surfaces as an outage. It surfaces six weeks later as a support ticket phrased "search never finds anything."
How Do You Combine Keyword and Vector Search?
Run BM25 and vector retrieval in parallel on the same query, then merge the two ranked lists with Reciprocal Rank Fusion. RRF scores by position rather than by raw relevance number.
It became the default because it sidesteps the problem that ruins hand-rolled hybrid search: BM25 scores and cosine similarities are not on the same scale and never will be. A BM25 score of 14.2 and a cosine of 0.83 cannot be averaged into anything meaningful. Fusing on rank ignores both magnitudes and asks only where each document landed in each list, which is why it needs no normalisation layer and still outperforms hand-weighted blending. Digital Applied's May 2026 hybrid search reference documents the standard rank constant of 60, inherited from Elasticsearch's production default, and notes that dropping toward 30 or 40 sharpens top-1 precision at some cost to top-10 recall.
The same reference collects Doug Turnbull's March 2025 evaluation on the WANDS e-commerce benchmark: BM25 alone reached 0.6983 NDCG, pure vector search 0.6953, and a tuned hybrid configuration 0.7497, a lift of roughly 7.4% over either leg alone. Sit with that for a second. Two methods scoring within half a percent of each other, combined, beat both by a margin users can feel.
Fusion is a default, not a law. Chauhan makes the opposite point in the same piece: for a raw error code pasted straight out of a log, BM25 alone produces a cleaner top-K than the hybrid result, because the dense leg drags in near-misses that dilute a perfect lexical match. If your product's search traffic is mostly identifiers, weight accordingly or route those queries lexically and skip fusion entirely.
Does Reranking Justify the Latency?
Usually yes, if you rerank a shortlist of fifty rather than five hundred, because that stage buys the biggest single quality jump in the pipeline. It costs a few hundred milliseconds.
A cross-encoder does something structurally different from your retrieval legs. Retrieval compares a query vector against document vectors computed months ago, independently, with no knowledge of each other. A reranker pushes the query and one candidate document through a model together, so it can see how the question and the passage interact. Far more accurate. Far too slow to run across a whole corpus, which is precisely why it belongs at stage three over a shortlist the cheap stages already narrowed.
| Reranker | Reported latency | What distinguishes it |
|---|---|---|
| Cohere Rerank 3.5 | ~595–603 ms | The strongest zero-effort default if you run no GPU infrastructure |
| Voyage rerank-2.5 | ~595–603 ms | 32K-token context; domain-tuned variants for code and legal corpora |
| Jina Reranker v3 | 188 ms | Listwise scoring over up to 64 documents, 81.33% Hit@1 |
| Nemotron reranker | 243 ms | 83.00% Hit@1 |
Latency and Hit@1 figures come from Particula's reranker comparison. The hosted numbers include a network round-trip, which explains most of the gap between the top two rows and the bottom two. Read that table as an argument about hosting, not about model quality.
The pricing models differ more than the models do. Voyage bills rerank-2.5 per token, at $0.05 per million; Cohere bills per search, in the range of a tenth to a quarter of a cent depending on tier. Which is cheaper depends entirely on how long your candidate documents are, so run your own numbers on your real chunk sizes before picking on price.
What Does AI Search Cost Per Query?
A well-built hybrid query costs fractions of a cent, with embedding nearly free, retrieval running on your own compute, and reranking dominating whatever the bill turns out to be.
Embedding the corpus is the line item people fear and it is almost never the problem. As of 2026, text-embedding-3-small runs $0.02 per million tokens and 3-large $0.13, with the Batch API halving both if you can wait 24 hours for an initial backfill. A 50,000-document corpus averaging 800 tokens per document is 40 million tokens: eighty cents on the small model, and forty on batch. That is the whole index.
Query-time embedding is smaller still, because a query is maybe 20 tokens. The reranker is the recurring cost, and it scales with shortlist size rather than corpus size, which is the single most useful thing to know when budgeting. Halving your shortlist from 100 to 50 halves that line. It rarely halves your quality.
Actually, let me back up, because I have skipped the cost nobody puts in the spreadsheet: re-embedding. Every time you change embedding models you re-index the entire corpus, and every time you change chunking strategy you do it again. Budget for three full re-indexes in year one. You will use them.
When You Should Not Build Any of This
Under about ten thousand documents with a mostly lexical query mix, a well-configured Postgres full-text index will beat a rushed hybrid pipeline on both quality and your calendar. Ship that. Instrument it. Come back when your logs show real semantic queries going unanswered, not before.
The Build Order That Works
Sequence matters more than tool choice here, because each step tells you whether the next one is worth building.
- Read your search logs before you write any code. Export the last thousand queries and sort by result count. The zero-result ones sit at the top, and the first twenty will usually be an invoice number, a customer name and a string of hex. That distribution decides how you weight your two legs.
- Get chunking right, then stop touching it. Chunk on document structure rather than a fixed token count, keep headings attached to the body they introduce, and remember that every change here forces a full re-index.
- Ship the lexical leg first. BM25 in Postgres, Elasticsearch, or whatever your database already offers. This is your quality floor and your fallback when the model provider has a bad afternoon.
- Add the dense leg into the same store. Qdrant, Weaviate, Elasticsearch and Milvus now all hold sparse and dense representations in one collection with native RRF fusion, so the separate vector database next to a separate search cluster is an architecture you can retire. If you are choosing infrastructure from scratch, my opinionated SaaS MVP stack covers what pairs well with what.
- Fuse with RRF at k=60, then measure. Build a set of 50 labelled query-document pairs from your own logs. Fifty is enough to catch a regression and small enough that you will actually maintain it.
- Add the reranker last, on the top 50. Measure the quality delta against the latency it costs. If the delta is small, your fusion weights are probably wrong and no reranker will fix that for you.
- Add query rewriting only if step 5 showed unanswered semantic queries. A small model expanding "how do I cancel" into the vocabulary your docs actually use is cheap. It is also a second network hop in front of every search, so it earns its place or it goes.
One more thing that gets skipped in multi-tenant products: filter by tenant before you rank, not after. If your app already routes access through a role matrix, search has to obey the same matrix. On Callidus, the multi-tenant clinic SaaS I built on React and Firebase, six roles with different access to clinical and financial records meant every read path went through shared role helpers rather than inline checks. A search index is a read path. Retrofitting isolation into one after launch is the same painful migration it is anywhere else, except now it is also a ranking bug.
What About Late Interaction?
ColBERT-style late interaction is worth knowing about and mostly not worth deploying yet. It represents a document as one vector per token and scores by taking the maximum similarity between each query token and every document token, which is more precise than a single dense vector and far heavier to store. Qdrant's own multivector documentation is refreshingly blunt about the trade: a single logical document becomes hundreds of token-level vectors, so you set the HNSW m parameter to 0 on that field and use it strictly as a rerank stage rather than for first-pass retrieval. Otherwise the RAM bill finds you.
That is the shape of the year. Sparse and dense in one store, fusion on rank, a cross-encoder on the shortlist, and late interaction waiting in the wings for teams whose precision requirements justify the memory. None of it depends on which embedding model you picked. If you are working out where a search feature sits among the other AI features worth adding to a business product, search is usually the one with the clearest before-and-after, because you can measure zero-result rate on day one and again on day thirty.
Before you provision anything, do step one. Export your last thousand queries, sort by result count, and read the empty ones yourself. How many of them would a keyword index have answered on its own — and what does that number say about the pipeline you were about to build?
If you are earlier than that and still working out how a subscription product should be structured, start with what the SaaS delivery model actually involves and come back to retrieval when you have a corpus worth searching.
