Most of the RAG systems I have been asked to look at did not have a retrieval problem. They had a scope problem. Someone read that general-purpose models invent details when asked about private data, reached for a vector database, and shipped a pipeline that answers questions nobody asks against documents that stopped being true in March.
RAG earns its place in a SaaS product when three conditions hold at the same time: the corpus is genuinely too large to paste into a prompt, it changes faster than any retraining cycle could follow, and a confidently wrong answer costs you a support ticket or a renewal. Drop one of those and something cheaper will do the job.
The reason this is worth writing down now rather than two years ago is that the first condition moved under everyone's feet. Claude Opus 4.6 and Sonnet 4.6 both carry million-token context windows, and Fabio Akita's April 2026 write-up on the end of the mandatory vector database puts 200,000 tokens at roughly 150,000 words. That is Crime and Punishment, in one request. A document pile that demanded a retrieval pipeline in 2024 now fits in the prompt with room left over.
What Does RAG Actually Buy a SaaS Product?
RAG buys grounded answers over data the model has never seen, with citations to the source, and no retraining when that data changes.
Everything else people credit it with either follows from that or does not happen at all. It does not make the model smarter. It does not teach it your product's voice. It narrows what the model is permitted to look at, and it hands the user a link they can check.
That second half matters more in B2B than most founders expect. A consumer chatbot that guesses gets a shrug and a thumbs-down. A B2B SaaS product that guesses at a customer's contract terms gets an email from their legal team. The citation is the feature. Retrieval is just how you produce one.
The running economics are unremarkable until you scale. Embedding is cheap: text-embedding-3-small runs $0.02 per million tokens and text-embedding-3-large $0.13, with the Batch API halving both in exchange for a 24-hour turnaround. Storage is cheap. Digital Applied's May 2026 cost model prices a single RAG query at roughly $0.0056 once embedding, retrieval, an optional rerank and generation are added together. None of those line items is what hurts. The pipeline around them is, and it never appears on an invoice.
The Vector Database Was Never the Point
The vector database was a workaround for context windows that could not hold your document, and for a large share of B2B corpora that constraint expired last year.
Akita puts the trade in blunt numbers. Building the pipeline costs 40 to 80 engineering hours before a single user query returns anything, call it $1,600 to $3,200 of someone's time, and from then on you own chunking, embedding, index tuning and a refresh job that nobody wants to be on call for. Skip it and a 200,000-token query costs $0.63 on Sonnet 4.6 or $3.15 on Opus 4.6, falling to roughly $0.10 per follow-up once prompt caching warms up.
Run those two against each other for a real product. A knowledge base of a few hundred pages serving a few hundred queries a day will never pay back the pipeline's build cost, let alone its maintenance. At ten thousand queries a day across half a million documents the arithmetic flips hard, and retrieval wins by a distance. Which side of that line is your product on? Most teams I ask have never worked out the number, which is how a six-week infrastructure project gets approved on vibes.
Actually, that understates one thing. Cost is not the only axis. A million-token prompt is slow to first token in a way users notice, and a retrieval step that returns eight passages keeps the answer snappy even when the corpus is small. Latency can justify RAG on its own, at corpus sizes where the money says otherwise.
When Should a SaaS Team Skip RAG Entirely?
Skip it when the corpus fits in a prompt, when the answer lives in one database row, or when nobody has asked the question yet.
The first case is now much wider than it was. Product documentation, a changelog, a policy handbook, an onboarding guide: individually these are tens of thousands of tokens, not millions. Load the lot, cache the prefix, move on.
The second case is the one that embarrasses people. "How many seats are left on my plan" is a SELECT statement. Routing it through an embedding model, a similarity search and a language model is an expensive way to lose precision on a question your database answers exactly. I covered the broader version of this when I wrote about which AI features in a SaaS business actually pay for themselves, and retrieval sits in the same bucket: worth it when the alternative is a human reading documents, not worth it when the alternative is a WHERE clause.
The third case is the expensive one, because it looks like diligence. Teams build retrieval before they know which questions users ask, then discover that seventy percent of real queries were about billing and could have been handled by four canned answers and a link. Ship the search box first. Log what people type into it. The log is your requirements document, and it costs nothing.
RAG, Fine-Tuning, or Just Prompting
Three approaches, three different cost shapes. The mistake is comparing them on accuracy alone, because they fail at different times and for different reasons.
| Approach | Build cost | Time to first version | Handles data that changes | Best fit |
|---|---|---|---|---|
| Just prompting | Hours of someone's afternoon | Same day | Yes, immediately, as long as the data fits the window | Small stable corpora, anything under a few hundred pages, every first version |
| RAG | £5,000–£40,000 for a mid-market build, then a refresh job forever | 1–3 weeks | Yes, at the price of a re-embed cycle | Large corpora, frequent updates, citation requirements, per-tenant isolation |
| Supervised fine-tuning | £10,000–£60,000, plus the labelling behind it | 4–8 weeks | No; a knowledge refresh costs 10x to 100x what the same update costs under RAG | Fixed tone, fixed output schema, regulated formats, or distilling a large model at volume |
Those build ranges come from Dr. Phil Winder's June 2026 decision framework, and the number that reframes the whole comparison is buried in his fine-tuning column. LoRA compute on a 7B to 13B model costs $50 to $500. That is nothing. The expense is the training data, at roughly one to two weeks of experienced engineering time per thousand labelled examples, which is why a fine-tuning project runs four to eight weeks while a RAG build runs one to three. Winder gives RAG the win in about 80% of cases, and I would not argue with that split.
Fine-tuning still takes two categories outright. The first is form rather than fact: a fixed tone, a rigid output schema, a regulated document format. The second is distillation, where you train a small model to match a frontier model on one narrow task at roughly a tenth of the inference cost, and the savings clear the training bill within weeks at production volume. Digital Applied puts that crossover at around a million queries a month. Below it, you are optimising a bill you do not have yet.
Pure Vector Search Loses on the Queries B2B Users Actually Type
Semantic similarity is the wrong instrument for an exact string, and B2B support search is largely made of exact strings.
Tian Pan, an engineer who previously worked at Uber and Brex, catalogues the query classes where dense embeddings systematically underperform: error codes like ERR_SSL_VERSION_OR_CIPHER_MISMATCH, product SKUs where RTX-4090 and RTX-4070 sit as near-neighbours in vector space despite being different products, and fully qualified function names. The signal from a rare token gets averaged across the whole vector until it stops being distinctive.
Tuesday afternoon, a support lead pastes 0x80070005 into your search box and gets back three articles about general troubleshooting. Nothing throws. Nothing is slow. Your dashboards stay green. The answer is simply wrong, and the only person who knows is the one who gave up and opened a ticket.
The fix is not exotic, and it is not free either. Pan's targets for a healthy hybrid system are recall@10 between 85% and 91% with MRR above 0.80, and he is direct about what it takes to get there: an Elasticsearch benchmark from 2025 he cites saw a naive reciprocal-rank-fusion hybrid add only about 1.3% NDCG over a plain BM25 baseline, while a tuned tiered approach reached 7.5%. Fusion weights are a job, not a checkbox. Worth remembering too that dense models have been losing to BM25 on zero-shot cross-domain retrieval since the original BEIR paper in 2021, which is not a new result anyone should be surprised by.
If you are already on Postgres you do not need a second database for any of this. pgvector for the dense side, a tsvector index for the lexical side, one fusion query on top, and the whole thing inherits the row-level security policies you already wrote instead of forcing you to reimplement tenant isolation somewhere new. I wrote up the pattern for React and Supabase with row-level security separately, and it transfers directly: the policy that scopes a tenant's rows scopes their embeddings too.
Chunking Is a Product Decision
Chunk size gets treated as a tuning parameter. It is closer to a product decision. Split a contract mid-clause and you will retrieve half an obligation. Split a changelog by release and every chunk stands on its own. Ask what a complete answer looks like in your domain, then make that the chunk. Tune afterwards.
How Often Do Embeddings Need Refreshing?
Refresh on a cadence set by how fast each kind of content decays, because staleness never announces itself in your metrics.
This is the failure mode I would put money on being live in most production RAG systems right now. Pan's write-up on stale embeddings reports production retrieval recall drifting from 0.92 to 0.74 with no deploy and no incident, previously top-ranked documents sliding from position 2 to position 8. Similarity scores stay high because the stale document is still semantically relevant. Latency stays flat. The damage is distributional, spread thin across hundreds of queries, so no single result looks broken enough to investigate.
His shelf-life framing is the useful part: API reference documentation goes stale in about two weeks, compliance documents in around six months, architectural overviews in one to two years. One refresh cadence across all three either burns money on the slow content or serves lies about the fast content.
Separate re-indexing from re-embedding while you are at it. Making a new document searchable is cheap and takes minutes. Regenerating vectors because the text changed, the chunking changed, or you upgraded the embedding model is the expensive operation, and it is the one people schedule carelessly. One team quoted in that piece spent $12,000 a month re-embedding a 1TB corpus weekly, on API calls alone, before any compute for chunking or index rebuilds.
Build a RAG Feature in This Order
None of this is specific to AI. It is the same discipline that makes any software-as-a-service product worth its subscription: ship the smallest thing that answers a real question, then measure before you build the next one. The order below is what I follow, and each step earns the right to the next.
- Ship a search box and log every query. No model, no embeddings. Two weeks of logs will tell you what your users actually want, and the answer is usually narrower and more boring than the roadmap assumed.
- Answer the top queries with a cached prompt. Load the relevant documents whole, cache the prefix, and see how far that gets you. For a lot of products this is the finished feature, and the reason I default to a simple, boring core in the stack I use for a SaaS MVP is that it survives contact with real usage.
- Add lexical search when exact strings appear in the logs. Postgres full-text search covers error codes, SKUs and identifiers, which is where dense retrieval would have quietly failed you.
- Add embeddings only when paraphrase queries show up. "How do I stop the sync breaking overnight" will not match on keywords. That is the query type embeddings exist for.
- Fuse the two and tune the weights against your own logs. Not against a public benchmark. Your queries are not BEIR.
- Instrument retrieval separately from generation. Log recall@10 and click position on retrieved passages. Without that, a slide from 0.92 to 0.74 is invisible until churn shows up in a board meeting.
Most teams start at step 4 and never build steps 1 or 6. That is why so many RAG features feel like they half-work, and why nobody on the team can say precisely how.
So before you provision a vector database this quarter, do one thing: export the last thousand searches your users ran and read them yourself. Count how many are exact strings, how many are questions page one of your docs already answers, and how many came back empty. What would you build differently if you knew that number today?
