Which vector database is best for RAG?

On most projects this is not the decisive choice. Up to a few hundred thousand chunks, the vector extension in the database you already run is enough and spares you a second system to operate. The real differences come from chunking, ranking and measurement.

Would a bigger model fix it?

No. If the wrong passage was retrieved, a bigger model summarises the wrong passage more fluently. When retrieval is broken, scaling the model raises cost, not accuracy.

At what point do I actually need RAG?

Total length matters more than document count. If all your content fits in the model's context window at an acceptable cost, skip RAG — passing it directly is simpler and more accurate. Once the archive outgrows that limit, a retrieval layer becomes unavoidable.

← All notes

Enterprise retrieval

Six common mistakes when building RAG

RAG grounds a language model's answers in passages pulled from your own documents. Standing one up takes a week; making it trustworthy takes months. What separates the two is rarely the model — it is how documents are split, what gets retrieved and how the result is measured.

Published: 2026-08-28
Built by: Zero-Shot Team
3 min read

1. Splitting documents blindly

The most common setup cuts text into fixed-length chunks and stores each one separately. That produces tables severed mid-row, clauses detached from their headings, and provisions with no indication of which contract they belong to.

Splitting should follow the document's own structure: headings, clauses, tables, sections. Simply prefixing each chunk with its document name and section heading raises accuracy visibly, without changing the model at all.

2. Relying on semantic search alone

Vector search captures meaning but misses exact matches. Search for a product code, a clause number or an abbreviation and you get something semantically close and factually wrong.

The fix is to run both: keyword search and vector search together, with the results merged and reranked. This single change is the largest one-off accuracy gain we see on enterprise archives.

3. Answering without citations

An answer with no source is an unverifiable claim. Even when the system is right, trust never forms, because the user has no way to spot the times it is wrong.

Every answer should carry a clickable reference to the document and section it came from. There is a second benefit: diagnosing a bad answer takes minutes, because you can see exactly which passage was retrieved.

4. Treating permissions as an afterthought

In a corporate archive, not everyone may see every document. If the permission check is applied after generation, the damage is already done: the model has read and summarised content it should never have seen.

Filtering has to happen during retrieval, bound to the user's identity. This is not a security detail but a first-day architectural decision — bolting it on later usually means rebuilding.

5. Letting the index go stale

Documents change; the index does not. Six months in, the system confidently returns wrong answers, because its sources are accurate but outdated.

A refresh path belongs in the first release: changed documents reprocessed, deleted documents removed from the index. Showing each passage's date in the answer also lets users catch staleness themselves.

6. Shipping without measurement

This is both the most common and the most expensive mistake. The system looks good on a handful of demo questions, goes live, and then meets real ones.

A hundred questions collected from real users, with correct answers and correct sources written by hand, is enough. Measure two things separately: was the right passage retrieved, and was the right answer produced from it. Without that separation you can never tell which half is broken.

In short

Split along the document's structure, and carry its origin in every chunk.
Run keyword and vector search together.
Cite a clickable source on every answer — for trust and for debugging.
Filter by permission during retrieval, not after generation.
Measure retrieval and generation separately; a hundred questions is enough.

Common questions

Which vector database is best for RAG?: On most projects this is not the decisive choice. Up to a few hundred thousand chunks, the vector extension in the database you already run is enough and spares you a second system to operate. The real differences come from chunking, ranking and measurement.
Would a bigger model fix it?: No. If the wrong passage was retrieved, a bigger model summarises the wrong passage more fluently. When retrieval is broken, scaling the model raises cost, not accuracy.
At what point do I actually need RAG?: Total length matters more than document count. If all your content fits in the model's context window at an acceptable cost, skip RAG — passing it directly is simpler and more accurate. Once the archive outgrows that limit, a retrieval layer becomes unavoidable.

More notes

All notes →