When retrieval-augmented generation took off, vector databases went from niche to default-purchase almost overnight. Every RAG tutorial reaches for one. But a dedicated vector database is infrastructure with real operational cost, and a lot of teams adopt one reflexively when something simpler would do. Here's what they actually are and how to decide whether you need one.

What a vector database does

The job is narrow and specific. AI systems represent text (or images, or audio) as embeddings — long lists of numbers, vectors, where similar items sit close together in the vector space. To retrieve relevant content, you take your query's embedding and find the stored embeddings nearest to it. That "find the nearest vectors" operation is nearest-neighbour search, and doing it fast over millions of vectors is the entire reason vector databases exist.

The hard part is scale. Comparing a query against every stored vector is simple but slow once you have millions. Vector databases use approximate nearest-neighbour indexes — clever data structures that find almost the closest vectors very fast, trading a tiny bit of accuracy for a huge speed gain. That index is the core technology. Everything else (metadata filtering, storage, APIs) is supporting cast.

When you genuinely need one

A dedicated vector database earns its place when:

Scale is large. Once you're storing millions of vectors and querying them with low latency, you need a purpose-built index. This is the clearest case.

Latency matters and volume is high. Real-time retrieval at high query rates is exactly what these systems are optimised for.

You need rich filtering alongside similarity. Good vector databases let you combine "nearest vectors" with metadata filters ("...but only from documents tagged X, created after Y") efficiently.

If you're operating at serious scale with demanding latency, a dedicated vector database is the right tool and a clear win.

When you probably don't

Here's the part the hype skips. At modest scale — thousands to low hundreds of thousands of vectors — you may not need a dedicated vector database at all:

Your existing database may already do it. Many mainstream databases now support vector search natively or via an extension. If you're already running one, adding vector search to it keeps everything in one system — one thing to operate, back up, and reason about — instead of standing up and syncing a second specialised store. For a lot of teams this is the pragmatic best answer.

An in-memory index may be enough. For smaller corpora, a lightweight library that holds the index in memory can be plenty fast with near-zero operational overhead. No server, no service, no sync.

The numbers are smaller than you think. People dramatically overestimate how many vectors they have. A few thousand documents chunked is tens of thousands of vectors — trivially handled without specialised infrastructure.

The real decision

Strip away the hype and the decision comes down to a few honest questions:

• How many vectors, realistically? Thousands? Almost any approach works. Millions and up? You need a real index.
• What latency, at what query rate? Demanding real-time at high volume points to a dedicated system.
• One system or two? Every additional piece of infrastructure is operational cost — deployment, monitoring, backups, keeping data in sync. Adding vector search to a database you already run is often worth a small performance compromise to avoid running a second system.
• Do you need advanced features? Sophisticated hybrid search, fine-grained filtering, and managed scaling are where dedicated vector databases pull ahead.

The takeaway

Vector databases are excellent, specialised tools that are essential at scale. They are not a mandatory component of every RAG project. Start by asking how many vectors you actually have and what your latency and operational constraints really are. If the answer is "not that many" and "nothing extreme," the right move is often the boring one: use vector search in the database you already operate, and reach for dedicated infrastructure only when you've outgrown it. Buying the heavy machinery before you have the workload is a classic way to add complexity that earns nothing.