Vector databases explained: the backbone of modern AI applications

Vector databases are the silent backbone of modern AI applications. They make semantic search and retrieval-augmented generation (RAG) practical in the first place. This article explains understandably how they work, which decisions really count when it comes to implementation – and why data protection is the question that is most often asked too late.

Why classic search reaches its limits

A conventional keyword search only finds what is exactly named. If someone searches for “termination period”, she will not find a hit if the document says “end of contract”. This is where many knowledge databases fail: The knowledge is available, but not findable. Anyone who has ever experienced how an employee searches for ten minutes on the intranet and then asks a colleague knows the problem from practice.

How Vector Databases Work

Instead of storing texts as pure strings, content is stored in so-called Embeddings transformed – Number vectors with typically several hundred to several thousand dimensions that represent the Significance of a text. Similar contents lie close together in the vector space. A search query is sent through the same model and the database provides the passages whose vectors are closest to it.

Two details of this are important for practice. First, the similarity is determined via a distance measure, usually cosine similarity – so “fits content” is a calculation variable, not an evaluation. Secondly, the database does not search exactly for speed reasons, but approximate (Approximate Nearest Neighbor, usually via an HNSW index). This is why the same request can occasionally deliver slightly different hits on very large stocks. This is not a problem for the search – you should only know it before you think it is a mistake.

What solution? Mostly the ones you already have

The market seems confusing, the decision is rare. There are roughly three ways:

• Extension of an existing database. pgvector turns PostgreSQL into a vector database. For the vast majority of medium-sized applications, this is completely sufficient – and no additional system comes into operation that wants to be secured, monitored and updated.
• Specialised vector databases like Qdrant, Milvus or Weaviate. They are worthwhile for very large stocks, high query rates or if filtering and vector search must be closely interlinked.
• Search engines with vector function like Elasticsearch or OpenSearch. Interesting if a full text search is running in the house anyway.

The honest rule of thumb: Below a few million text sections, the choice of database is not what determines quality. Those who discuss Milvus against Qdrant in fifty thousand documents are optimizing in the wrong place.

Chunking: the underestimated set screw

Because the actual quality comes before that. Documents must be broken down into sections before they become vectors – and how you cut determines significantly what is found later. Too small sections lose the connection: A paragraph in which “the deadline is 14 days” is written, without there being any indication of the deadline, is worthless as a hit. Too large sections dilute the vector, because an embedding over five pages means everything and nothing at some point.

What works in practice is based on the structure of the document instead of a fixed number of characters – sections along headings, with some overlap at the edges and with accompanying context, such as document titles and chapters in the section itself. This sounds banal and is the difference between a system that is used and one that is fallow after three weeks.

Why pure vector search alone is not enough

Semantic search is strong in meaning and weak in strings. If you are looking for the item number “TR-4471-B”, a file number or a surname, you want an exact hit – and this is where the vector search is unreliable, because such strings hardly have any meaning that an embedding could depict.

Therefore, productive systems combine both: Hybrid search from classical keyword search and vector search, whose results are merged and re-sorted. If you only rely on vectors, you build a system that shines with flow text and fails with master data. Where relationships between objects are more important than their description, it is also worth looking at Knowledge Graph and GraphRAG.

What Embeddings Reveal About Your Documents

This is the point that regularly appears too late in projects. An embedding acts like a harmless series of numbers – but it is a condensed coding of the source text. Research on so-called embedding inversion shows that a significant part of the original content can be reconstructed from vectors and that it is precisely the delicate components that come back particularly reliably: names, numbers, addresses, identifiers.

In practice, this means that anyone who gives embeddings of confidential documents to an external service passes on the documents. A vector index is just as vulnerable as the stock from which it was created.

This leads to two specific requirements. First, the vector stock is under the same level of protection as the source documents – for personal or business-critical content, this clearly speaks for operation in-house or in a private deployment. Secondly, we need a Erasing concept: If a document is deleted because a retention period expires or someone asserts their right to deletion, the associated vectors must also be deleted. This does not happen by itself, and who does not plan it from the beginning, later has a database that keeps content that officially no longer exists. As a whole, we describe in the contribution to Introduction of a RAG system in SMEs.

Use in practice: RAG

With Retrieval-Augmented Generation, the vector database first searches for the most relevant passages from your own data. Only these are given to the AI as context. The result: fact-based answers with references instead of invented statements.

The vector database is the component that decides on quality and speed – but not alone. If the retrieval finds the wrong passages, even the best language model cannot build a correct answer from it. This is precisely why you have to measure retrieval and response quality separately; Which indicators are suitable for this is in the contribution to Evaluation of RAG systems. An application example with high requirements for occupancy shows the contribution to RAG in the legal system.

When you do not need a vector database

Not every search task requires semantic search. If your inventory is small and well-structured, if users are looking for unique identifiers anyway, or if a clean full-text search with well-maintained metadata solves the problem – then a vector database is an additional operational expense without value. The need arises where people ask in their own words and the answer is in flow text, which is formulated differently than the question.

Conclusion

Vector databases are no longer an exotic special tool, but an extension that in many cases finds space in the existing PostgreSQL installation. Success is rarely decided by the product, but by the work around it: meaningfully cut sections, a hybrid search for exact hits, measured quality – and a data protection concept that treats embeddings as they are, namely as a copy of your content. If you want to assess what is necessary for your document stock, talk to us. How we set up RAG systems is on our side AI consultancy and RAG systems – GDPR-compliant and on request completely on-premise.

Additional sources

• Morris et al. – Text Embeddings Reveal (Almost) As Much As Text
• pgvector – vector search for PostgreSQL