Introducing RAG system in SMEs: Use cases, data sovereignty and the questions before the start

The figures of the Bitkom study on the digitalization of the economy 2026 are clear: 41 percent of companies in Germany use AI, up from 17 percent in the previous year. More interesting than the jump, however, are the hurdles that the same survey calls – 41 percent cite privacy uncertainty, 37 percent unclear costs, and a third report that AI became more expensive than expected. It is precisely at these three points that a decision is made whether a RAG project will support SMEs. What we do is on our side RAG and AI agents.

Why RAG is the pragmatic start

At its core, retrieval-augmented generation means: The language model does not respond from its training, but from your documents – and names the source. This solves two problems at once, which otherwise block the entry. Hallucinations become verifiable because each statement brings a proof. And the model does not have to be trained on company data, which avoids effort, costs and the most sensitive data protection issues from the outset.

How this works technically, we have described in detail elsewhere – on the role of Vector databases combination with knowledge graphs GraphRAG. This article goes the other way: What does such a project require organizationally before the first line of code is created?

The use case decides, not the technology

The most common mistake when getting started is to start with the technology instead of asking which question should actually be answered. A first use case is appropriate if it meets four conditions:

• The question is repeated. If the same information is sought by different people several times a week, there is a measurable benefit. One-time special searches do not justify a system.
• The answer is in documents. RAG finds what was written. Knowledge that lies exclusively in the minds of experienced colleagues cannot lift a retrieval – this is a documentation project, not an AI project.
• An error is apparent. Whoever gets the answer must be able to judge whether it is true. This is why internal assistance systems work much better for professionals than information systems for laypeople.
• There is one responsible. Someone from the specialist department has to decide which documents apply and which ones are outdated. Without this role, the knowledge base becomes obsolete faster than the system creates benefits.

Typical candidates who meet these criteria: service and maintenance documentation, offer and contract templates, internal policies and work instructions, technical standards. The classic “one chatbot for everything” does not fulfill a single one of them.

What has happened since 2. August 2026 additionally applies

Here it is worth taking a close look, because the headlines mislead in both directions. Since August 2, 2026, the Transparency obligations under Article 50 the EU AI Regulation: Anyone who interacts with an AI system must be able to recognize this, and AI-generated content must be labeled. The obligations for high-risk systems, on the other hand, were imposed by the Digital Omnibus (Regulation (EU) 2026/1744, in force since 27. July 2026) pushed backwards – stand-alone high-risk systems according to Annex III on the 2. December 2027, AI as a safety component in Annex I products on 2 August 2028.

For an internal knowledge assistant, this means that a system that selects the appropriate maintenance instruction for service technicians is not usually a high-risk system anyway. Nevertheless, the obligation of transparency applies, as does the obligation under Article 4, to empower their own employees in dealing with AI.

Things are different as soon as personnel decisions are touched. Applicant pre-selection, performance evaluation and credit assessment are explicitly high-risk applications in accordance with Annex III – with conformity assessment, quality management system and documented human oversight. The deadline was postponed, not the substance: Anyone who uses a RAG system on personnel files today already determines with architecture, logging and data source whether the conformity assessment will later become a formality or a reengineering project. What actually came into effect on August 2nd and what did not, we have own contribution to the EU AI Act broken down; The classification for your company summarizes our page for AI governance together.

The question is not whether a RAG system is regulated, but which documents it is allowed to read. This answer is usually already in the authorization concept – it only has to be transferred to the system.

Data sovereignty is more than the hosting location

When 41 percent of companies cite data protection insecurity as a hurdle, they usually mean the question of where the data flows. It can be answered cleanly: A RAG system can run entirely in its own infrastructure or in a private deployment without a document going outside.

Hardly anyone asks the more difficult question in advance – and it is the one on which projects fly in the company: A RAG system shall not undermine the existing authorisation concept. If a document stock is indexed without regard to access rights, each user can then access content via a cleverly asked question that he should never have opened in the file system. Salary lists, dismissal drafts, offer calculations – everything that lies in a folder that the system is allowed to read. The permissions must therefore be passed through to the search results, not filtered in the interface.

What makes RAG projects fail

In addition to the permission theme, three patterns are repeated. Firstly, document quality: Ten versions of the same manual in a folder cause the system to reliably quote the wrong one – without order in the inventory, no technology helps. Secondly, the lack of measurement. Whether a system gets better can only be judged if retrieval and response quality are assessed separately; how to do this is stated in our contribution to the Evaluation of RAG systems. And thirdly, the expectation: a system that works great with fifty curated documents does not automatically get better by adding five thousand unchecked ones. It's getting worse.

From proof of concept to operation

The path that has proven itself is unspectacular. At the beginning, there is a narrow application case with a manageable, but real document stock and a fixed group of test users from the specialist department. This proof of concept answers the only question that matters at this point: Are the answers good enough for people to use the system voluntarily? Only then will there be a connection to the source systems, authorization logic, operating processes and the expansion to other stocks.

Regarding costs: Reliable figures only arise when three things have been determined – the status and quantity of documents, the number of source systems to be connected and the question of whether on-premise is operated or in a managed environment. Whoever mentions a lump sum before this clarification, calculates the uncertainty, at your expense. Among other things, the PoC has the purpose of making figures out of estimates.

Conclusion

RAG is therefore the sensible entry into AI in SMEs because it manages without training on its own data, makes answers verifiable and can run completely in-house. The difficulties are rarely in the model, but in the preparatory work: an application case that meets the four criteria, an ordered document stock, a cleanly submitted authorization concept and a measure of quality. If you want to assess whether and with which use case an entry carries with you, talk to us An initial interview costs nothing but an hour.

Additional sources

• Bitkom Research – Digitalization of the Economy 2026
• European Commission – Regulatory framework on Artificial Intelligence
• EUR-Lex – Regulation (EU) 2026/1744 (Digital Omnibus on AI)

Note: This article reflects the publicly reported status and does not replace individual legal advice. How a specific system should be classified should be examined on a case-by-case basis.