Skip to content
Thomas Karner
Menu

· updated · Thomas Karner

RAG in business: how language models work with your own knowledge

What retrieval-augmented generation (RAG) is, which weaknesses of language models it addresses, and what companies should consider around data protection and roll-out.

Over the years, most companies have stored their knowledge in manuals, contracts, wikis, tickets and e-mails. Much of it is only found if someone still remembers where it is. Language models such as GPT, Claude or Mistral promise to help – but they don’t know your internal documents. Retrieval-augmented generation, or RAG, closes that gap.

What RAG is

The idea comes from a 2020 research paper by Facebook AI Research, now Meta (Lewis et al., 2020). Instead of relying only on what it learned in training, the system first retrieves relevant passages from a knowledge base and passes them to the language model as context. The model then writes its answer based on those sources.

In simplified terms, a RAG system has three parts:

  1. Indexing: documents are split into sections and prepared for fast search – usually as vectors that capture the meaning of a text.
  2. Retrieval: for a given question, the most similar sections are found.
  3. Generation: the question and the retrieved sections go to the language model together, which produces an answer – ideally with a reference to the source.

Language model without RAG

Which problems RAG solves

Missing knowledge. A language model only knows what was publicly available up to its training cut-off. Internal documents, current price lists or the latest version of a policy are not part of it. With RAG, this information is added at the time of the query – without retraining the model.

Language model with RAG

Made-up answers. Language models generate text by calculating likely continuations. That often sounds convincing but isn’t always correct – these errors are called hallucinations. Katharina Zweig explains clearly why a language model doesn’t “know” whether it knows something (Zweig, 2025). RAG doesn’t eliminate the problem, but it reduces it considerably: the model relies on concrete passages, and the answer can be checked against its source (Gao et al., 2024).

Limited context. You can’t simply hand a model a hundred documents and hope for a good answer. Even models with large context windows don’t use long inputs reliably: information in the middle is used far less well than information at the beginning or end (Liu et al., 2024). A good pre-selection of relevant passages is therefore essential.

Data protection: where the data really goes

A common argument for RAG is that documents stay inside the company. That is only partly true. It is correct that the language model is not trained on your data, so it doesn’t permanently “learn” anything. But the retrieved passages are sent to the model with every request. If the model runs at a cloud provider, that content leaves your company.

In practice, this means:

  • Cloud models require a data processing agreement under Art. 28 GDPR and a commitment that inputs are not used for training.
  • For highly confidential data, the language model can be self-hosted, in your own data centre or with a European hosting provider.
  • Search should respect permissions. If someone isn’t allowed to open a document, they shouldn’t see it through the assistant either.

Since February 2025, the EU AI Act also requires companies that deploy AI systems to ensure their staff have sufficient AI literacy (Art. 4). A roll-out should therefore include training and clear usage rules.

Karen Hao’s book about OpenAI shows how hungry the large AI providers are for data (Hao, 2025) – a good reason to decide deliberately which data goes where.

Where RAG helps day to day

  • Onboarding: new employees find answers in manuals and process descriptions without interrupting colleagues.
  • Customer service: enquiries can be answered faster based on product documentation and past tickets.
  • Policies and contracts: questions such as “What notice period applies in contract X?” are answered with a reference to the relevant clause.
  • Scattered knowledge: information from several departments and systems becomes findable through a single interface.

What matters in a roll-out

The quality of a RAG system depends less on the language model than on the data and the search. In practice, these factors decide:

  • Data quality: outdated or contradictory documents lead to outdated or contradictory answers. Maintaining the sources is unavoidable.
  • Preparation: how documents are split has a strong effect on retrieval quality. Tables, scans and PDFs often need special handling.
  • Evaluation: before roll-out, test with real questions from daily operations and measure how often the answers are right.
  • Citations: every answer should show where it comes from. That builds trust and makes errors visible.

Outlook

Current developments are moving towards so-called agents: systems that don’t just look things up, but carry out several search steps on their own, compare sources or trigger follow-up actions. From an information systems management perspective, this is above all a question of process design: where may a system act on its own, and where does it need human approval?

Conclusion

RAG makes language models genuinely useful for companies, because it connects them with their own knowledge and makes answers verifiable. It isn’t automatic, though: data quality, permissions and data protection decide whether a prototype becomes a tool employees trust.

Sources

← Back to the blog

Let's talk about your project.

In a free initial call we clarify what you need and how I can help – no strings attached.

contact@thomas-karner.com +43 680 140 33 20

Free initial call (30 min)