Retrieval-Augmented Generation - RAG

What is Retrieval-Augmented Generation?

Large Language Models (LLMs) are trained on a vast amount of unstructured data and are useful for generating text. They power your chatbots, AI translators, summarizers, and much more.

The problem? LLMs are fixed in time, and we live in a world that’s always in flux (panta rhei). Because their knowledge stops at their training data, LLMs can give outdated or inaccurate answers, or even hallucinate one. Without controls, your typical chatbot will always have an answer for you (it will make up one if it needs to).

The solution is RAG, and the core idea is simple: retrieve current, relevant information and hand it to the LLM. Its answers will be grounded in facts rather than memory alone.

Retrieval-Augmented Generation Process

The RAG process can be broken down into the following phases:

Get external data

The first step is to identify the relevant data sources. This can be domain-specific, such as legal documents or medical journals. It can also be a collection of unstructured documents or the World Wide Web.

Machines understand numbers (0s and 1s). So we need a way to break down our data sources(text) into numbers(binary). We do this using embedding models, AI models that convert text into numerical representations (vectors). These vectors are then stored in a vector database.

Retrieve relevant chunks

Next, we perform a relevancy search. The user query is converted into a vector embedding and compared against the vectors stored in the database. For example, consider a financial advisor using a smart Document QA chatbot to answer a client's question: "What risks did Some-company flag in its latest annual report?"

The system will retrieve the "Risk Factors" section from the document because they are relevant to the researcher’s input.

Relevance was calculated using vector operations and a specific similarity measure, such as Cosine similarity.

Augment the LLM prompt

Next, the retrieved chunks are added to the user's query, giving the LLM the relevant context to answer.

Case study

If you'd like to see the RAG process in action, I've developed a Document QA chatbot: upload your documents, ask questions, and get answers grounded in your files, with citations. Try it here: DocQA