RAG for (newbies), piece by piece

Imagine you join a new company and want to know:

How many days of annual leave can I carry over to next year?

A general LLM may know how leave policies usually work, but it does not know the exact policy at your company. It might give a common answer, tell you that policies vary, or confidently guess something wrong.

Now imagine we first search the company handbook and find the paragraph that says employees can carry over five days. We place that paragraph beside the question and ask the LLM to answer using it.

That is the basic idea behind Retrieval-Augmented Generation, or RAG.

1. Retrieve information related to the question.
2. Augment the question with that information.
3. Let the LLM generate an answer from the combined context.

The LLM is still doing the writing and reasoning. RAG simply gives it the right notes before it starts.

LLMs know a surprising amount because they were trained on a huge amount of text. But knowing a lot is different from having the exact information needed for every question.

An LLM may not know:

• your company’s private documents
• a product change made yesterday
• the latest version of an internal policy
• a detail hidden inside a long PDF
• which source should support its answer

The knowledge inside an LLM is also stored indirectly in its model weights. It cannot browse those weights like a database and fetch one exact record whenever we ask.

RAG helps by keeping the knowledge outside the model, where it can be updated, searched, filtered, and shown to the LLM only when it is useful.

Knowledge can be old. An LLM has a training cutoff. Information created or changed after training is not automatically available to it. A RAG system can search a data source that is updated regularly.

Private data was never in training. Your support tickets, research notes, contracts, and internal documentation are not part of a public model’s knowledge. RAG provides controlled access to this information without retraining the entire model.

The model can make up missing details. When the model does not have enough information, it may still produce an answer that sounds reasonable. Relevant context gives it something concrete to reason from.

Exact recall is difficult. Models are good at patterns and language, but they are not reliable databases. RAG handles the lookup, while the LLM handles understanding and explanation.

A RAG system has two broad parts.

The first part prepares the data so it can be searched. The second part uses that prepared data to retrieve context and generate an answer.

We will look at preparation, retrieval, and generation separately. This makes it easier to see what each part is responsible for.

Before we can retrieve anything, we need to turn the external data into something searchable.

1. Start with external data. The data can come from PDFs, web pages, Notion, Google Drive, a database, support tickets, or any other source your application needs.

2. Load and parse it. Loading gets the file or record into the pipeline. Parsing extracts useful content from it. For a PDF, this may mean recovering paragraphs, headings, tables, and metadata instead of treating the file as one large block. Good parsing matters because retrieval cannot recover information that was lost or scrambled during extraction.

3. Split the text into chunks. Documents are usually too large to retrieve as one unit. We split them into smaller pieces called chunks.

A chunk might be a few paragraphs, one section, or a small part of a table. It should contain enough context to make sense on its own, but stay focused enough to match a specific question.

4. Create embeddings. An embedding model converts every chunk into a list of numbers called a vector. The vector represents the meaning of the text.

Chunks about similar ideas tend to have vectors that are close to each other, even when they use different words.

5. Save them in a vector database. We store each vector together with its original chunk and useful metadata. The vector database is built to search these vectors quickly.

At this point, the preparation is complete. Our data is now ready for retrieval.

Chunk overlap deserves a closer look because the percentage can feel abstract. The diagram below uses a fixed chunk size of 512 tokens and shows exactly what changes when we repeat part of one chunk inside the next.

Now a user asks a question.

The same embedding model converts the question into a vector. We then compare this query vector with the chunk vectors already stored in the database.

One common comparison method is cosine similarity. In simple terms, it measures whether two vectors point in a similar direction. A higher score usually means the question and chunk are closer in meaning.

The database sorts the chunks by similarity and returns the top-k results.

If k = 3, we retrieve the three chunks that appear most relevant to the question.

This is semantic search. A question about "time off that moves to next year" can still match a chunk containing "annual leave carryover," even though the wording is different.

The similarity score is calculated once for every candidate chunk. The query vector stays the same while each stored chunk vector takes its turn in the comparison. After all scores are calculated, the chunks can be ranked from most similar to least similar.

We now have three useful ingredients:

1. the system prompt, which tells the model how to behave
2. the user’s question
3. the top-k retrieved chunks

We combine them into one prompt. This is the augmentation step.

It may look roughly like this:

You are a helpful company assistant. Answer only from the provided context.
Context: Employees may carry over up to five days of annual leave.
Question: How many leave days can I carry over?

The LLM reads this complete prompt and generates the final answer:

You can carry over up to five days of annual leave.

The answer is useful because the LLM can explain naturally, but it is grounded because the important fact came from the company handbook.

RAG is easier to understand when we separate the jobs:

• The parser turns source files into usable text.
• The chunker divides that text into searchable pieces.
• The embedding model represents meaning as vectors.
• The vector database stores and searches those vectors.
• The retriever selects the most relevant chunks.
• The LLM reads the retrieved context and writes the answer.

RAG does not make the model know everything. It gives the model a way to look up what it needs.

A RAG answer has two chances to fail.

The retriever may fail to find the evidence, or the LLM may receive good evidence and still produce a poor answer. If we only score the final answer, we cannot tell which part needs fixing.

Start with a small evaluation dataset. Each example should contain:

• a realistic question
• the evidence retrieval should find
• an optional reference answer
• the expected behavior when no answer exists

Then evaluate in pipeline order. Test retrieval first, verify that enough evidence reached the LLM, and only then evaluate generation.

This follows the practical eval workflow described by Hamel Husain, where evaluations should be specific to the product and useful for debugging. Jason Liu’s RAG eval framing reduces the system to three things: question, retrieved context, and answer. Looking at their relationships makes failures much easier to locate.

The final answer can only be as good as the information that reaches the model.

If parsing loses a table, chunking separates an important sentence from its context, or retrieval chooses the wrong chunks, the LLM may still answer poorly. More retrieved chunks do not always help either. Too much irrelevant context can distract the model.

This is why real RAG systems need evaluation. We should test whether the right information was parsed, whether the retriever found it, and whether the final answer used it correctly.

But the core idea stays simple: find the right information first, then ask the LLM to answer with it.