What Is RAG in AI? How Retrieval-Augmented Generation Works, With a Starter Project

What is RAG in simple terms?

A large language model (LLM) knows what was in its training data and nothing about your company’s leave policy, last week’s price list or a client contract. You could paste the document into the chat each time, but that breaks down with hundreds of documents.

RAG solves this with an open-book approach. When someone asks a question, the system looks up the few most relevant passages in your documents and pastes only those into the prompt, with an instruction such as “answer using only the context below”. The model then writes a grounded answer.

The term comes from a 2020 research paper by Patrick Lewis and colleagues, which combined a pre-trained language model with a searchable index of Wikipedia. Today RAG is the standard pattern behind internal knowledge bots, customer support assistants and “chat with your PDFs” tools.

How does RAG work, step by step?

A RAG system has two phases: preparing the documents once (indexing), and answering each question (retrieval and generation).

Step	Phase	What happens	Example: HR policy bot
1. Load	Indexing	Collect the source documents and extract clean text	Leave policy PDF, travel policy, payroll FAQ
2. Chunk	Indexing	Split text into passages of a few hundred words, often with some overlap	The leave policy becomes 40 chunks
3. Embed	Indexing	Turn each chunk into a list of numbers (a vector) that captures its meaning	Each chunk becomes a vector of, say, 768 or 3,072 numbers
4. Store	Indexing	Save vectors, chunk text and metadata (file, page, date) in a vector database	Stored with “Leave Policy v3, page 4”
5. Retrieve	Each question	Embed the question and find the chunks with the closest vectors	“Can I carry forward leave?” finds the carry-forward clause
6. Augment	Each question	Build a prompt: instructions + retrieved chunks + the question	“Answer only from the context. If it is not there, say you don’t know.”
7. Generate	Each question	The LLM writes the answer, ideally with citations	“Up to 10 unused days can be carried forward (Leave Policy, p. 4).”

What are embeddings and vector databases?

An embedding is a list of numbers that represents the meaning of a piece of text. An embedding model is trained so that texts with similar meaning get vectors that point in similar directions. “How many leaves can I carry over?” and “Unused leave can be carried forward” share few exact words but end up close together. That is why RAG search works on meaning, not just keywords.

Real embedding models produce long vectors. Google’s Gemini embedding models, for example, output 3,072 numbers by default, and can be truncated to fewer. To see the idea, here is a toy version with only three numbers per chunk, standing for how much each chunk is about leave, money and travel:

import math

# Toy embeddings: [leave, money, travel]
chunks = {
    "Employees get 18 days of paid leave per year.": [0.9, 0.1, 0.0],
    "Unused leave up to 10 days can be carried forward.": [0.8, 0.2, 0.1],
    "Travel claims must be filed within 30 days with receipts.": [0.1, 0.6, 0.9],
    "Salary is credited on the last working day of the month.": [0.0, 0.9, 0.1],
}

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

def retrieve(query_vector, k=2):
    scored = [(cosine(query_vector, v), text) for text, v in chunks.items()]
    return sorted(scored, reverse=True)[:k]

# "How many leave days can I carry forward?"
query = [0.7, 0.2, 0.1]
for score, text in retrieve(query):
    print(f"{score:.3f}  {text}")

Output:

0.999  Unused leave up to 10 days can be carried forward.
0.977  Employees get 18 days of paid leave per year.

Cosine similarity scores how closely two vectors point the same way, from 1 (same direction) downwards. The two leave chunks score highest; the travel and salary chunks score about 0.35 and 0.29 and are left out. Those top two chunks are what the model would receive. In a real system the numbers come from an embedding model, not from you.

A vector database stores millions of these vectors and finds the nearest ones quickly. Options range from pgvector, an open-source extension that adds vector search to PostgreSQL (with operators such as <=> for cosine distance), to dedicated databases such as Pinecone, Qdrant, Weaviate and Chroma. For a first project, the vector store built into your automation tool or a free-tier hosted database is enough.

When should you use RAG, and when not?

Situation	Best approach
Answers must come from private or frequently changing documents	RAG
Users need to see which document an answer came from	RAG with citations
The whole knowledge base is small, a few hundred pages	Often simpler to put everything in the prompt; Anthropic suggests this for knowledge bases under about 200,000 tokens (roughly 500 pages)
You want the model to write in a particular style or format every time	Better prompts or examples; fine-tuning if that is not enough
Answers depend on live numbers in a database (orders, stock)	Let the model query the database or an API as a tool, rather than embedding records

RAG is not a way to make a model smarter. It is a way to give it the right reading material at the right moment.

What is a good first RAG project?

Project: a policy question bot for a small team. Pick 5 to 10 public or non-confidential documents you understand well, such as a college’s academic rules, your society’s bye-laws or a product’s help pages. Avoid anything with personal data.

1. Write 20 test questions first, with the correct answer and the page it comes from. Include 3 questions the documents do not answer.
2. Build the pipeline. No-code route: in n8n, load files into a vector store node with an embeddings sub-node, then connect a chat trigger to an AI Agent that uses the vector store as a tool. Code route: a short Python script using an embedding API and a local vector store such as Chroma or pgvector.
3. Write a strict prompt: answer only from the context, cite the file and page, and say “I don’t know” if the context does not contain the answer.
4. Run your 20 questions and score each answer: correct, partly correct, wrong, or correctly refused.
5. Change one thing and re-run: chunk size, number of chunks retrieved, or the prompt. Record the score each time.

The scoring table is what makes this a portfolio project rather than a demo. It shows you can measure quality, which is what employers look for. See AI projects for your resume for how to write it up, and our n8n tutorial for beginners if you have not used n8n yet.

Why does RAG give wrong answers? Common failure modes

Failure	What you see	Common fix
Bad text extraction	Answers miss content from scanned PDFs or tables	Use OCR for scans; convert tables to clean text; check extracted text by eye
Chunks too small or too big	Answers lose context, or the right sentence is buried in noise	Try a few chunk sizes with overlap, and keep headings with their paragraphs
Chunk lacks context	“The limit is ₹5,000” is retrieved, but for which policy?	Add the document title and section to each chunk before embedding; Anthropic’s “contextual retrieval” does this
Keyword misses	Searches for codes, names or policy numbers fail	Combine vector search with keyword search (hybrid search, for example BM25)
Right chunk ranked too low	The answer exists but is not in the top results	Retrieve more candidates, then use a reranking model to reorder them
Hallucinated answer	The model answers confidently from general knowledge	Strict “only from context” prompt, require citations, test unanswerable questions
Stale or conflicting documents	Answers quote an old version of a policy	Store dates and versions as metadata; remove or filter out superseded files
Access leaks	A user sees content from documents they should not see	Filter retrieval by the user’s permissions before anything reaches the model

Retrieval quality matters more than the choice of language model. Anthropic reported in 2024 that adding context to chunks, combined with keyword (BM25) search, reduced failed retrievals by 49% in its tests, and by 67% when a reranking step was added. The lesson for beginners: when answers are wrong, inspect what was retrieved before blaming the model.

How is RAG related to AI agents?

Classic RAG always retrieves once and then answers. In agentic RAG, the model decides whether to search, what to search for, and whether to search again, using retrieval as one tool among several. Most modern assistants mix both. Our explainer on what AI agents are covers the agent side, and AI vs machine learning vs deep learning shows where LLMs fit in the bigger picture.

Frequently Asked Questions

What does RAG stand for in AI?

RAG stands for retrieval-augmented generation. The system retrieves relevant passages from a set of documents and adds them to the prompt, so a large language model can generate an answer grounded in that material.

Is RAG better than fine-tuning?

They solve different problems. RAG gives a model access to specific, changing information and makes it easy to cite sources. Fine-tuning changes how a model behaves, such as its style or format. For answering questions from company documents, RAG is usually the first choice because documents can be updated without retraining.

Do I need a vector database for RAG?

For anything beyond a handful of documents, you need some way to store embeddings and search them by similarity. That can be a dedicated vector database, PostgreSQL with pgvector, or the vector store built into a tool such as n8n. For very small knowledge bases, you can skip retrieval and place all the text in the prompt.

Can I build a RAG chatbot without coding?

Yes. Workflow tools such as n8n have document loaders, embedding models, vector store nodes and AI agent nodes that you connect visually. Coding helps with custom document cleaning and evaluation, but a first working RAG bot can be built without it.

Why does my RAG chatbot give wrong answers?

The most common causes are poor text extraction from PDFs, chunks that are too small or lack context, relevant passages not ranked high enough, and prompts that let the model answer from general knowledge. Check which chunks were retrieved for each wrong answer before changing the model.

Sources and methodology

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv, submitted 22 May 2020): origin of the term and approach. Checked September 2026.
• Google AI for Developers, Gemini API embeddings: model names and 3,072-dimension default output; RAG use case. Checked September 2026.
• pgvector on GitHub, pgvector/pgvector: open-source vector search for Postgres and distance operators. Checked September 2026.
• Anthropic, Introducing Contextual Retrieval (19 September 2024): 49% and 67% reductions in failed retrievals; under-200,000-token guidance. Checked September 2026.

Method: technical facts come from the linked papers and official documentation. The HR policy example, starter project and failure-mode fixes are the ISS Editorial Team’s own material. The Python example uses hand-made three-number vectors for illustration; it was run on the data shown and produced the output shown.

Next steps

Write your 20 test questions this week and build the simplest RAG bot that can answer them. If you want to build RAG pipelines, agents and automations in a live cohort with project reviews, see the AI & Agentic Systems curriculum.

If it fits, you can apply for free. You speak with admissions first and pay only after you accept an offer.