Vector RAG with LLMs

RAG stands for Retrieval-Augmented Generation. It combines a vector database with a large language model (LLM) like GPT or Claude, letting the AI answer questions using your own documents instead of relying only on what it learned during training. This topic shows you how to build a RAG system.

Why RAG Exists

LLMs have two key limitations on their own:

They know nothing about events or documents created after their training cutoff
They know nothing about your private data — your company documents, your codebase, your customer records

RAG solves both problems by retrieving relevant information from a vector database and inserting it into the prompt before the LLM generates its answer.

The RAG Pipeline

RAG ARCHITECTURE
─────────────────────────────────────────────────────────────

STEP 1: User asks a question
  "What is our company's refund policy?"
       │
       ▼
STEP 2: Embed the question
  Question → Embedding Model → Query Vector
       │
       ▼
STEP 3: Search the vector database
  Query Vector → Vector DB → Top 3 relevant document chunks
       │
       ▼
STEP 4: Build an augmented prompt
  "Using the following context, answer the question:
   
   Context: [retrieved chunk 1] [retrieved chunk 2] [retrieved chunk 3]
   
   Question: What is our company's refund policy?"
       │
       ▼
STEP 5: Send to LLM
  Augmented Prompt → LLM (GPT, Claude, etc.) → Final Answer
       │
       ▼
STEP 6: Return answer to user
  "Our refund policy allows returns within 30 days..."

Without RAG vs. With RAG

Scenario	Without RAG	With RAG
Ask about private company policy	LLM has no knowledge, makes something up	LLM reads the actual policy document and answers accurately
Ask about recent events	LLM's knowledge stops at training cutoff	LLM uses retrieved current information
Ask about a specific PDF you uploaded	LLM never saw that document	LLM retrieves relevant sections and answers from them

Building a Simple RAG System

Step 1: Index Your Documents

def index_documents(documents):
    for doc in documents:
        chunks = chunk_text(doc["content"])
        for i, chunk in enumerate(chunks):
            vector = get_embedding(chunk)
            index.upsert(vectors=[{
                "id": f"{doc['id']}-chunk-{i}",
                "values": vector,
                "metadata": {"text": chunk, "source": doc["title"]}
            }])

documents = [
    {"id": "policy-1", "title": "Refund Policy", "content": "Customers may return items within 30 days..."},
    {"id": "policy-2", "title": "Shipping Policy", "content": "Standard shipping takes 5-7 business days..."},
]

index_documents(documents)

Step 2: Retrieve Relevant Context

def retrieve_context(question, top_k=3):
    query_vector = get_embedding(question)
    results = index.query(
        vector=query_vector,
        top_k=top_k,
        include_metadata=True
    )

    context_chunks = [match["metadata"]["text"] for match in results["matches"]]
    return "\n\n".join(context_chunks)

Step 3: Build the Augmented Prompt and Call the LLM

from anthropic import Anthropic

client = Anthropic(api_key="YOUR_ANTHROPIC_KEY")

def ask_rag_question(question):
    context = retrieve_context(question)

    prompt = f"""Answer the question using only the context below.
If the context does not contain the answer, say you don't know.

Context:
{context}

Question: {question}"""

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=500,
        messages=[{"role": "user", "content": prompt}]
    )

    return response.content[0].text

answer = ask_rag_question("How long do I have to return an item?")
print(answer)

Expected Output

Based on the refund policy, you have 30 days from the date of
purchase to return an item.

The LLM grounds its answer in the retrieved document rather than guessing. This dramatically reduces hallucination — a major problem when LLMs answer questions outside their training knowledge.

Best Practices for Production RAG

Tell the LLM to Admit Uncertainty

Always instruct the model to say "I don't know" when the retrieved context lacks the answer. This single instruction prevents the model from fabricating plausible-sounding but false answers.

Cite Sources

Include the source document name with each retrieved chunk and ask the LLM to cite which source it used. This builds user trust and makes answers verifiable.

context_with_sources = "\n\n".join(
    f"[Source: {match['metadata']['source']}]\n{match['metadata']['text']}"
    for match in results["matches"]
)

Tune top_k Carefully

Retrieving too few chunks risks missing the answer. Retrieving too many wastes tokens and can confuse the model with irrelevant information.

top_k Value	Effect
1–2	Fast, risks missing relevant context
3–5	Balanced, works well for most use cases
10+	Comprehensive but costly, can dilute relevance

Re-rank Before Sending to the LLM

Fetch more candidates than needed (such as 10), apply a re-ranking model, then send only the top 3–5 highest-quality chunks to the LLM. This improves answer accuracy without raising retrieval costs significantly.

Common RAG Failure Modes

Problem	Cause	Fix
LLM ignores the context	Weak prompt instructions	Explicitly instruct: "answer only using the context"
Wrong chunks retrieved	Poor chunking strategy	Adjust chunk size, add overlap
Outdated answers	Vector database not refreshed	Set up scheduled re-indexing
Slow responses	Too many retrieved chunks	Reduce top_k, add re-ranking

RAG is the most common production use case for vector databases today. It powers AI chatbots that answer questions about company documentation, customer support systems, and internal knowledge assistants used across thousands of businesses.

Previous lesson

Back to course

Next lesson