LangChain Advanced RAG

A basic RAG pipeline splits documents, stores embeddings, retrieves chunks, and asks the model to answer. That recipe works well for demos. Real users ask vague questions, use different wording than the documents, and expect accurate answers every time. Advanced RAG techniques close those gaps by improving the question, the search, and the final context.

Where Basic RAG Fails

• A user types "refund rules?" while the document says "return and reimbursement policy".
• The best chunk ranks fifth, and the prompt only includes the top three.
• A follow-up like "what about for students?" lacks the context to search.
• The retriever returns three chunks that repeat the same sentence.

The Upgraded Pipeline

 User question
      |
      v
 [1] Rewrite the question        ← clearer query
      |
      v
 [2] Retrieve many candidates    ← wide net (20 chunks)
      |
      v
 [3] Rerank by true relevance    ← keep the best 4
      |
      v
 [4] Build compact context
      |
      v
 [5] Model answers with sources

Technique 1: Query Rewriting

Users write short, messy questions. A rewriting step asks the model to turn the question into a clean search query, and it can fold in chat history for follow-ups.

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

rewrite_prompt = ChatPromptTemplate.from_template(
    "Rewrite the question as a clear, standalone search query.\n"
    "History: {history}\nQuestion: {question}\nQuery:"
)
rewriter = rewrite_prompt | model | StrOutputParser()

query = rewriter.invoke({
    "history": "We discussed the premium plan.",
    "question": "what about for students?",
})
# "Is there a student discount on the premium plan?"

Technique 2: Multi-Query Retrieval

One question can hide several search angles. The model writes three versions of the question, and the retriever searches with all three. The system merges the results and removes duplicates.

 "refund rules?"
      |
      v
 Model writes variations:
   - "return policy for purchased items"
   - "how to get money back after a purchase"
   - "refund eligibility and time limit"
      |
      v
 Search with each → merge → remove duplicates

variants_prompt = ChatPromptTemplate.from_template(
    "Write 3 different search queries for: {question}\nOne per line."
)
variants_chain = variants_prompt | model | StrOutputParser()

def multi_search(question):
    lines = variants_chain.invoke({"question": question}).split("\n")
    seen, results = set(), []
    for q in [question] + [l.strip() for l in lines if l.strip()]:
        for doc in retriever.invoke(q):
            if doc.page_content not in seen:
                seen.add(doc.page_content)
                results.append(doc)
    return results

Technique 3: Hybrid Search

Meaning-based search handles paraphrases well. Keyword search handles exact codes, names, and rare terms well. Hybrid search runs both and blends the rankings. Searching for "Error E-4021" benefits from keyword matching, because embeddings may blur the number.

 Question → Vector search  →  ranked list A  ┐
 Question → Keyword search →  ranked list B  ┴→ Blend → final ranking

Technique 4: Reranking

The first retrieval step favors speed. A reranker is a second, more careful model that reads the question and each candidate together and scores true relevance. Retrieve 20 candidates cheaply, then rerank to keep the best 4.

 20 candidates  →  Reranker scores each  →  keep top 4  →  prompt

Hosted rerank services and open-source cross-encoder models both plug into LangChain as document compressors.

Technique 5: Compressing Context

A retrieved chunk may contain one useful sentence and nine unrelated ones. Contextual compression trims each chunk down to the relevant part. Shorter context lowers cost and reduces distraction.

Technique 6: Citing Sources

Trust grows when the answer points to its evidence. Keep metadata such as file name and page number, then ask the model to cite them.

def format_docs(docs):
    parts = []
    for i, d in enumerate(docs, 1):
        src = d.metadata.get("source", "unknown")
        parts.append(f"[{i}] ({src}) {d.page_content}")
    return "\n\n".join(parts)

prompt = ChatPromptTemplate.from_template(
    "Answer from the numbered context. Cite numbers like [1].\n"
    "If the context lacks the answer, say you do not know.\n\n"
    "{context}\n\nQuestion: {question}"
)

Measuring Improvements

Change one technique at a time and test on a fixed set of 20 to 50 real questions. Track whether the correct chunk appears in the retrieved list and whether the final answer is correct. A technique that does not move these numbers does not deserve its extra cost.

Choosing Techniques by Problem

• Vague questions call for query rewriting.
• Missed matches call for multi-query or hybrid search.
• Right chunk, wrong order calls for reranking.
• Long, noisy prompts call for compression.
• Low user trust calls for citations.

Real-World Scenario

A travel company runs a help bot over visa guides for many countries. Users type short messages such as "visa for kids?" and the basic bot returns random matches. The team adds query rewriting, which expands the message into a full search query with the country from earlier chat turns. A reranker then moves the best chunk to the top. Correct answers rise from 62 percent to 85 percent on the team's fixed test list. Each technique earned its place through measured gains.

Practice Tasks

1. Write a rewriting prompt that turns three vague questions into full search queries.
2. Retrieve with and without multi-query and count the distinct chunks returned.
3. Add source numbers to a context block and ask the model to cite them.
4. Make a list of ten test questions and record which chunk should answer each.

Key Points

• Advanced RAG improves the question, the search, and the context.
• Rewriting and multi-query handle vague or varied wording.
• Hybrid search blends keyword and meaning-based matching.
• Rerankers and compression sharpen the final context.
• Test each change on a fixed question set.