Databricks RAG Applications

RAG stands for Retrieval-Augmented Generation. A RAG application searches your own documents for relevant passages and hands those passages to a language model, which then writes an answer grounded in that material. The pattern lets a chatbot answer questions about company policies, product manuals, or research papers that the model never saw during training.

The Open-Book Exam Analogy

Picture two students taking a test. One student relies on memory alone and sometimes guesses. The other student may open a reference book, find the right page, and write an answer based on the text. RAG gives a language model the open-book advantage. The model looks up facts before it responds.

CLOSED BOOK VS OPEN BOOK
──────────────────────────────────────────────────────────
PLAIN MODEL:   Question ──► Model (memory only) ──► Answer (may guess)

RAG:           Question ──► Search your documents
                                  │
                       relevant passages
                                  ▼
                        Model reads passages ──► Answer with sources

Why RAG Helps

• Answers reflect current company information because the documents update independently of the model.
• Private data stays in your governed lakehouse instead of inside a model.
• Responses can cite the source passage, which builds trust.
• Teams avoid the cost and effort of training a custom model.

The Two Phases of RAG

Phase 1: Prepare the Knowledge

Engineers load documents, split them into passages, and store them in a Delta table. A Vector Search index then converts each passage into an embedding.

Phase 2: Answer Questions

The application embeds the user's question, retrieves the closest passages, builds a prompt that includes them, and asks the model to answer.

RAG PIPELINE
──────────────────────────────────────────────────────────
PHASE 1 (done ahead of time)
Documents ──► split into chunks ──► Delta table ──► Vector index

PHASE 2 (done for every question)
Question ──► retrieve top chunks ──► build prompt ──► model ──► answer

Chunking: Cutting Documents into Pieces

Models read a limited amount of text per request, and search works best on focused passages. Chunking splits long documents into short pieces, often 200 to 500 words each. Overlapping chunks by a few sentences keeps ideas from breaking at the edges.

CHUNKING A POLICY DOCUMENT
──────────────────────────────────────────────────────────
Leave Policy (12 pages)
   ├── Chunk 1: Annual leave rules
   ├── Chunk 2: Sick leave rules
   ├── Chunk 3: Parental leave rules
   └── Chunk 4: How to request leave
Each chunk keeps its document name and page number.

Write the Retrieval and Answer Steps

from openai import OpenAI

# Retrieve relevant chunks
hits = index.similarity_search(
    query_text="How many sick days do employees get?",
    columns=["chunk_text", "doc_name"],
    num_results=3
)
chunks = [row[0] for row in hits["result"]["data_array"]]
context = "\n\n".join(chunks)

# Build the prompt
prompt = f"""Answer the question using only the context below.
If the context does not contain the answer, say you do not know.

Context:
{context}

Question: How many sick days do employees get?"""

# Ask the model
client = OpenAI(api_key="YOUR_TOKEN",
                base_url="https://YOUR-WORKSPACE-HOST/serving-endpoints")

answer = client.chat.completions.create(
    model="YOUR-ENDPOINT-NAME",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.1
)
print(answer.choices[0].message.content)

The instruction "using only the context" keeps the model close to your documents. The permission to say "I do not know" reduces made-up answers.

Improve Answer Quality

Better Retrieval

Try hybrid search, add metadata filters, and tune chunk size. Poor retrieval causes most weak answers.

Better Prompts

State the role, the rules, and the format. Ask the model to quote the source name.

Better Data

Remove duplicate, outdated, and contradictory documents. A clean knowledge base produces clean answers.

Evaluate Before Launch

Build a test set of real questions with approved answers. Run the application on the test set and score the results for correctness, relevance, and grounding in the retrieved text. Mosaic AI Agent Evaluation and MLflow support this review process. Repeat the test after every change to chunking, prompts, or models.

Check	Question It Answers
Retrieval relevance	Did the search return useful passages?
Groundedness	Does the answer stay within the passages?
Correctness	Does the answer match the approved answer?
Safety	Does the answer avoid harmful or private content?

Protect Private Information

Unity Catalog permissions control which tables and indexes each user can read. Apply the same rules to RAG sources so the chatbot never reveals a document that the asking user cannot open. Remove personal data from documents when the use case does not require it.

Example: HR Policy Assistant

A company loads its employee handbook into a Delta table and builds a vector index. Employees ask questions such as "How do I request parental leave?" The assistant retrieves the leave chunks and writes a short answer with the handbook section name. A link to the full policy appears below every reply.

HR POLICY ASSISTANT
──────────────────────────────────────────────────────────
Employee question
      │
      ▼
Vector index returns 3 handbook chunks
      │
      ▼
Model writes a short answer + section name
      │
      ▼
Reply with a link to the full policy

Key Points

• RAG retrieves relevant passages and gives them to a language model before it answers.
• The pattern keeps answers current and grounded in private documents.
• Chunking splits documents into focused pieces for better search.
• Prompts tell the model to use only the supplied context.
• Test sets measure retrieval, groundedness, correctness, and safety.
• Unity Catalog permissions protect source documents.