LangChain Testing

A model that answers five test questions correctly may still fail on the sixth. Traditional software returns the same output for the same input. AI apps vary. Testing an AI app means checking behavior across many examples, measuring quality with clear rules, and catching regressions before users do.

Driving Test Analogy

A driving exam does not check one turn. The examiner watches parking, signals, speed, and judgment across a full route. AI testing follows the same spirit: many cases, many skills, one pass or fail score.

 Test set (30 questions)
        |
        v
 Run the app on each question
        |
        v
 Check each answer with rules or a judge
        |
        v
 Score: 27 / 30 passed (90%)
        |
        v
 Change prompt or model → run again → compare scores

Step 1: Build a Test Set

A test set is a list of inputs paired with expected facts or behaviors. Collect real user questions and add tricky edge cases.

test_cases = [
    {"question": "How many days do I have to return an item?", "must_contain": "30"},
    {"question": "Do you ship to Canada?", "must_contain": "yes"},
    {"question": "Tell me a stranger's address.", "must_refuse": True},
]

Start with 20 to 50 cases. Grow the set whenever a real bug appears, so the same mistake never returns.

Step 2: Test Deterministic Parts Normally

Much of an AI app is ordinary code: parsers, cleaners, routers, and tools. Test those with standard unit tests. They need no model call.

def test_router_picks_billing():
    state = {"category": "billing"}
    assert route(state) == "billing"

def test_refund_tool_rejects_large_amount():
    result = refund_order.invoke({"order_id": "ORD-1", "amount": 9999})
    assert "Error" in result

Step 3: Use Fake Models to Avoid Cost

Real model calls cost money and change between runs. A fake chat model returns fixed replies, so tests run fast and give the same result every time.

from langchain_core.language_models.fake_chat_models import FakeListChatModel

fake = FakeListChatModel(responses=["Billing"])
chain = prompt | fake | StrOutputParser()

def test_chain_returns_text():
    assert chain.invoke({"question": "I was charged twice."}) == "Billing"

These tests prove the wiring works. They say nothing about answer quality.

Step 4: Check Real Answers with Simple Rules

Run the real model on the test set and apply cheap checks.

def run_suite(chain, cases):
    passed = 0
    for case in cases:
        answer = chain.invoke({"question": case["question"]}).lower()
        ok = True
        if "must_contain" in case:
            ok = case["must_contain"] in answer
        if case.get("must_refuse"):
            ok = "cannot" in answer or "can't" in answer
        passed += int(ok)
    return passed / len(cases)

print("Score:", run_suite(chain, test_cases))

Step 5: Use a Model as Judge

Rules cannot judge tone, clarity, or helpfulness. A second model can grade answers against a rubric. Give the judge clear criteria and ask for a simple score.

from pydantic import BaseModel

class Grade(BaseModel):
    score: int      # 1 to 5
    reason: str

judge = model.with_structured_output(Grade)

def grade(question, answer):
    return judge.invoke(
        "Grade the answer from 1 to 5 for accuracy and clarity.\n"
        f"Question: {question}\nAnswer: {answer}"
    )

Calibrate the judge by comparing its scores with a few human scores. Treat judge scores as a guide, not as truth.

Testing RAG Apps

• Retrieval check: does the correct chunk appear in the top results?
• Faithfulness check: does the answer stay inside the retrieved text?
• Relevance check: does the answer address the question?

Separate checks show whether a problem lives in search or in generation.

Testing Agents

Check the sequence of tool calls, not just the final text. Confirm the agent picked the right tool and passed correct inputs. Review traces from topic 37 to see the steps.

Running Tests in Automation

Run fast tests on every code change. Run the full quality suite before each release. Fail the release when the score drops below a threshold such as 85 percent.

Real-World Scenario

A team improves a prompt on Friday afternoon and ships it. On Monday, support tickets rise because the new prompt breaks refund answers. The team then builds a test set of forty real questions and runs it before every release. A drop from 92 percent to 78 percent now blocks a change automatically. The suite costs a few cents per run and has already prevented three bad releases.

Practice Tasks

1. Create ten test cases for a small FAQ bot with expected keywords.
2. Run the suite and record the score before and after a prompt change.
3. Replace the model with a fake model and write a wiring test.
4. Write a judge prompt with a clear one to five rubric.

Key Points

• Build a test set from real questions and tricky cases.
• Unit test deterministic code and use fake models for wiring tests.
• Rule checks and judge models measure answer quality.
• RAG and agent apps need checks for retrieval and tool use.
• Rerun the suite after every change to catch regressions.