LangChain LangSmith Tracing

An AI application behaves like a black box. A question goes in, an answer comes out, and the steps between stay hidden. When the answer is wrong, guessing the cause wastes hours. LangSmith opens the box. It records every step as a trace, so you can see prompts, model replies, tool calls, timings, and costs in one place.

Flight Recorder Analogy

Airplanes carry flight recorders that capture every action. After an incident, investigators replay the recording and find the cause. A trace plays the same role for your AI app.

 TRACE: "Summarize my refund policy"
 |
 +-- Retriever          0.4s   returned 4 chunks
 |
 +-- Prompt builder     0.0s   1,250 input tokens
 |
 +-- Chat model         2.1s   180 output tokens   $0.0004
 |
 +-- Output parser      0.0s
 |
 Total: 2.5s

What a Trace Contains

  • Inputs and outputs for every step.
  • Timing for each step and for the whole run.
  • Token counts and cost for model calls.
  • Errors with the step that failed.
  • Metadata and tags such as user type or app version.

Turning Tracing On

Create a free account at the LangSmith website and generate an API key. Then set environment variables. No code changes are needed for LangChain and LangGraph apps.

pip install langsmith
import os

os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_API_KEY"] = "your-key-here"
os.environ["LANGSMITH_PROJECT"] = "support-bot"

Run your chain as usual. Open the project in the LangSmith dashboard and every call appears as a trace.

Tracing Your Own Python Functions

The traceable decorator records functions that are not part of LangChain.

from langsmith import traceable

@traceable(name="clean_user_text")
def clean(text):
    return text.strip().lower()

The function now appears as a step inside the parent trace.

Adding Tags and Metadata

Tags and metadata make traces searchable. Attach them in the run config.

result = chain.invoke(
    {"question": "What is the refund window?"},
    config={
        "tags": ["production", "refund-flow"],
        "metadata": {"user_tier": "free", "app_version": "1.4"},
    },
)

Later, filter the dashboard by tag to study one feature or one group of users.

Debugging a Bad Answer

  1. Open the trace for the failing question.
  2. Check the retriever step. Wrong chunks explain most bad answers.
  3. Read the final prompt. Missing context or confusing instructions show up here.
  4. Inspect the model reply and the parser. Format errors appear at this stage.
  5. Fix the weakest step and run the question again.

The Playground

Open any model step in the playground to edit the prompt and rerun it in the browser. Quick experiments there save time compared to changing code and restarting the app.

Watching Cost and Speed

Dashboards chart latency, error rate, and token use over time. A sudden jump signals a problem, such as a prompt that grew too long or a tool that slowed down. Set alerts so the team learns about issues before users complain.

Privacy Considerations

  • Traces store prompts and replies, which may include personal data.
  • Mask sensitive fields before sending them to a tracing service.
  • Limit dashboard access to people who need it.
  • Follow your company's data retention rules.

Common Mistakes

  • Tracing disabled in production, which leaves no clues during an incident.
  • Skipping tags, which makes thousands of traces hard to search.
  • Sharing API keys in public code.

Real-World Scenario

A support chatbot starts giving odd answers about delivery times. The developer opens the failing trace and finds that the retriever returned an outdated policy page. The prompt and model looked fine, and the real problem hid in the data. The developer removes the old page, reruns the question, and the answer corrects itself. The trace saved hours of guessing and pointed straight at the weak step.

Practice Tasks

  1. Turn on tracing with environment variables and run one chain.
  2. Add tags to a run and filter by that tag in the dashboard.
  3. Decorate a custom cleaning function with traceable and find it in the trace.
  4. Compare token counts and latency across two prompt versions.

Key Points

  • A trace records every step of a run with inputs, outputs, time, and cost.
  • Environment variables switch tracing on without code changes.
  • traceable adds custom Python functions to traces.
  • Tags and metadata make traces searchable.
  • Read traces from the retriever to the parser to find the weak step.
Previous lesson
Back to course
Next lesson