LangChain Fallbacks and Retries

External services fail. A model provider may slow down, return a rate limit warning, or go offline for a few minutes. An application that crashes at every hiccup frustrates users. Retries and fallbacks make an app resilient. A retry tries the same step again. A fallback switches to a backup plan.

Road Trip Analogy

A driver who finds a blocked road waits a minute and tries again, since traffic often clears. A closed bridge calls for another route. Retries match the first situation, and fallbacks match the second.

 Request
    |
    v
 Primary model ──fails──→ Retry (up to 3 times, with waiting)
    |                          |
    | success                  | still failing
    v                          v
 Answer                  Fallback model ──fails──→ Friendly error message
                               |
                               v
                            Answer

Why Failures Happen

• Rate limits: too many requests in a short time.
• Timeouts: the provider takes too long.
• Server errors: a temporary fault on the provider's side.
• Bad output: the reply fails to match a required format.
• Context overflow: the prompt exceeds the model's window.

Built-In Model Retries

Most chat models accept a retry setting that handles short network faults.

from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-4o-mini", max_retries=3, timeout=30)

Retrying Any Runnable

The method with_retry wraps any chain. Exponential waiting doubles the pause after each failure, which gives a struggling service time to recover. Jitter adds a small random delay so many clients do not retry at the same instant.

reliable_chain = chain.with_retry(
    stop_after_attempt=3,
    wait_exponential_jitter=True,
)

 Attempt 1 fails → wait ~1s
 Attempt 2 fails → wait ~2s
 Attempt 3 fails → raise the error

Adding a Fallback Model

The method with_fallbacks lists backup runnables. LangChain tries them in order when the primary raises an error.

from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic

primary = ChatOpenAI(model="gpt-4o-mini", max_retries=1)
backup = ChatAnthropic(model="claude-haiku-4-5", max_retries=1)

safe_model = primary.with_fallbacks([backup])
reply = safe_model.invoke("Give me one tip for better sleep.")

Relying on two providers protects the app when one has an outage.

Fallbacks for Whole Chains

Fallbacks work on chains too. A strong but pricey chain can fall back to a cheaper chain, or a chain with a strict format can fall back to a looser one.

strict_chain = prompt | model.with_structured_output(Answer)
loose_chain = prompt | model | StrOutputParser()

robust = strict_chain.with_fallbacks([loose_chain])

Fallback for Context Overflow

A long prompt can overflow a small model. Point the fallback at a model with a larger window so the request still succeeds.

Limiting Which Errors Trigger Fallbacks

Some errors deserve no retry, such as an invalid API key. Retrying cannot fix a wrong key. Use the exceptions_to_handle option to list the error types that should trigger the backup.

robust = primary.with_fallbacks(
    [backup],
    exceptions_to_handle=(TimeoutError, ConnectionError),
)

Friendly Final Messages

When every path fails, show a clear message such as "The assistant is busy. Please try again in a minute." Avoid raw error text. Log the real error for developers.

try:
    reply = safe_model.invoke(question)
except Exception as err:
    print("The assistant is busy. Please try again in a minute.")
    log_error(err)

Common Mistakes

• Retrying endlessly and creating a traffic jam.
• Using a fallback with a different prompt style that breaks the output format.
• Ignoring cost differences between the primary and backup models.
• Skipping tests of the fallback path, which stays unused until the worst moment.

Real-World Scenario

A ticket booking app relies on one model provider. During a busy sale, the provider returns rate limit errors for several minutes. The app retries twice with growing waits, then switches to a backup provider. Customers notice nothing, and the sale continues. Afterward, the team reviews the logs, adds a second backup, and tunes the wait times. A weekly test forces the primary to fail, so the team knows the backup path works before the next real outage.

Practice Tasks

1. Wrap a chain with with_retry and force a failure to see the attempts.
2. Add a fallback model and simulate a failure on the primary.
3. Choose two exception types that should trigger a fallback and two that should not.
4. Write a friendly error message for users when every path fails.

Key Points

• Retries repeat a step. Fallbacks switch to another path.
• with_retry adds exponential waiting with jitter.
• with_fallbacks tries backup models or chains in order.
• Choose which errors should trigger a fallback.
• Show friendly messages and log the real errors.