LangChain Context Windows
A model cannot read unlimited text. Every model owns a fixed reading space called the context window. Everything you send, plus everything the model writes back, must fit inside that space. Understanding this limit saves money and prevents confusing errors.
What Is a Token
Models do not read whole words. They read tokens, which are small chunks of text. A short common word usually equals one token. A long or rare word splits into several. As a rough guide, one token equals about four English characters, and 100 tokens equal about 75 words.
Text: "LangChain makes AI apps simple" Tokens: [Lang] [Chain] [ makes] [ AI] [ apps] [ simple] Count: 6 tokens
The Window as a Desk
Picture a desk with a fixed size. Papers pile up on the desk during a conversation: instructions, history, documents, and the model's own answers. A full desk has no room for new papers. Once the window fills, the call fails or the oldest content disappears.
CONTEXT WINDOW (example: 8,000 tokens) +-----------------------------------------------+ | System rules | 300 tokens | | Chat history | 3,200 tokens | | Retrieved text | 2,500 tokens | | New question | 100 tokens | | Room for reply | 1,900 tokens | +-----------------------------------------------+
Input Tokens and Output Tokens
Providers count input tokens and output tokens separately. Output tokens usually cost more. Both count toward the window. A long prompt leaves less room for the answer.
Reading Token Usage in LangChain
Every reply carries usage data.
reply = model.invoke("Explain gravity in two sentences.")
usage = reply.usage_metadata
print(usage["input_tokens"])
print(usage["output_tokens"])
print(usage["total_tokens"])Counting Before You Send
Counting first lets your code decide whether the text fits.
text = "A long document goes here..."
count = model.get_num_tokens(text)
print(count)For message lists, the helper count_tokens_approximately gives a fast estimate.
from langchain_core.messages.utils import count_tokens_approximately
total = count_tokens_approximately(messages)Why Cost Grows Quickly
Chat applications resend the whole history with every turn. Turn one sends 100 tokens. Turn two sends 100 plus the first answer plus the new question. By turn twenty the request may carry thousands of tokens, and each token costs money.
Turn 1: [Q1] → small Turn 2: [Q1][A1][Q2] → medium Turn 3: [Q1][A1][Q2][A2][Q3] → larger Turn 20: [ ... long history ... ] → very large
Bigger Windows Are Not a Free Fix
Some models offer very large windows. Large windows still cost more and respond slower. Models also pay less attention to details buried in the middle of a huge prompt. Smart selection of content beats stuffing everything in.
Strategies for Staying Inside the Limit
Trim Old Messages
Keep only the most recent turns. Topic 14 shows how.
Summarize Early History
Replace old turns with a short summary that keeps the key facts.
Retrieve Instead of Paste
Send only the document chunks relevant to the question. Retrieval chapters in this course explain the method.
Limit Reply Length
Set a max token value so answers never swallow the remaining space.
Common Mistakes
- Pasting a whole book into one prompt.
- Ignoring reply space when planning the prompt size.
- Assuming characters equal tokens.
- Skipping usage logs, which hides cost growth.
Real-World Scenario
A study assistant helps a student revise a long science chapter. At first the app pastes the entire chapter into every prompt. Costs climb and replies slow down. The developer counts tokens, finds that the chapter alone takes 9,000 tokens, and switches to a retriever that sends only the three most relevant paragraphs. The prompt shrinks to 1,200 tokens. Answers arrive faster, the bill drops, and the quality stays the same. Measuring the window first led to the fix.
Practice Tasks
- Count the tokens of a short story paragraph and estimate its word count.
- Build a chat loop that prints total tokens after each turn and watch the growth.
- Set a max token limit of 50 and observe how the reply changes.
- Calculate the cost of a conversation with twenty turns if every turn resends the full history.
Key Points
- The context window is the total space for input and output.
- Tokens are text chunks, roughly four characters each.
- Chat history grows with every turn and raises cost.
- usage_metadata and get_num_tokens reveal token counts.
- Trimming, summarizing, and retrieval keep prompts small.