Why 'maximum context length exceeded' Keeps Happening After You Switched Models

You built a RAG chain, it worked on your test questions, you shipped it. A week later it starts throwing 400s in production on exactly the questions users care about, the long ones with lots of retrieved context. So you switch from gpt-3.5-turbo to a bigger-context model, redeploy, and the error goes away. For a while. Then it comes back, because you did not fix the bug, you just moved the wall further away.

The error is not "the model is too small". The error is that your chain builds the prompt by piling everything it has into one string and hoping it fits. That is not a strategy, it is a coin flip whose odds get worse as your corpus grows and your users get chattier. A bigger context window makes the coin flip come up heads more often. It does not make it stop being a coin flip.

The real fix is boring and permanent: count tokens before you build the prompt, and spend a fixed budget deliberately. This article shows the budget algorithm and the token-counting code, and why the popular advice ("just truncate", "just use a 128k model") is the thing keeping you stuck.

The exact error you are seeing

On the legacy openai Python SDK (below 1.0) it looks like this:

openai.error.InvalidRequestError: This model's maximum context length is 4097
tokens, however you requested 13886 tokens (13630 in your prompt; 256 for the
completion). Please reduce your prompt; or completion length.

On the current SDK (1.x and later) the same condition is a BadRequestError carrying HTTP 400, with the same message body:

openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's
maximum context length is 4097 tokens, however you requested 13886 tokens
(13630 in your prompt; 256 for the completion)...", 'type':
'invalid_request_error'}}

Read the numbers, because they tell you exactly what is wrong. 13630 in your prompt; 256 for the completion. The 256 is your max_tokens. The model does not have 4097 tokens for the prompt and 4097 for the answer; it has 4097 for both together. If you ask for a 256-token answer, your prompt gets 3841. Most chains never subtract the completion reserve, which is why they overshoot by a small, maddening amount.

Which chains trigger it, and why bigger models do not fix it

In LangChain the usual culprits are the ones that stuff retrieved documents into the prompt without a token check: RetrievalQAWithSourcesChain, ConversationalRetrievalChain (history plus retrieved docs, so it grows on two axes at once), and SQLDatabaseChain when the table schema or sample rows are large. The default stuff document chain does exactly what its name says.

Here is the mechanism people miss. Your prompt size is roughly:

tokens = system_prompt
       + conversation_history        (grows every turn)
       + k * avg_chunk_size          (grows as you retrieve more)
       + user_question
       + max_tokens (completion)     (reserved, not optional)

Switching to a 128k model multiplies the ceiling but changes none of the growth terms. A support bot that retrieves k=8 chunks today will retrieve k=8 larger chunks after you re-index a bigger knowledge base, hold a longer conversation with a persistent user, and overflow 128k the same way it overflowed 4k. You have paid roughly ten times as much per call to buy time, not a fix. That is the trap: the bigger model hides the bug during exactly the testing window where you would have caught it.

Why "just truncate the prompt" is the wrong instinct

The advice you will find on the first page of results is to truncate the prompt to fit. Think about what that does mechanically. A naive truncate slices the tail off a concatenated string. In a RAG prompt the tail is usually the last retrieved chunk and the user's question. So truncation preferentially deletes the thing the user asked and keeps the boilerplate system header. The request now succeeds and returns a confident, wrong answer, which is strictly worse than a 400 you could have caught.

Even "smart" truncation that drops whole chunks from the end is wrong, because retrieval order is not relevance order once you have reranked or when your vector store returns by index position. You must drop by score, keeping the chunks most likely to contain the answer. Truncation throws away information blind; budgeting throws away the least valuable information on purpose.

The fix: count tokens, then allocate a budget

Step 1: A token counter for the model you actually call

For OpenAI models, tiktoken gives an exact count. Load the encoding by model so you get the right one automatically:

import tiktoken

def count_openai_tokens(text: str, model: str = "gpt-4o-mini") -> int:
    try:
        enc = tiktoken.encoding_for_model(model)
    except KeyError:
        enc = tiktoken.get_encoding("o200k_base")  # current default family
    return len(enc.encode(text))

Do not reuse this number for other providers. Anthropic's Claude models use a different tokenizer, and the honest way to count Claude tokens is the API's own count endpoint rather than a guessed local encoder. A local Llama or Mistral model uses its own SentencePiece or BPE tokenizer, which you load from the model files. A rough character heuristic (four characters per token) is fine for a safety margin but never for the actual budget check, because the models that bite you are the ones near the edge.

def count_tokens(text: str, provider: str, model: str, tokenizer=None) -> int:
    if provider == "openai":
        return count_openai_tokens(text, model)
    if provider == "anthropic":
        # use the Anthropic client's token counting endpoint, not tiktoken
        return anthropic_client.messages.count_tokens(
            model=model,
            messages=[{"role": "user", "content": text}],
        ).input_tokens
    if provider == "local" and tokenizer is not None:
        return len(tokenizer.encode(text))
    # last-resort ceiling estimate, deliberately pessimistic
    return len(text) // 3

Step 2: The budget allocator

Spend the context window in priority order. Reserve the completion, take the system prompt off the top, keep as much recent history as fits, then pack chunks by descending score:

from dataclasses import dataclass

@dataclass
class Chunk:
    text: str
    score: float

def build_prompt(model_ctx: int, completion_reserve: int,
                 system_prompt: str, history: list[str],
                 chunks: list[Chunk], question: str,
                 count) -> tuple[str, list[Chunk]]:
    # 1. completion reserve is subtracted first
    budget = model_ctx - completion_reserve

    # 2. system prompt and the question are mandatory
    fixed = count(system_prompt) + count(question)
    budget -= fixed
    if budget < 0:
        raise ValueError("system prompt + question alone exceed the window")

    # 3. history, newest first, drop oldest turns that do not fit
    kept_history = []
    for turn in reversed(history):
        t = count(turn)
        if t <= budget:
            kept_history.insert(0, turn)
            budget -= t
        else:
            break

    # 4. fill remaining budget with highest-scoring chunks
    used = []
    for ch in sorted(chunks, key=lambda c: c.score, reverse=True):
        t = count(ch.text)
        if t <= budget:
            used.append(ch)
            budget -= t

    prompt = "\n\n".join(
        [system_prompt, *kept_history,
         *[c.text for c in used], question]
    )
    return prompt, used

Two properties make this correct where truncation is not. The completion reserve is subtracted before anything else, so you never hit the 256 for the completion overshoot. And chunks are dropped by lowest score, so the answer-bearing context survives while filler is discarded. If even the system prompt and question do not fit, you get a clear error at build time instead of a 400 from the API, which is the moment to shorten the system prompt or split the question, not to reach for a bigger model.

Step 3: Wire it into the chain

If you are on LangChain and do not want to hand-roll retrieval, the framework-native version of this idea is ConversationTokenBufferMemory for history and a document chain type other than stuff. But the moment your correctness depends on which chunks survive, own the packing step yourself as above and pass the assembled context in. Do not let a hidden stuff chain make the decision for you.

Verification: prove it stays under the limit

Add an assertion right before the API call and log the breakdown. If the assert never fires under a load test that includes your longest real questions and longest conversations, the bug is gone, not hidden:

prompt, used = build_prompt(
    model_ctx=128000, completion_reserve=1024,
    system_prompt=SYSTEM, history=history, chunks=retrieved,
    question=q, count=lambda t: count_tokens(t, "openai", "gpt-4o-mini"),
)
total = count_tokens(prompt, "openai", "gpt-4o-mini") + 1024
assert total <= 128000, f"budget bug: {total}"
print(f"prompt={total-1024} completion=1024 chunks_kept={len(used)}")

The chunks_kept number is your early warning: if it starts dropping as your corpus grows, that is retrieval quality quietly degrading, which I cover in why RAG retrieval gets worse as the corpus grows. Budgeting caps the cost; it does not improve what you retrieve.

What people get wrong

"Switch to gpt-4o / a 128k or 1M context model." A band-aid, not a fix. It raises the ceiling and reappears at scale, and you pay for every one of those tokens on every call. Budgeting first means the bigger model becomes an option you choose for quality, not a crutch you need for correctness.

"Just truncate to the last N characters." This deletes the end of the prompt, which is usually the highest-value chunk and the user's question, and turns a loud 400 into a quiet wrong answer.

"One token counter for every model." tiktoken counts OpenAI tokens. Claude and local models tokenize differently, sometimes by 20 to 30 percent on the same text. Counting with the wrong tokenizer either wastes budget or overshoots the real limit. This is the same class of bug as a token counter that overcounts base64 attachments.

"Set max_tokens really high to be safe." Backwards. max_tokens is subtracted from your prompt budget. A huge completion reserve leaves almost no room for context and makes the overflow happen sooner.

When it is still broken

• Function/tool schemas count too. If you pass tool definitions, their JSON schema is part of the prompt tokens. A large tool list can silently eat a thousand tokens you never budgeted. Count them.
• Streaming does not change the limit. Streaming affects how the answer arrives, not the context math. The 400 fires before the first token streams.
• The message wrapper has overhead. Chat format adds a few tokens per message for role markers. If you are within a hair of the limit, add a small fixed per-message pad to your count.
• Embeddings have their own, separate limit. If the error is on an embeddings call, not a chat call, you are sending a chunk that is too long to embed; split at ingestion time, not query time.

Get the budget right once and the whole class of error disappears, on every model, at every corpus size. That is the difference between fixing the bug and renting distance from it.