Your Agent Keeps Calling Tools That Don't Exist: Fix the Toolset, Not the Retry

Your agent is running. It needs to look something up, and instead of calling search_documents, which is right there in the tool list, it calls document_search, which is not. The framework dutifully replies:

document_search is not a valid tool, try another one.

So the model tries another one. It calls find_document. Also not a tool. Then lookup_docs. Then search_docs. Each attempt is confident, plausible, and wrong, and your retry loop feeds every failure straight back into the context until you hit the max iterations and the run dies having accomplished nothing.

The instinct is to improve the error message or add smarter retry logic. That is treating the symptom. The model is not confused about what the error means. It is generating the most probable tool name for the job, and the most probable name is not the one you happened to choose. You fix that by changing the input, not by shouting the same correction louder.

The error you are seeing

Depending on your framework the wording varies, but it is some version of this, returned as the tool result and fed back to the model:

document_search is not a valid tool, try another one.

The tell that you are in the failure mode this article is about is not the error itself. It is that the error repeats with a different invented name each time. One wrong call is a typo the model can recover from. A loop of distinct wrong names means the model cannot find the right tool in the set you gave it, and no amount of retrying will conjure it.

Why the retry does not self-correct

A tool call is generated the same way as any other token sequence: the model predicts the most likely continuation given everything in context. When it emits document_search, it is not reading your tool list and picking wrong. It is producing the name that best fits the shape of the request, weighted by everything it has ever seen. If your actual tool is called search_documents and two dozen other tools are crowding the list, the probability mass for "the tool that searches documents" is smeared across several plausible names.

Now you append "document_search is not a valid tool, try another one" to the context. What did that add? It removed exactly one wrong answer from an infinite space of wrong answers. It did not tell the model which name is right. So the next most probable name comes out, and it is find_document, and you are no better off. This is why the loop cycles through synonyms: you are subtracting one at a time from a space you cannot exhaust.

Two underlying causes drive almost all of these loops:

• Too many tools. Every tool you bind competes for the model's attention in the same decision. Past a certain count, selection accuracy falls and the model starts blending names together.
• Ambiguous descriptions. If search_documents and query_knowledge_base and retrieve_context all describe themselves as "search for relevant information", the model has no signal to distinguish them and will pick, or invent, whichever fits the phrasing of the moment.

A minimal reproduction, and the description rewrite that fixes it

Here is the shape of a toolset that reliably produces the loop. The names are close, the descriptions overlap, and there are more of them than the model can keep straight.

# Reproduces the hallucination loop
tools = [
    {"name": "search_documents", "description": "Search for information"},
    {"name": "query_knowledge_base", "description": "Search the knowledge base for info"},
    {"name": "retrieve_context", "description": "Retrieve relevant context"},
    {"name": "find_records", "description": "Find records"},
    {"name": "lookup_data", "description": "Look up data"},
    # ...15 more, half of them overlapping...
]

Given "find the onboarding policy", the model has five tools that all claim to search, and it will happily emit document_search or find_document because those fit the request as well as any real name does. Now collapse the overlap and make each description say precisely what it takes and returns:

tools = [
    {
        "name": "search_documents",
        "description": (
            "Full-text search over the internal document store. "
            "Use for policies, guides, and written docs. "
            "Input: a natural-language query string. "
            "Returns: up to 5 matching document chunks with titles."
        ),
    },
    {
        "name": "get_customer",
        "description": (
            "Fetch one customer by their exact customer ID. "
            "Use ONLY when you already have the ID. "
            "Input: customer_id (string). Returns: the customer record."
        ),
    },
    # a handful of clearly distinct tools, not twenty near-duplicates
]

Two things changed. There is now one obvious tool for "search for a document", so the probability mass concentrates on the real name. And each description states its input and output, which stops the model reaching for a different search-like tool by mistake. In practice, tightening descriptions and cutting duplicates fixes the majority of these loops without touching the retry code at all. It is the same lesson as Stop Regex-Parsing LLM Output: constrain what the model can produce instead of cleaning up after it.

When you genuinely have many tools: namespacing and selection

Some agents legitimately need fifty tools. You cannot bind all fifty to one call and expect clean selection. Route instead. Group tools into namespaces and make selection a two-step decision.

1. Categorise. Prefix tools by domain: docs.search, docs.get, billing.get_invoice, billing.refund. The prefix is a hint the model can reason about, and it keeps unrelated names from colliding.
2. Pre-select a subset. Before the tool-calling step, use a cheap retrieval or classification pass to pick the five to ten tools relevant to the current request, and bind only those. The model never sees the other forty in that call, so it cannot pick from them and cannot blur them together.

The point is always the same: the number of candidates in a single decision is what drives accuracy. Namespacing and pre-selection both exist to keep that number small.

Retry policy: cap it, and correct with real names

Retrying is not useless, it is just usually done wrong. Two rules make it worth having:

• List the valid names verbatim. Do not say "try another one". Say which tools exist. That actually adds information the model can use.
• Cap it hard. One or two corrective attempts, then fail loudly. An uncapped loop is a token and latency sink that produces nothing.

MAX_TOOL_RETRIES = 2

def handle_invalid_tool(name, valid_names, attempt):
    if attempt >= MAX_TOOL_RETRIES:
        raise ToolSelectionError(
            f"Model requested unknown tool {name!r} "
            f"after {attempt} attempts. Valid tools: {valid_names}"
        )
    # Corrective message that actually helps: name the real options
    return (
        f"{name!r} is not available. "
        f"Choose exactly one of: {', '.join(valid_names)}."
    )

What people get wrong

Improving the error string. Rewording "is not a valid tool, try another one" into a friendlier sentence changes nothing, because the problem was never that the model misunderstood the message. It understood it fine and produced the next wrong name. Effort spent on error copy is effort not spent on the toolset.

Turning up the temperature to "explore" more. This is backwards. Higher temperature makes the model more likely to sample an off-list name, not less. For tool selection you generally want lower temperature, so the distribution concentrates on the most probable, and therefore most correct, name.

Adding more tools to cover the gap. If the model reached for document_search, the temptation is to add an alias tool by that name. Now you have two tools that do the same thing, the overlap that caused the problem is worse, and the next request finds a third gap. Consolidate, do not proliferate. Before shipping, put the toolset through an eval that measures selection accuracy, exactly as in How to Evaluate an LLM Feature Before It Touches a Customer.

When it is still broken

• The model calls a real tool with wrong arguments instead. That is a schema problem, not a selection one. Tighten the parameter descriptions and use strict/structured argument validation so bad arguments are rejected before execution.
• It picks the right tool but at the wrong time. Your descriptions describe what the tool does but not when to use it. Add an explicit "Use this when..." and "Do not use this for..." line to each.
• It works in isolation but loops inside a long conversation. The tool list is getting buried under conversation history. Keep the tool definitions near the end of the prompt or in the dedicated tools field, not stranded above thousands of tokens of chat.
• Only your smallest model does it. Selection is one of the first capabilities to degrade as models shrink. If a 7B model cannot hold fifteen tools straight, either route to fewer tools per call or use a larger model for the selection step specifically.

Stop trying to argue the model out of its wrong answers one at a time. Give it a short list of clearly distinct tools with names that match what it would naturally reach for, and the hallucinated calls mostly disappear on their own.