Why Tool Use and Function Calling Keep Failing Silently

Reliable tool use and function calling is not a prompting trick. It is an engineering discipline: strict schema validation, deliberate control over when the model must call a tool versus when it decides on its own, parallel calls only where the work is genuinely independent, and an eval harness that catches a wrong tool selection before a customer does.

Picture a support agent wired to two hundred internal tools: refund lookups, shipping trackers, account resets, escalation queues. A customer asks about a delayed order. The agent calls a tool, gets back a clean JSON payload, and the trace looks perfect. Except it queried the wrong shipping provider's API, because two tools had near-identical descriptions and the model picked the one that scored slightly higher on its own ranking. Nothing in the log says "error." The output is wrong, four steps downstream, and nobody notices until a customer does.

That is the failure mode this piece is about. The model picks the tool and executes the call. Your job is designing the guardrails around that choice, and evaluating whether it picked the right tool with the right arguments, every time, not just in the demo.

Key takeaways

• Schema-valid JSON is not the same as a correct tool call. A call can pass every validation check and still hit the wrong endpoint or the wrong tool entirely.
• Forcing tool use changes cost, not just behavior. Claude Sonnet 5's tool-use system prompt runs 354 tokens under auto tool choice and 474 under a forced choice, on every single request.
• Parallel tool calls only save latency when the calls are genuinely independent. Fire two calls where the second depends on the first's result and you get a race condition, not a speedup.
• Scaling past a dozen tools with on-demand discovery fixes the context-window and cost problem, and introduces a quieter failure mode: a wrong-tool call that still executes successfully.
• Tool-use accuracy needs its own eval, separate from schema validation. The Berkeley Function Calling Leaderboard (BFCL V4) tracks accuracy, cost, and latency together, because a model that is usually right and occasionally silent is a different risk than one that is usually right and occasionally confidently wrong.

I go deep on the discipline of scoping what a tool is allowed to touch, before you ever get near an eval, in Agents That Actually Work. This piece is about the layer under that: the mechanics of the call itself.

What "reliable" tool calling actually means

Most teams measure tool-use reliability by whether the JSON parses. That is necessary and nowhere near sufficient. A schema-valid call can still name the wrong tool, fill a correct field with a plausible but wrong value, or call a real tool against the wrong record. Schema conformance tells you the model followed the contract's shape. It says nothing about whether the model understood the user's intent.

I define reliable tool use as three properties holding together on every call: the right tool for the intent, the right arguments for that tool, and a result your system can verify before it acts on it. Drop any one of the three and you have a system that looks clean in the trace and is wrong in production.

Forced vs. auto tool choice: when to constrain the model's decision

Most APIs give you three tool-choice modes: let the model decide (auto), forbid tool calls entirely (none), or force a call (any, or a named tool). The choice is not just a reliability lever. It is a cost line item, and it shows up on the invoice whether or not anyone on the team is watching for it.

Anthropic's own pricing table makes the gap concrete. Claude Opus 5 adds 286 tokens of system-prompt overhead under auto or none, and 406 tokens when you force a call with any or tool. Claude Sonnet 5 runs 354 tokens versus 474. That roughly 100-to-150-token gap holds, in similar proportion, across every current model in the lineup, and it is charged on every request, not once per session.

Forcing a call earns its cost when the workflow has no valid text-only branch: a data-extraction pipeline that must always emit a structured record, a routing agent that must always pick a queue. Leave tool choice on auto when a direct answer is a legitimate outcome, a support bot answering from memory instead of hitting an API for something it already knows. Forcing tool use on a task that sometimes has no tool-shaped answer produces calls that exist to satisfy the constraint, not the customer, and you pay the higher rate for the privilege.

Strict schema validation: closing the gap between valid JSON and correct call

Strict schema validation, the strict: true flag most providers now support on tool definitions, guarantees the shape of the output. It does not guarantee the model chose the right shape to begin with.

The gap between valid JSON and a correct call is where most production incidents live. A refund tool and a chargeback tool can both accept an order_id and a reason field. Strict validation cannot tell you the model picked the wrong one, because both calls are perfectly well-formed. Only a check that compares the call against known intent, run as an eval, catches that class of error. I treat schema validation as the floor, not the finish line. It rules out malformed output so the eval harness can spend its effort on whether the output is right.

Parallel tool calls: what breaks when calls aren't actually independent

Parallel tool calls are the obvious latency win. Instead of calling three tools in sequence, the model requests all three in one turn and your application executes them concurrently. That only works when the three calls do not depend on each other's output.

Picture an order-management agent that checks inventory and reserves an item in what looks like a single parallel step. If the reservation call fires before the inventory check resolves, you can reserve stock that was never confirmed available, and now two systems disagree about what is true. The fix is not to disable parallel calls. It is to be explicit about which calls are truly independent, reads against different systems, and which have a data dependency that forces sequencing, any call whose arguments come from another call's result.

Scaling past a dozen tools: on-demand discovery instead of context stuffing

Stuff fifty tool schemas into every request and you pay for fifty tool descriptions on every turn, whether the task needs one or ten. Past roughly a dozen tools, most teams move to on-demand discovery: the agent searches a tool index for what it needs and loads only the relevant schemas into context. Anthropic ships this as a server-side tool search tool for exactly this reason, and the underlying idea is not new. The 2023 Gorilla paper out of UC Berkeley paired a fine-tuned model with a document retriever and beat GPT-4 at generating correct API calls, by retrieving the right documentation instead of holding all of it in context at once.

Discovery solves the cost and context-window problem. It trades a loud failure for a quiet one. A wrong-argument error used to surface fast, in the tool result itself. A wrong-tool error from bad retrieval does not: the call executes successfully, against the wrong capability, and nothing in the trace looks broken until the output is wrong downstream. If you scale tool count with discovery, discovery needs its own eval, separate from your argument-schema checks. This is the same shape of problem I cover in how orchestration patterns decompose work across specialized agents. The more you delegate the choice of which capability handles a task, the more that choice itself becomes the thing you have to measure.

Handling tool errors and retries without failing silently

A tool call fails for reasons that have nothing to do with the model: a timeout, a rate limit, a downstream service returning a 500. How you report that failure back to the model decides whether it retries sensibly or fabricates a result and moves on.

Return the raw exception and most models either give up or invent a plausible-sounding answer instead of a real one. Return a structured error, error type, whether it is retryable, what changed since the last attempt, and the model can decide correctly: retry a timeout once, escalate a permission error immediately, do not retry a validation error with the same bad arguments. I cap retries at a fixed budget per tool call and force an explicit "I could not complete this" response when the budget runs out. Silent infinite retries and silent fabricated success are the same failure wearing different symptoms.

Evaluating tool-use accuracy: benchmarks and building your own harness

Public benchmarks tell you how a model performs on someone else's tools. The Berkeley Function Calling Leaderboard, now on version 4, separates models with native function-calling support from those faking it through prompted text generation, and it tracks cost and latency next to accuracy instead of accuracy alone. That is a useful sanity check when picking a model. It is not a substitute for testing your own tools.

Your harness needs three things a public benchmark cannot give you: your actual tool schemas, your actual failure modes (the near-duplicate tools, the ambiguous field names you already know are a problem), and a judge that scores "right tool, right arguments" against your ground truth, not a generic rubric. This is the same discipline that runs through every part of the AI-Native thesis: the machine does the work of selecting and calling the tool, and the human's job contracts to building and reading the eval that tells you whether it chose correctly.

MCP as the interoperability layer: one protocol instead of N custom integrations

Every tool you wire up by hand is a custom integration with its own auth, its own schema quirks, its own error format. The Model Context Protocol standardizes that connection. Anthropic describes it as a shared way to connect models to tools and data sources, now adopted across Claude, ChatGPT, VS Code, and Cursor, closer to a USB-C port for AI applications than a proprietary integration layer.

MCP does not fix tool selection or argument accuracy. It removes the N-times-M integration tax, so the engineering effort you would have spent writing custom connectors goes into the part that determines reliability: schema strictness, tool-choice policy, and the eval harness. When I am helping a team decide between building custom tool integrations and adopting MCP servers, the answer usually comes down to how many external systems they are already connecting and how much of that plumbing they want to own going forward. If tool calls need to hand off work to other agents rather than just call an API, the same interoperability question shows up again in a different shape; I cover how that handoff should be structured in agent-to-agent communication.

The same discipline scales beyond a single agent. Once tool calls start triggering handoffs between multiple agents instead of a single loop, the coordination problem changes shape entirely; I map that out in multi-agent systems and orchestration.

What's the difference between tool use and function calling?

They are the same mechanism under different names. Function calling was the original term, describing a model returning a structured call to a named function with arguments. Tool use is the broader label most providers use now, because the "function" can be a search, a database query, a file edit, or a call to another agent, not just a narrow API function.

How do I stop an AI agent from calling the wrong tool?

Start by making tool names and descriptions unambiguous. Near-duplicate tools with similar descriptions are the most common cause of wrong-tool selection. Then add an eval that specifically checks tool selection against ground truth, separate from argument-schema validation, because a schema check happily passes on a perfectly formed call to the wrong tool.

Should I force tool use or let the model decide when to call one?

Force it only when a text-only response is never a valid outcome for the task, like a pipeline that must always emit a structured record. Leave it on auto whenever a direct answer is legitimate, because forcing tool use on those tasks produces calls that exist to satisfy the constraint rather than the question, and it costs more tokens on every request regardless.

How many tools can an LLM agent reliably work with at once?

Most teams see tool-selection accuracy hold up cleanly through roughly a dozen well-differentiated tools in context at once. Past that, on-demand discovery, retrieving only the relevant schemas for a given task, beats stuffing every schema into every request, both on cost and on accuracy, provided you evaluate the discovery step itself and not just the calls it produces.

If you are staffing a team to build this, the schema discipline, the tool-choice policy, the eval harness, that work is exactly what a ViitorCloud AI engineer ships from day one. Reliable tool use gets built in before the first tool goes live, not bolted on after the first incident.