How reliably do local models actually call tools?
We had a number. It came from twelve calls. So we ran thirteen hundred and found out it belonged to a different model.
One sentence, load-bearing
A self-hosted agent runs nineteen applications from a chat window. Since it moved off a cloud provider, every model call in the fleet goes through one local gateway. Its documentation says this:
The Spark returns a tool call as prose in roughly 8% of calls. Measured: 11/12, then 1 failure printing<tools>{"name": ...}</tools>withfinish_reason: stop. It is not a broken gateway and not a wrong model — it is a rate. Anything calling tools in a loop must expect it and retry. chezmoi · docs/AGENTS.md
The reasoning is right, and it changed the code: agent loops retry instead of erroring. The trouble is the evidence. One failure in twelve.
What twelve calls can tell you
A rate quoted without an interval is the reason this study exists. Drag the sample size and watch what the same 8% is actually compatible with.
At n=12 the interval runs from 1.5% to 35.4%. That range spans three different engineering decisions: at 1.5% nobody notices, at 8% retries are load-bearing, and at 35% the model is not usable for tool calling and the fleet should go back to a cloud provider. The observation could not tell them apart, and re-reading it never will.
Two things we had to measure before designing anything
Temperature zero is not a sample
Repetitions only count if the system is stochastic. It is not: the same prompt sent eight times returned byte-identical tool calls, 8/8, on two different models. Fifty trials at temperature 0 would have produced a table that looked exactly like a sample of fifty and a confidence interval that was fiction. Everything here runs at the production temperature=0.3, and every task carries four phrasings.
Concurrency buys nothing
The intuition that four workers get four times the work done is false on this engine. Generation throughput is flat; only latency moves.
Ollama serialises. Four workers finish the same work in the same wall-clock while each request waits behind three others — so a run at concurrency 4 costs nothing extra and returns latency figures inflated by queueing that bought nothing. All results here were collected at concurrency 1.
Five outcomes, because two of them look identical
The naive check is if response.tool_calls: — worked or didn't. That check cannot see the failure this study is about. When a model serialises a call into ordinary prose, the array is empty and the reply is chatty, so the check records a refusal. It was not a refusal: the model picked the right tool and wrote it into a field nobody reads. One is a parsing problem a retry clears; the other is a prompting problem retrying does nothing about.
The same five-way classifier the benchmark runs, in your browser. Load a real response from the run, or write your own and watch it get scored.
The failure is real. It belongs to a different model.
The documented shape reproduces character for character — but on the fleet default it happens at 0.5%, not 8%. The 8% belongs to coder, and coder fails in a completely different syntax.
Of calls on the twelve tasks that require a tool. A structured call is the only shape a caller can consume; anything here is a call the loop never sees.
Two models, two syntaxes. A parser written from either sample recovers nothing from the other:
This matters more than the rate. Retrying is cheap, but in all thirteen coder cases the tool and the arguments were correct — only the envelope was wrong. A shape-agnostic parser recovers the call instead of spending another generation on it.
The failures are also concentrated rather than uniform. All thirteen fall on three of the twelve tool tasks and none on the other nine, so an aggregate figure overstates the risk almost everywhere and understates it on the requests that actually trigger it.
The benchmark was wrong about the best-behaved model
The first scoring put chat last, at 68.5%. Then we read the responses it had marked as failures.
prompt · "moude has been flaky since this morning. Restart it."
"I need to confirm with you before restarting Moude, as it will stop the service. Are you sure you want to proceed with the restart now?"
graded: no tool_calls → refusal → 0
The tool's own description ends: "Confirm with Erwin before stopping something he may be using." The model was obeying it. The grader counted obedience as incapability.
Same 1310 stored responses, re-scored with a confirmation outcome added. No further GPU time — which is the entire argument for storing responses rather than verdicts.
chat asked to confirm on 58.3% of its calls on the two destructive tasks. No other model did it once. The ranking had inverted for the model behaving most carefully.
This is not specific to our benchmark. Any tool-use evaluation that scores "did it emit a call" penalises exactly the caution that safety-oriented tool descriptions are written to produce — and the penalty is invisible unless someone reads the prose.
A second error of the same kind: scoring memory_save as the only right answer to "remember this" put every model between 0% and 28%. Of 72 calls, 52 chose remember_fact — a different tool that also saves a durable fact, with nothing in either description telling a user's phrasing which to use. That is a finding about the tool belt, not about the models.
Argument construction is solved. Choosing the tool is not.
Once a model emits a structured call to the right tool, the arguments are schema-valid essentially always — 2 failures in 764 calls, no invented enum members, no missing required fields. Every point of loss is in selection. Effort spent on argument-repair logic in an agent loop is effort spent on the part that already works.
And the small models win. The two 8B aliases are indistinguishable from each other and both beat the 80B and the 30B. On a belt of sixteen clearly-described tools, parameter count did not buy tool-calling accuracy.
A task where every model fails is usually a task problem; a task where they split is a model difference. home_control is the clearest split — the large models call home_states first, a reasonable opening move in a multi-turn plan that a single-turn benchmark cannot credit.
The reasoning budget is a cliff, not a gradient
Second agent turn: the model has called a tool, the full result is back in context, and it is asked an open question about it. Sweep the token budget and watch the answer disappear.
Tools stay bound, as they are in production, so a model may answer, call another tool, or return nothing. "Returned nothing" means empty content — the budget went on reasoning and no answer came back.
There is no budget at which the model returns a shorter answer instead of no answer. Below the threshold the failure is total. Mean completion length plateaus around 2 750 tokens once the budget stops binding, so the documented setting of 8192 is not over-provisioned — it is about 3× the requirement, which is the right margin for a distribution whose failures are all-or-nothing.
The same failure turned up somewhere nobody was watching for it: on open-ended knowledge questions the 8B vision model spent its entire budget and returned nothing 26% of the time.
Same accuracy, seventeen times the wait
Bubble area is mean completion tokens — the reason for the gap. It is not generation speed: the thinking model generates faster per token and spends about 1 200 of them producing a one-line tool call. Thinking is billed in wall-clock.
The p95 is the number that matters for a chat surface. For chat it is 69.7 s against a median of 12.4 s — and a tail like that is not a slow response, it is a response that has not arrived.
What we changed
For this fleet
- The 8% figure belongs on
coder, not on the gateway. Loops defaulting tochatare retrying against 0.5%. - A prose-call parser has to be shape-agnostic, and recovering the call beats retrying it.
- Keep the token budget at 8192; treat 4096 as the floor. Below that the failure is total.
- The vision model needs a budget guard for open-ended prose, or should not serve it.
- Two memory tools with no disambiguating cue is a bug in the belt, not in the models.
- Consider
coderas the default for tool dispatch: same accuracy, 17× faster, with a known and recoverable failure envelope.
For anyone building one of these
- Check determinism before choosing a sample size. Ours would have been fifty copies of one sentence.
- Read the prose your grader is discarding. The largest error here cost 18 points on one model and inverted the ranking, and was only visible by reading responses scored zero.
- Store responses, not verdicts. Every correction was applied to a finished run for free. The one field we hadn't stored is the one thing that needs another 4.6 hours.
- A rate without an interval is not a result. The measurement that started this was correct. It was reported as "roughly 8%" when the evidence supported "somewhere between 1.5% and 35%".