Agent systems frequently reduce to a repeated decision: given the current goal and observation, which tool should be invoked next? The useful output is a discrete choice from a known tool set, optionally accompanied by a confidence score that can gate automatic execution versus human confirmation. Long free-form reasoning may help a developer understand the choice after the fact, but the runtime path only needs the selected tool identifier and enough signal to decide whether to trust it.
A decision-first model is a natural fit for this routing step. It maps the current state description to a structured tool selection without generating an intermediate paragraph that must later be parsed. This post describes how that routing decision is tested on The Decision Bench. It does not claim success rates or latency figures for any particular agent. Those results require a fixed tool inventory, a fixed set of tasks, and a completed measurement pass.
The Tool-Selection Decision
The action space is the set of available tools. Typical examples include search, calculator, code interpreter, database query, email send, calendar read, and a special “finish” or “ask user” action. The exact inventory is fixed before a measurement series begins and is recorded as part of the experimental condition.
The model receives a compact state description: the user goal, a summary of previous steps, and any relevant observations from the last tool call. It returns the identifier of the chosen tool and a confidence score. Secondary scores over the remaining tools are kept when the runtime exposes them; the margin between the top choice and the second choice is often more informative than the absolute value of the top score alone.
Keeping the output structured has two practical consequences. First, the agent runtime can dispatch the tool call without an extra language-model parse step. Second, every selection can be logged with the same fields used throughout this site—state reference, chosen tool, confidence, latency, model version—so that later analysis remains possible.
What the State Description Must Contain
The quality of the routing decision depends heavily on what the model is allowed to see.
A minimal state includes the original user goal and a short history of prior tool names and outcomes. Richer states may include retrieved snippets, error messages from the last tool, or a running scratchpad. I record the exact schema of the state features for each experiment. Changing the schema changes the decision problem; results collected under one schema are not automatically comparable to results collected under another.
Token budgets matter. If the history is truncated, the model may lose the information that would have justified a different tool. The truncation rule is therefore part of the logged condition. Golden trajectories that were correctly routed under the full state are re-checked under the truncated state so that the effect of information loss remains visible.
Logging the Routing Decision
The logging checklist follows the established pattern with a few agent-specific fields:
Model version and runtime
Tool inventory identifier (so the set of legal actions is known)
State reference or hash
Primary tool chosen
Primary confidence
Secondary scores when available
Latency of the routing decision
Whether a confidence threshold forced a fallback (for example, “ask user”)
Eventual outcome of the tool call when it is later known (success, error, or timeout)
One structured record per routing step keeps the analysis flexible. Aggregate success rates are computed later. The per-step record makes it possible to isolate the exact states on which the model selected a tool that subsequently failed, or on which confidence was high yet the choice was wrong.
Latency is measured from the moment the state features are ready until the structured tool decision is available to the agent runtime. Cold-start costs of loading the decision model are kept separate from warm routing times.
Evaluation Dimensions That Matter
Overall accuracy against a human-preferred tool sequence is the most commonly requested number. It is not sufficient by itself.
Precision on high-confidence selections matters more for automation. If the agent only auto-executes when confidence exceeds a threshold, the error rate inside that high-confidence band determines whether the system reduces human oversight or simply injects silent mistakes. I therefore report accuracy conditioned on confidence bands, not only the global average.
The cost of different errors is rarely uniform. Selecting a destructive or irreversible tool (send email, write to production database) when a safer tool was appropriate is usually more expensive than selecting a merely suboptimal search query. Error analysis therefore groups tools by risk tier when such a tiering exists.
Sequences also matter. A model may choose a reasonable tool at each individual step yet still produce an overall trajectory that fails the user’s goal because the order of tools was wrong. Trajectory-level success is recorded alongside step-level accuracy so that both views remain available.
Failure Modes Specific to Tool Routing
Several patterns appear repeatedly in this scene.
High-confidence selection of a tool that cannot solve the current sub-goal.
Oscillation between two tools without progress.
Premature selection of “finish” while required information is still missing.
Failure to select “ask user” when the state is genuinely ambiguous.
Latency spikes that cause the agent to stall or to reuse a stale routing decision.
Out-of-distribution states—novel user goals, tool error messages the model has never seen, or inventories that have been extended since the model was last evaluated—are collected as a separate slice. Performance on that slice is reported independently so that average numbers do not hide brittleness.
Practical Limits of a Decision-Only Router
A structured tool-selection model does not replace planning. It only proposes the next atomic action under the state it is given. Multi-step look-ahead, backtracking, and recovery from tool failures remain the responsibility of the surrounding agent loop or of a higher-level planner.
The model also inherits the limits of its state representation. If critical information is never placed in the state features, the router cannot use it. Expanding the state schema or adding retrieval before the routing decision are separate design choices that require their own measurement.
Finally, success on a fixed tool inventory does not guarantee success when tools are added, removed, or changed in signature. Each change to the action space requires a fresh evaluation pass; the previous numbers become historical.
What Success Looks Like in This Scene
Success is not a single accuracy figure. It is a combination of:
Acceptable step-level accuracy on the defined tool inventory
High-confidence selections that are reliable enough to support automatic execution for low-risk tools
Appropriate use of fallback actions when confidence is low or the state is ambiguous
Latency that stays inside the agent’s real-time or interactive budget
Complete logs that allow every high-confidence routing error to be inspected
If those conditions hold under realistic tasks, the decision-first model becomes a viable routing component. If any of them fail, the failure itself is the useful result. The Decision Bench records both outcomes with the same discipline.
How This Fits the Rest of the Site
Field Tests place decision models inside concrete agent loops. Bench Tests supply the calibration, latency, and accuracy methods that are reapplied to tool-routing traffic. Deploy Notes examine what changes when the same router runs under ONNX, FastAPI, browser, or mobile constraints. Failure Log entries begin from the exact states that produced high-confidence selection of the wrong tool or unexpected latency under sustained agent operation.
The governing rule does not change. Show the state. Show the tool that was chosen. Show the confidence. Show the timing and the inventory conditions. When those elements remain visible, the question “Can a decision model choose the right tool for an agent?” receives an answer grounded in evidence rather than in a polished demonstration.
No comments yet — be the first to share a thought.