The next improvement in AI agents may come from changing how they use tools, not only from making the underlying models larger. A new study from PricewaterhouseCoopers researchers finds that many current language models can call external functions as reliably—or more reliably—when those functions are presented as typed Python code instead of the structured JSON format common in agent systems.
The paper, submitted to arXiv on August 6, compares programmatic tool calling with native JSON tool calling across 14 OpenAI and Anthropic models. The researchers tested both interfaces on a 309-item subset of version four of the Berkeley Function Calling Leaderboard, or BFCL, plus smaller experiments designed to isolate long call chains, parallel requests, and crowded contexts.
The result is not a universal victory for code. Programmatic tool calling matched or exceeded JSON accuracy in 11 of the 14 models on the main evaluation, but the aggregate score across every model was slightly lower for the code interface. Three older OpenAI models struggled with multiline scripts, while newer models generally handled them better. That split makes the study less a verdict on one format than evidence that model capability and interface design interact.
Two ways to give an agent tools
In a conventional tool-calling setup, a model selects a function and returns a JSON object containing the required arguments. If the next step depends on the first result, the agent often needs another model turn. For parallel work, the response must enumerate each call in the required structure.
The programmatic alternative in the new study exposed the same functions as typed Python stubs. The model wrote a script that could call several stubs, perform intermediate calculations, or fan out requests in one execution. No real external service was contacted during the benchmark; the stubs returned their arguments so the evaluation could check whether each function and parameter was correct.
On the main BFCL subset, two GPT-5.6 variants improved by 10.6 percentage points over their own JSON baselines. In a separate 32-item parallelism test, programmatic calling matched or beat JSON in 13 of 14 models. The study also reports that code maintained complete enumeration at 100 calls in targeted tests with Claude Sonnet 5, while the JSON approach began dropping calls above a model-specific threshold.
TENS Analysis: the interface is part of the intelligence
The practical finding is that an agent’s measured ability is partly a property of its surrounding harness. The same model, task, and functions can produce different outcomes when the interface changes how work is expressed. A benchmark score therefore describes a model-and-system combination, not an isolated intelligence floating above its software environment.
Code appears especially useful when repetition or composition can be expressed compactly. A loop can represent dozens of calls without requiring the model to spell out dozens of separate objects. A script can also preserve intermediate values and make transformations explicit. Those are structural advantages, but only when the model reliably writes executable code and the runtime is safely constrained.
The study’s token results reinforce that workload shape matters. Programmatic calling carried a fixed prompt overhead and used 1.5 times as many input tokens in the chaining experiment. The balance changed at high fan-out: the researchers calculated a crossover near 26 calls, after which enumerating JSON objects consumed more tokens than expressing the work in code.
That creates a more useful design question than asking which format is best. Developers should ask where a workflow sits on three axes: how many calls it makes, how strongly later steps depend on earlier results, and how dependable the selected model is at code generation. A short, tightly controlled call may favor JSON. A broad fan-out or transformation pipeline may favor code.
What the benchmark cannot prove
The Berkeley team that created BFCL describes function calling as a core capability for agentic systems and designed the benchmark to cover serial calls, parallel calls, abstention, and multi-step behavior. Its published research also warns that long-horizon reasoning and dynamic decision-making remain open challenges. The new paper narrows the question to the interface, but it does not eliminate those wider difficulties.
Its strongest limitation is also its cleanest experimental control: the functions were echo-return stubs rather than live services. That makes argument scoring deterministic, but it does not test authentication failures, rate limits, changing data, unsafe side effects, or a tool returning an unexpected value. The ablation sets contained only 31 to 52 items per condition, producing wide confidence intervals. Individual model differences should therefore be treated as directional.
There is also a security tradeoff. JSON restricts a model to a predefined call schema. Executable code can compose work more naturally, but it expands what the runtime must isolate, inspect, and limit. Reliability gains in a sandbox do not by themselves justify broader execution privileges in production.
The study’s most durable conclusion is not that JSON should disappear. It is that agent builders should evaluate the tool interface as rigorously as the model. The future of capable agents may depend as much on choosing the right execution grammar for each workload as on choosing the highest-ranked model.
Sources: The Bitter Lesson of Tool Calling research paper; University of California, Berkeley’s Berkeley Function Calling Leaderboard research and official benchmark documentation; CodeAct research published at the International Conference on Machine Learning.
Featured image: Programming and network diagnostics displayed on a computer screen in Nairobi. Photo: Lewis Ngugi via Wikimedia Commons. CC0 1.0 public domain dedication. Center-cropped from the original to 16:9 and resized to 1,920 × 1,080 pixels; no generative or substantive alteration.
