Tools or code: an LLM case study¶
The agent tools rest on a claim: a language model that builds a simulation through them gets a model that is valid, runs and gives the right answer more often than the same model writing SimPy code for the same request. This page is where that claim is checked. It is a case study on three models from two vendors and twelve tasks, not a ranking of models and not evidence about tasks unlike these. Whatever comes out is reported, including a result that goes against the tools.
What is measured¶
Each task is a request as a practitioner would type it, with no block name and no API in it, and
with an answer known from theory: queues, a job shop with rework, priorities, batching, a
resource on a rhythm, a loss system, one stock-and-flow model, one population of agents and one
tank with rates. A task's file under benchmarks/llm_study/tasks/ holds the request, the
derivation of the answer and of its tolerance, and a reference solution for each arm; the tests
run both and check that they land inside the tolerance.
The arms get the same request and the same number of model replies:
- Naga tools, block by block. The first round of the study: the model adds blocks and records one call at a time, validates, runs and reads the results through the tools, and says the answer. An edit answers with the problems it introduced, each with a code and a hint, and the model may correct itself. A model's own Python never runs. These attempts are kept as they were made; the tools have changed since, so this arm is not run again.
- Naga tools, whole model. The tools as reshaped after that round:
get_model_formatdescribes the model file once,write_modeltakes the whole file in one call and answers with its problems, then the model runs it. The local loop offers the compact tool list, Claude Codenaga mcp --tools compact. - SimPy code. The model writes a Python script that uses SimPy. The script is run, and when it fails the model is given what it printed and may send a corrected one.
Three outcomes are recorded per attempt, each implying the one before:
| Outcome | Naga tools | SimPy code |
|---|---|---|
| Valid | the model file passes validation and is not empty | the script parses and imports SimPy |
| Runnable | it runs to the end within the time limit | it exits without an error within the time limit |
| Correct | the answer given is within the tolerance, and the model was run through the tools | the answer printed is within the tolerance |
A statistical tolerance is five standard deviations of the SimPy reference's own estimate over twenty seeds, at the run length the request asks for; a task with nothing random in it has a tolerance derived from its edge effects or its integration step.
Two comparisons that do not compare with each other¶
- Local models run inside the engine's own tool loop (what
naga assistruns), served by Ollama on a laptop, with a compact set of tools. This measures a model in one loop. - Claude Code runs headless in an empty folder: for the tools arm with the
naga mcpserver and no other tool, for the code arm with a shell limited to running its script. This measures a whole coding agent, with its own planning and its own retries, so its numbers say nothing about how it ranks against the local models.
Results¶
The tasks¶
| Task | What is asked | Answer | Accepted within |
|---|---|---|---|
mm1 |
One clerk, one queue (M/M/1): the mean time a customer spends in the shop | 12 | 2.1 |
mmc |
Three agents, one queue (M/M/3): the mean wait before a call is answered | 3.407 | 0.77 |
md1 |
A machine with a fixed cycle time (M/D/1): the mean wait before a job starts | 2.25 | 0.3 |
tandem |
Two machines in a row (tandem M/M/1 queues): the mean time a part spends in the line | 19.17 | 2.5 |
rework |
Machining and inspection with rework (a Jackson network): the mean time a job is in the shop | 26.67 | 3.1 |
priority |
Urgent jobs go first (a non-preemptive priority M/M/1 queue): the mean time of an urgent job | 7.5 | 0.53 |
batch |
Trays of five through a tunnel oven (batching): the mean time of an item | 7 | 0.12 |
carpark |
A car park with ten spaces and no queue (an Erlang loss system): the share turned away | 9.954 | 0.98 |
shift |
A robot that works 65 minutes and charges 35 (a capacity schedule): the mean time of a parcel | 13 | 0.25 |
cooling |
A cup of coffee cooling (system dynamics, one stock): its temperature after 15 minutes | 35.62 | 0.5 |
repair |
Twenty machines that fail and are repaired (agents with two states): how many run on average | 18.18 | 0.27 |
tank |
A tank filled by a pump on a timer and drawn from steadily (rates): the share of time it is empty | 16.67 | 0.5 |
Local models, in the engine's own tool loop¶
Attempts made from 2026-10-13 to 2026-10-14: 12 tasks, 1 to 3 attempts per task and arm, at most 25 model replies per attempt. Naga 0.6.0, SimPy 4.1.1, Python 3.12.4.
gemma4:12b was stopped after 42 of the 72 planned attempts (3 per task and arm); its cells count what was made.
hf.co/unsloth/gemma-4-E4B-it-GGUF:Q4_K_M was stopped after 84 of the 108 planned attempts (3 per task and arm); its cells count what was made.
| Model | Arm | Attempts | Valid | Runnable | Correct | Median replies | Median tool calls | Median input tokens (k) | Median output tokens (k) | Median cost (USD) | Median minutes |
|---|---|---|---|---|---|---|---|---|---|---|---|
gemma4:12b |
Naga tools, block by block | 21 | 3/21 (14%) | 3/21 (14%) | 1/21 (5%) | 25 | 24 | 172 | 0.579 | 3.68 | |
gemma4:12b |
SimPy code | 21 | 20/21 (95%) | 20/21 (95%) | 13/21 (62%) | 1 | 1 | 0.223 | 1.31 | 7.08 | |
hf.co/unsloth/gemma-4-E4B-it-GGUF:Q4_K_M |
Naga tools, block by block | 36 | 1/36 (3%) | 0/36 (0%) | 0/36 (0%) | 25 | 24 | 234 | 2.27 | 6.91 | |
hf.co/unsloth/gemma-4-E4B-it-GGUF:Q4_K_M |
Naga tools, whole model | 12 | 7/12 (58%) | 6/12 (50%) | 3/12 (25%) | 10 | 9 | 93.9 | 4.07 | 12.1 | |
hf.co/unsloth/gemma-4-E4B-it-GGUF:Q4_K_M |
SimPy code | 36 | 35/36 (97%) | 34/36 (94%) | 18/36 (50%) | 1 | 1 | 0.23 | 1.76 | 3.86 |
Correct attempts per task:
| Task | gemma4:12b, Naga tools, block by block |
gemma4:12b, SimPy code |
hf.co/unsloth/gemma-4-E4B-it-GGUF:Q4_K_M, Naga tools, block by block |
hf.co/unsloth/gemma-4-E4B-it-GGUF:Q4_K_M, Naga tools, whole model |
hf.co/unsloth/gemma-4-E4B-it-GGUF:Q4_K_M, SimPy code |
|---|---|---|---|---|---|
mm1: One clerk, one queue (M/M/1): the mean time a customer spends in the shop |
0/2 | 2/2 | 0/3 | 1/1 | 3/3 |
mmc: Three agents, one queue (M/M/3): the mean wait before a call is answered |
0/2 | 0/2 | 0/3 | 0/1 | 2/3 |
md1: A machine with a fixed cycle time (M/D/1): the mean wait before a job starts |
0/2 | 1/2 | 0/3 | 1/1 | 1/3 |
tandem: Two machines in a row (tandem M/M/1 queues): the mean time a part spends in the line |
1/2 | 2/2 | 0/3 | 0/1 | 2/3 |
rework: Machining and inspection with rework (a Jackson network): the mean time a job is in the shop |
0/2 | 2/2 | 0/3 | 0/1 | 2/3 |
priority: Urgent jobs go first (a non-preemptive priority M/M/1 queue): the mean time of an urgent job |
0/2 | 2/2 | 0/3 | 0/1 | 0/3 |
batch: Trays of five through a tunnel oven (batching): the mean time of an item |
0/2 | 1/2 | 0/3 | 0/1 | 1/3 |
carpark: A car park with ten spaces and no queue (an Erlang loss system): the share turned away |
0/2 | 1/2 | 0/3 | 0/1 | 1/3 |
shift: A robot that works 65 minutes and charges 35 (a capacity schedule): the mean time of a parcel |
0/2 | 0/2 | 0/3 | 0/1 | 1/3 |
cooling: A cup of coffee cooling (system dynamics, one stock): its temperature after 15 minutes |
0/1 | 1/1 | 0/3 | 1/1 | 1/3 |
repair: Twenty machines that fail and are repaired (agents with two states): how many run on average |
0/1 | 1/1 | 0/3 | 0/1 | 2/3 |
tank: A tank filled by a pump on a timer and drawn from steadily (rates): the share of time it is empty |
0/1 | 0/1 | 0/3 | 0/1 | 2/3 |
Claude Code, the whole coding agent¶
Attempts made from 2026-10-14 to 2026-10-14: 12 tasks, 3 attempts per task and arm, at most 50 agent turns per attempt. Naga 0.6.1, SimPy 4.1.1, Python 3.12.4. Model ids as reported by Claude Code: claude-haiku-4-5-20251001.
| Model | Arm | Attempts | Valid | Runnable | Correct | Median replies | Median tool calls | Median input tokens (k) | Median output tokens (k) | Median cost (USD) | Median minutes |
|---|---|---|---|---|---|---|---|---|---|---|---|
haiku |
Naga tools, block by block | 36 | 35/36 (97%) | 35/36 (97%) | 27/36 (75%) | 17.5 | 16.5 | 429 | 3.52 | 0.098 | 0.957 |
haiku |
Naga tools, whole model | 36 | 36/36 (100%) | 36/36 (100%) | 35/36 (97%) | 6 | 5 | 83 | 4.49 | 0.0582 | 0.8 |
haiku |
SimPy code | 36 | 36/36 (100%) | 35/36 (97%) | 32/36 (89%) | 3 | 2 | 31.4 | 3.92 | 0.0358 | 0.562 |
Correct attempts per task:
| Task | haiku, Naga tools, block by block |
haiku, Naga tools, whole model |
haiku, SimPy code |
|---|---|---|---|
mm1: One clerk, one queue (M/M/1): the mean time a customer spends in the shop |
3/3 | 3/3 | 3/3 |
mmc: Three agents, one queue (M/M/3): the mean wait before a call is answered |
2/3 | 3/3 | 3/3 |
md1: A machine with a fixed cycle time (M/D/1): the mean wait before a job starts |
3/3 | 3/3 | 3/3 |
tandem: Two machines in a row (tandem M/M/1 queues): the mean time a part spends in the line |
3/3 | 3/3 | 3/3 |
rework: Machining and inspection with rework (a Jackson network): the mean time a job is in the shop |
3/3 | 3/3 | 2/3 |
priority: Urgent jobs go first (a non-preemptive priority M/M/1 queue): the mean time of an urgent job |
1/3 | 3/3 | 2/3 |
batch: Trays of five through a tunnel oven (batching): the mean time of an item |
2/3 | 3/3 | 3/3 |
carpark: A car park with ten spaces and no queue (an Erlang loss system): the share turned away |
1/3 | 3/3 | 3/3 |
shift: A robot that works 65 minutes and charges 35 (a capacity schedule): the mean time of a parcel |
3/3 | 3/3 | 3/3 |
cooling: A cup of coffee cooling (system dynamics, one stock): its temperature after 15 minutes |
3/3 | 3/3 | 3/3 |
repair: Twenty machines that fail and are repaired (agents with two states): how many run on average |
2/3 | 3/3 | 3/3 |
tank: A tank filled by a pump on a timer and drawn from steadily (rates): the share of time it is empty |
1/3 | 2/3 | 1/3 |
Why the larger local model was stopped short¶
Three attempts per task and arm were planned for each local model. The smaller model's run was completed. The larger model's run was stopped once it had attempted every task in both arms, about three fifths of the way through, because it was already showing the smaller model's pattern: its tools arm almost never ended with a correct model while its code arm was right more often than not, and the remaining rounds, at several minutes per attempt on a laptop, could not have reversed that. Its tables count only the attempts that were made.
What came out¶
The study ran twice. The first time the tools built a model block by block, and the claim did not hold: with both local models and with Claude Code, writing SimPy code gave a correct answer more often, in fewer turns and for less. The small local models rarely finished a model at all: they placed blocks with wrong parameters, left them unconnected and ran out of replies. The cost came from the number of round trips, each of which reads the whole conversation again, not from what the model wrote.
So the tools changed shape. A model now writes the whole model file in one call, with
write_model, after one call to get_model_format that describes the file; the answer is "ok"
or the problems with their places in the file, and a run and a table of results follow. The
second run, the rows "whole model" above, measures that: Claude Code built a correct model
through the tools on more tasks than it wrote correct SimPy, in a few calls, reading far less
than before though still more than the code arm, and the smaller local model, which had built no
correct model block by block, now built some. Three of the whole-model Claude attempts (the
first round of tank and carpark) were made again after fixes that round found; the earlier
attempts are not counted.
What the tools give that the code does not, a model that is validated, can be opened in an editor, diffed and run again by someone else, or followed as a twin, is not measured here. The fixes the two runs found (an argument sent as text, a required argument put inside another, a distribution's unknown parameter, a condition reading a capacity, a timer profile's period) are in the changelog.
What this does not show¶
- Twelve tasks with textbook answers are not a practitioner's work. A request with a known answer is also one a model may have seen solved.
- A script can print a formula's value without simulating anything, and so can a reply. The tools arm must at least have run its model; the code arm must at least import SimPy. Neither check proves the number came from the run.
- The tools arm is given a system prompt that describes the workflow, the code arm a short one: each arm gets the help its interface comes with, which is part of what is compared.
- A local model's result depends on the server's settings, the context length above all: a conversation that no longer fits is cut silently.
Run it again¶
Every attempt is one file under benchmarks/llm_study/attempts/, written when it ends. A run
can be stopped at any time; started again, it skips what is on file.
$ python benchmarks/llm_study/study.py local --models gemma4:12b --repeats 3
$ python benchmarks/llm_study/study.py claude --model haiku --repeats 3
$ python benchmarks/llm_study/study.py report
--tasks limits a run to some tasks and --arms to one arm (whole or simpy; the block-by-block arm is a record and is not run again). The Claude Code run takes
--max-turns, and starts the agent with only the permissions the arm needs: the tool server's
tools for one, reading and writing in its folder and running its own script for the other.