Once the data-quality agent was running, cost became the question. Each run made several model calls, and they added up. The tempting move is to grab the obvious knob, a smaller model or a shorter prompt, and see what happens. I wanted to know where the time went first, because the right knob depends on it.
An agent run isn't one long inference. It's a loop: read some data, call the model, run a tool, check a result, call the model again. The first cut is by stage:
end-to-end task time
= data reading
+ model execution
+ rule verification and other tool work
+ waiting in betweenThat alone settled a lot. Time spent reading data or verifying rules isn't something inference work can touch, so I fixed those on the tool side and kept the inference work for the part that was actually inference.
Inside model execution, the split that matters is context processing (prefill) versus generation (decode), because they are limited by different hardware resources.
Decode is limited by memory bandwidth. To produce one token, the model reads all of its weights. Take an 8B-parameter model in BF16, about 16 GB of weights, on a GPU with around 2 TB/s of bandwidth: that's roughly 8 ms per generated token at batch size one, no matter how small the computation is.
Prefill is limited by compute. The model processes all the input tokens together, at roughly two floating-point operations per parameter per token. For the same model, that's about 16 GFLOP per token, and at a realistic 150 TFLOP/s on an A100-class GPU it comes to around 0.1 ms per input token.
So on this rough model, an input token is something like 80 times cheaper than an output token. Whether context or generation dominates depends on the ratio of input to output, and on where that ratio sits relative to 80 to 1. These numbers are illustrative, not measurements of the internal setup, but the shape of the argument holds across models and hardware.
Here's what makes agent workloads unusual. Every turn sends the instructions again, the schema description again, and the results of earlier steps. If turn t carries a base context B plus h tokens of history per earlier step, the total input over T turns is
total prefill tokens = T * B + h * T (T - 1) / 2The history term grows with the square of the number of turns. With B = 3000, h = 500 and ten turns, that's 52,500 input tokens, most of them copies of tokens the model has already processed. Prefix caching exists for precisely this: it turns the repeated part into a lookup and leaves only the new tokens to compute.
On the arithmetic above, ten turns of 200 output tokens cost about 16 seconds of decode at batch size one, more than the five or so seconds of prefill. But production agents rarely run at batch size one. When many agent runs share a GPU, each decode step reads the weights once for the whole batch, so the bandwidth cost per token falls with batch size. Prefill cost per token doesn't. The more concurrent runs you have, the more the repeated context becomes the thing you're paying for.
Concurrency also helps with the other cost that per-call latency hides: the gaps. Model calls alternate with tool calls, so while one run waits on a tool, the GPU sits idle unless other runs fill it. Measuring only model latency misses this entirely.
A change counted only if it held up on all three. End-to-end task time, not per-call time. Resource use, because a faster run that needs twice the hardware isn't cheaper. And behavior: the rules the agent produced, and the path it took to get there, had to stay the same. An agent that gets faster but starts proposing different checks has changed, not improved.
The last one is easy to skip. Batching more runs together, which the arithmetic above pushes you toward, changes how the model is served, and small changes in serving can change what an agent decides to do. That's the question my later research on batching and agents looks at directly.
The actual savings depended on internal models and workloads, so I won't quote them. The method carries over, and it's the same reasoning behind my current work on the teacher side of on-policy distillation in vLLM, where repeated context is again a large share of the cost.