Inference servers batch unrelated requests together because it's far cheaper than running them one at a time. It's also known that this can change the text a model produces. For a chatbot, a slightly different sentence rarely matters. For an agent, text becomes an operation: it reads a record, sends a query, submits an answer. So the question is narrow. If the only thing that changes is who else shares the batch, can an agent end up doing something different?
Requests in a batch don't read each other's data. What they share is the shape of the computation. A matrix multiply over a batch of one and a batch of four can pick different kernels, different tile sizes, or a different split of the reduction dimension across thread blocks. Each of those adds the same products in a different order, and floating-point addition isn't associative. So the numbers your request gets can differ in the last bits depending on who else is in the batch.
Those bits are usually harmless. In BF16, numbers between 16 and 32 are spaced 0.125 apart, so a value around 20 that passes through BF16 can move by several hundredths from rounding alone. Greedy decoding picks the top token, and most of the time the top token wins by a wide margin. But a long agent run makes many decisions, and some of them are close: two tool arguments, or two ways to phrase a search, within a tenth of a logit of each other. At one of those, a difference in the last bits flips the argmax.
In plain generation, a flipped token changes some text. In an agent, it can change a tool call, and a tool call changes the world the agent sees next. One branch searches for one thing and the other for something slightly different, and they get back different results. From then on the two runs are conditioned on different observations, and every later decision is made in a different context. Nothing in the loop pulls them back together.
I spent a stretch of time measuring this on tool-use tasks with real environments. The forks are not rare curiosities. They reproduce, they show up across different serving setups, and some of them change whether the task ends up done correctly.
The part I find most useful is about memory. Suppose two branches end in exactly the same environment state: same files, same database rows. It's tempting to say they've converged. They haven't, because each agent carries its own history of what it read and did, and that history is the context for every next decision. Two agents in identical environments with different histories can still act differently.
That breaks a common shortcut. Checking that two runs ended in the same environment isn't the same as checking that they behaved the same, because the context window is state too, and it doesn't live in the environment.
The fix in principle is to make reduction order independent of batch composition, so a request computes the same numbers whatever it shares a batch with. Deterministic or batch-invariant serving modes do this, and they remove much of the effect. They aren't one switch that means the same thing everywhere, though: how complete the guarantee is depends on which kernels and code paths a model actually goes through. And it isn't free, because fixing the reduction order rules out the faster kernels that the batch shape would otherwise pick.
If you evaluate or train agents on shared inference infrastructure, the serving setup is part of your experiment. To reproduce an agent result, record the serving configuration, the executed actions and the retained history together. And when a measured difference between two agents is small, ask whether it's larger than what changing the batch alone would do.