A GPU program written in PyTorch is two programs running at once. The CPU walks through your Python, and every tensor operation it meets becomes a kernel launch that goes into a queue. The GPU drains that queue. As long as the CPU stays ahead, the GPU never waits and the CPU's own overhead is hidden.
A host synchronization breaks that arrangement. .item(), bool(tensor), tensor.any() inside an if: each of these needs a value that only exists on the GPU, so the CPU stops and waits until everything queued before it has finished. In a profile they show up as aten::_local_scalar_dense and cudaStreamSynchronize. They read like ordinary Python, which is why model code accumulates them.
I removed two sets of these from model code in vLLM-Omni. One was a clear win on the helper it touched. The other cut synchronizations by more than an order of magnitude and left end-to-end latency where it was. The second is the more instructive of the two.
PersonaPlex is a full-duplex speech model. Its prefill built the input embedding one position at a time, with a read back to the CPU at each position. For a span of n tokens that's n round trips, each one draining the queue.
The fix was to build the token stack for the whole span on the GPU and call the existing embedding module once. The hard constraint was numerical: padding, audio fallbacks and even the order of floating-point additions had to match the old path, because a different summation order is a different number. So the change came with bitwise regression tests and an integration test on the first append, and the full model suite, which covers FP32, BF16 and FP16 on CUDA, passed with 385 tests. On a helper benchmark the vectorized path was much faster for multi-token spans. That benchmark covers the helper only, not full-model serving, and I said so in the PR.
SenseNova-U1 generates an image over 50 denoising steps. Every forward pass asked the GPU which kinds of generation tokens were present, with .any().item(), even though the answer is known before the loop starts. Several other things that never change within a request were rebuilt every step as well: vision grid metadata, the CFG decision, a fixed noise-scale embedding. The change passes the known token types down through the model and TeaCache and prepares the per-request values once. Image features and timestep embeddings still update every step, as they must.
On an H200 (SenseNova-U1.5-8B-MoT, BF16, 1024×1024, 50 steps, CFG 4, batch size 1):
| Metric | Before | After |
|---|---|---|
_local_scalar_dense per 50-step loop | 700 | 3 |
cudaStreamSynchronize per loop | 853 | 60 |
| Denoise median, ms/step | 71.90 | 71.53 |
| End-to-end P50, ms | 3652 | 3643 |
All 16 compared image pairs were pixel-identical, across CFG-off, restricted CFG intervals, image editing and think-enabled cases. And the latency is the same, within run-to-run noise.
A synchronization costs you only when the CPU has something useful it could have done while waiting. Here it didn't. At batch size one, each denoising step is about 72 ms of GPU work, and the CPU's job per step, launching kernels, should be a small fraction of that. With or without the syncs, the CPU finishes its part early and ends up waiting on the GPU anyway. The syncs moved the waiting around without adding to it.
You can put a rough number on it. The change removed about 16 synchronizations per step and the step median dropped by 0.37 ms. Taken at face value, though it's inside the noise, that's roughly 20 microseconds per sync, in the range you'd expect for a round trip to the GPU. Over a 3.6-second image, 9 ms is a quarter of a percent. There was never much to win at this operating point, and I reported it that way rather than calling it a speedup.
The same change starts to pay off when the balance shifts toward the CPU. Three cases I'd expect it in, none of which I've measured here:
CUDA graphs. A captured graph replays a fixed sequence of kernels with almost no CPU involvement, and a host synchronization can't be captured at all. A loop full of .item() calls can't become a graph, so they have to go before graphs are even an option.
Small, fast steps. On lighter models or faster hardware, a step's GPU time shrinks while the CPU's launch work stays about the same. Once the CPU is the slower side, every sync adds directly to latency.
Asynchronous scheduling. Serving engines overlap the next step's scheduling with the current step's execution. A sync in model code forces the two back into lockstep.
Count before guessing. A profile filtered for _local_scalar_dense and cudaStreamSynchronize tells you how many there are per step. torch.cuda.set_sync_debug_mode("warn") reports synchronizing calls as they happen, and vLLM's test suite has a VLLM_GPU_SYNC_CHECK switch for the same purpose. Then look for three patterns: .item() on a hot path, tensor truthiness in a conditional, and per-step recomputation of something that is fixed for the whole request.