LLM inference & RL infrastructure
I lead teacher serving for on-policy distillation on vLLM's RL roadmap and work across vLLM-Omni's multimodal runtime.
I work on LLM inference and the infrastructure around it. Most of my time goes into vLLM, the open-source inference engine, and vLLM-Omni, its multimodal extension. The thread running through it is making serving work well for what comes after it: training loops, agents, and retrieval.
Before that I was at Meta, working on agent systems and how to run them efficiently.
Inference systems, and the research questions they raise, are what I spend my time on. My background is in computer science at Carnegie Mellon University, and I did my undergraduate studies at Xi'an Jiaotong University.
I build in the vLLM ecosystem, across the engine core, its RL-facing serving paths, pooling, and the vLLM-Omni multimodal runtime.
Author of the teacher-serving roadmap for on-policy distillation and owner of its milestones: prefix-cache reuse for scoring, score reuse and delivery to trainers, tensor-parallel candidate scoring, and multi-turn validation.
View roadmap →Responsible for DiT KV-cache adoption, diffusion batching and resource accounting, batching and streaming validation, and shape-aware CUDA graph capture for diffusion encoders.
View roadmap →In on-policy distillation a teacher scores every token the student writes, and scoring requests used to skip the prefix cache and recompute the shared context. I designed bounded prefix reuse across the KV cache manager and scheduler, including chunked prefill and preemption: 3.7–3.8× faster teacher scoring on long shared contexts with a 32B teacher at TP=4.
Design and implementation →Brought late chunking into vLLM's pooling path: one pass over a long document, chunk vectors pooled on the GPU before the embedding head, so each chunk carries the context around it. Designed through an RFC, built for offline use and then extended to the online /v1/embeddings API.
Runtime work for image-generation and speech models: TeaCache correctness and an order-of-magnitude cut in host synchronizations in the SenseNova-U1 denoising loop, KV caches kept in flash layout, vectorized PersonaPlex prefill, and explicit turn-based serving for full-duplex models.
Work in vLLM-Omni →Why scoring requests skipped the prefix cache, why the hit has to stop exactly at the first scored position, and a one-line cost model whose bound sits just above the measured 3.8×.
Why the mean has to come before the embedding head, why the sums need FP32, and what attention direction does to the idea.
Cutting scalar reads from 700 to 3 per image, why the latency didn't move, and the cases where it would.
From floating-point reduction order to a different tool call, and why the history an agent carries keeps the runs apart.
The prefill and decode arithmetic behind agent workloads, the quadratic cost of re-sent context, and why concurrency changes the answer.
The false-alarm arithmetic that forces a verification pipeline, and why deduplication is a logic problem.
Where per-record overhead hides, what JSON costs at volume, and how Compact Thrift encodes a record byte by byte.
I also do research on questions that come out of the systems work. Selected work, under review; titles are held back until the review period ends.