The Wire · Showcase
VLLM'S P2P WEIGHT SYNC CUTS RL MOE TRANSFER COSTS
By RepoJournal · Filed · About Local LLMs
13 people shipped this
gaugarg-nv
@gaugarg-nv
1 cited
am17an
@am17an
1 cited
wyettzeng
@wyettzeng
1 cited
hao-aaron
@hao-aaron
1 cited
KurodaKanbei
@KurodaKanbei
1 cited
KernelClint
@KernelClint
1 cited
BBuf
@BBuf
1 cited
alisonshao
@alisonshao
1 cited
allozaur
@allozaur
cabelo
@cabelo
fuzzifikation
@fuzzifikation
Xia-Weiwen
@Xia-Weiwen
mickqian
@mickqian
Forget all-gathering experts during RL rollout: vLLM's new sharded_rdt backend lets each worker pull only the slice it needs, turning a total_bytes transfer into a total_bytes divided by num_workers one.
vLLM's new sharded_rdt backend [1] is the RL efficiency story of the day, letting each inference worker pull only the slice it consumes under tensor and expert parallelism via NIXL/Ray Direct Transport. That joins the routed expert loading for gpt-oss [2], which lets RL engines like verl sync experts one by one instead of all-gathering. On the inference side, SGLang fused LongCat-Image's QKNorm and interleaved RoPE into one kernel [3] and reports H200 speedups, while vLLM fixed KV cache layout for Dots3 NOTE Omni [4]. If you run llama.cpp with tensor parallel, the meta tensor split state propagation fix [5] closes a batch of test-llama-archs failures, and the DeepSeek V4 rollback fix [6] is worth pulling for multi-seq stability. Not to be missed: vLLM now rejects oversized media before fully downloading it [7], which plugs a memory exhaustion hole in batch speech paths, and SGLang drops empty assistant turns for mistral_common tokenizers [8] to close an OpenAI-compat gap.
One email a day. Unsubscribe in one click.
Keep up with Local LLMs in about 3 minutes a day: what actually shipped — the commits, pull requests, releases, and security advisories that matter.
One email a day. Unsubscribe in one click. Read a past issue →
Action items
- → Upgrade vLLM to a build including sharded_rdt if you run RL with MoE models vllm-project/vllm [plan]
- → Review your batch speech endpoints for oversized media downloads and patch vllm-project/vllm [immediate]
- → Pull llama.cpp with the meta tensor split fix before running tensor parallel on Qwen3Next ggml-org/llama.cpp [plan]
References
- [1] [RL] P2P RDT weight sync ↗ vllm-project/vllm
- [2] Add routed expert loading for gpt-oss ↗ vllm-project/vllm
- [3] [diffusion] Fuse LongCat-Image QKNorm and interleaved RoPE ↗ sgl-project/sglang
- [4] [Model] Fix KV cache layout and optimize Dots3 NOTE Omni encoders ↗ vllm-project/vllm
- [5] [Tensor parallel] Fix meta tensor split state propagation ↗ ggml-org/llama.cpp
- [6] DeepseekV4: fix rollback with multi-seq ↗ ggml-org/llama.cpp
- [7] Reject oversized media before fully downloading it ↗ vllm-project/vllm
- [8] [OpenAI] Drop empty assistant turns for mistral_common tokenizers ↗ sgl-project/sglang
