$ the-wire · showcase
vLLM closes scale-out multimodal handoff hole
By RepoJournal · Filed · About Local LLMs
13 people shipped this
angt
@angt
1 cited
nikwen
@nikwen
1 cited
Xuan-Son Nguyen
@ngxson
1 cited
ZeldaHuang
@ZeldaHuang
1 cited
KernelClint
@KernelClint
1 cited
bakiburakogun
@bakiburakogun
1 cited
jacky.cheng
@yichiche
1 cited
Dayuxiaoshui
@Dayuxiaoshui
1 cited
BBuf
@BBuf
1 cited
zhejiangxiaomai
@zhejiangxiaomai
linitra24
@linitra24
hhhh1252023
@hhhh1252023
Sugar920
@Sugar920
vLLM now validates scale-out multimodal data before engine handoff, closing a trust gap in the generate route.
vLLM closed a security-shaped correctness hole in its scale-out multimodal API [1]. The generate route previously accepted a client-supplied multimodal features object and rebuilt internal tensors, layout choices, placeholder ranges, and cache keys without verifying they still matched the active model or the rendered data. The fix validates that data before handoff, so a client can no longer submit a rendered image and then alter its grid or other layout metadata to mismatch the model's expectations.
For hybrid-model deployments, vLLM extended Kimi-K3 prefill checkpoint optimization [2] to support speculative decoding, partial prefix caching with a prefix match unit smaller than the Mamba block size, and checkpoint blocks restored through KV connectors such as MooncakeStore. The cache manager and FlashKDA worker now share the same checkpoint-validity rules, so a checkpoint block is allocated and hashed only when the worker can write it. Separately, vLLM added a fused-MoE tuned configuration [3] for Qwen3.5-122B-A10B MoE shapes (E=256, N=512) on NVIDIA A100 80GB PCIe at tensor parallel size 2, which previously fell back to the default heuristic.
sglang continued its diffusion-kernel optimization push. It ported the Wan VAE decoder fast paths to the Qwen-Image VAE [4], whose underlying code is the Wan 2.1 VAE under other class names. That brings the lossless causal-conv data-movement path and the quality-gated RMSNorm-to-SiLU fusion to Qwen-Image generation. It also fused the LingBot Video MoE group-limited top-k index selection [5], collapsing a sequence of launch-bound CUDA ops into one kernel without changing the selected expert-id set. On the AMD side, a fix landed for the ROCm VAE Conv2D fast path that was breaking spatial-parallel decode [6].
llama.cpp simplified how its UI assets ship [7]. The project removed the build-time C++ helper and external gzip dependency from the asset embedding process, which eases cross-compilation. "The current method adds many restrictions on the build system, especially when packaging," said contributor angt [7]. Generated C++ stays in templates for readability, and UI assets remain fully embedded. Separately, llama.cpp added tuned Metal fa-vec rows for Q4_0 through Q5_1 on the M2 Max [8], and a new common flag, --log-jsonl, lands for JSONL logging [9].
Action items
- → If you run vLLM scale-out multimodal, review your generate path for reliance on client-returned features data and upgrade to include the validation fix. vllm-project/vllm [immediate]
- → A100 80GB PCIe users serving Qwen3.5-122B-A10B at TP=2 can adopt the new fused-MoE tuned config for better performance. vllm-project/vllm [plan]
- → If you build llama.cpp UI from source, verify the streamlined CMake asset embedding works for your packaging flow. ggml-org/llama.cpp [monitor]
References
- [1] Validate scale-out multimodal data before engine handoff ↗ vllm-project/vllm
- [2] [Kimi K3] Support internal prefix checkpoints with partial prefix caching and spec-decoding ↗ vllm-project/vllm
- [3] [Kernel] Add fused MoE tuned config for E=256,N=512 on NVIDIA A100 80GB PCIe ↗ vllm-project/vllm
- [4] [Diffusion] Port the Wan VAE decoder fast paths to the Qwen-Image VAE ↗ sgl-project/sglang
- [5] [diffusion] fuse LingBot MoE group-limited top-k index selection ↗ sgl-project/sglang
- [6] [AMD] Fix ROCm VAE Conv2D fast path breaking spatial-parallel decode (#34424) ↗ sgl-project/sglang
- [7] ui : embed assets directly with CMake ↗ ggml-org/llama.cpp
- [8] metal : add remaining fa-vec tunings for M2 Max ↗ ggml-org/llama.cpp
- [9] common: add --log-jsonl (#28437) ↗ ggml-org/llama.cpp
