$ the-wire · showcase
vLLM bounds validation errors, AutoRound FP8 arrives
By RepoJournal · Filed · About Local LLMs
15 people shipped this
BBuf
@BBuf
3 cited
arthw
@arthw
1 cited
nikwen
@nikwen
1 cited
jadenmach2
@jadenmach2
1 cited
lzhan011
@lzhan011
1 cited
Zhenzhong1
@Zhenzhong1
1 cited
NickLucche
@NickLucche
1 cited
Fridge003
@Fridge003
1 cited
marcelpetrick
@marcelpetrick
dhiltgen
@dhiltgen
王金旭
@wjinxu
ynankani
@ynankani
rasmith
@rasmith
arpera
@arpera
caihuali95
@caihuali95
vLLM shipped a security fix bounding validation-error response bodies and added AutoRound block-wise FP8 support, while llama.cpp and sglang pushed kernel fusions for Blackwell and ROCm.
vLLM '[Bugfix][Security] Bound the validation-error response body' [1] prevents unbounded error responses, a potential DoS vector. vLLM also added AutoRound format block-wise FP8 support [2], enabling quantization-aware inference for that format. For GPU memory, vLLM introduced `max_num_queued_reqs` and `max_num_queued_tokens` [3] to cap the previously unbounded request queue, letting operators reject work early and signal load balancers. In llama.cpp, the SYCL backend now supports both level zero and SYCL APIs for getting free GPU memory [4], overcoming limitations that left some Intel GPUs like Arc770 unsupported. The CUDA backend extended MoE fusion to specdec 2-8 tokens , and a new ROCm TOP_K path for rows over 1024 elements [5] targets DeepSeek-V4 long-context decoding. On macOS, new fa-vec tunings for M1 [6] arrived after a 48-hour sweep. sglang fused Qwen-Image residual norm with NVFP4 quantization [7] and FLUX.2 gated residual normalization [8] on Blackwell, plus an FP8 QKV projection fusion [9]. As the PR notes, the new sglang kernels let "one Blackwell JIT kernel now performs" multiple operations [7]. sglang also bumped sgl-deep-gemm to 0.1.7 [10] for nvfp4 megamoe support.
Action items
- → Upgrade vLLM to include the validation-error response body bound before exposing to untrusted input vllm-project/vllm [immediate]
- → If you run Intel GPUs, test llama.cpp with GGML_SYCL_GET_MEM_API for accurate free memory ggml-org/llama.cpp [plan]
- → Evaluate the new queue size limits in vLLM to protect TTFT on saturated serving vllm-project/vllm [plan]
References
- [1] [Bugfix][Security] Bound the validation-error response body (#54684) ↗ vllm-project/vllm
- [2] [AutoRound] Support AutoRound Format Block-Wise FP8 in vLLM ↗ vllm-project/vllm
- [3] [Core] Add `max_num_queued_reqs` and `max_num_queued_tokens` for queue size management ↗ vllm-project/vllm
- [4] [SYCL] Enhance to get the free memory of Intel GPU ↗ ggml-org/llama.cpp
- [5] ROCm: add radix TOP_K for long rows ↗ ggml-org/llama.cpp
- [6] metal : add fa-vec tunings for M1 ↗ ggml-org/llama.cpp
- [7] [Diffusion] Fuse Qwen-Image residual norm and NVFP4 quantization ↗ sgl-project/sglang
- [8] [Diffusion] Fuse FLUX.2 gated residual normalization on Blackwell ↗ sgl-project/sglang
- [9] [Diffusion] Fuse Qwen-Image FP8 QKV projection and Blackwell epilogue ↗ sgl-project/sglang
- [10] Bump sgl-deep-gemm to 0.1.7 ↗ sgl-project/sglang
