$ the-wire · showcase
vLLM ships five security fixes; SYCL gets the optimization push
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
vLLM closed three distinct memory and input-handling holes in one batch while llama.cpp and sglang spent the day on backend kernels and diffusion tail latency.
[Security] Widen int32 pointer arithmetic to int64 in CUDA kernels (#53131) vllm-project/vllm
Pointer arithmetic in the CUDA kernels now runs in int64 instead of int32, which the commit titles under a security tag. Anything driving very large allocations or offsets through those kernels was previously open to overflow; rebuild vLLM from a commit that includes this change before running untrusted or large-context workloads.
[Security] Share media byte budgets across batched requests (#68249) vllm-project/vllm
Media byte budgets are now shared across batched requests rather than accounted per request, so a batch of individually-within-limit multimode inputs can no longer exceed the intended total. This changes how multimodal batched serving behaves under load: expect different admission behavior for large image or audio batches.
fix(security): enforce prompt-count bound on pooling and scoring routes (#53473) vllm-project/vllm
The pooling and scoring routes now enforce a bound on prompt count, closing the same class of unbounded-input path already guarded on generation endpoints. Deployments exposing /pooling or scoring to untrusted callers should pin a build containing the fix.
fix(security): use segment-boundary-aware root_path stripping in auth middleware (#52794) vllm-project/vllm
Auth middleware strips root_path on segment boundaries instead of by prefix, so a path crafted to look like it sits under the root no longer passes through the prefix check. This is the one with the most direct exposure for anyone running vLLM behind a reverse proxy with a non-empty root_path.
sycl: add IQ4_NL and IQ4_XS weight reorder ggml-org/llama.cpp
Weight reordering for IQ4_NL and IQ4_XS on the SYCL backend cuts MUL_MAT time on an Intel Arc B70 from 315.9 to 70.1 microseconds at n=1 and from 944.2 to 87.0 at n=3, a speedup that grows with batch width. If you serve IQ4 weights on Arc hardware, this is the change to benchmark yourself.
[Fix] diffusion: Reject a LoRA adapter that matches no LoRA layer sgl-project/sglang
set_lora used to log "LoRA adapter ... did not match any LoRA layer" as a warning and return OK, silently serving the base model; this rejects the adapter instead. The scale is large: in a sample of 826 popular Hub LoRAs for Wan, Qwen-Image, FLUX and LTX-2, 116 files load zero layers on current main, and those will now fail loudly rather than quietly degrade output.
fix empty output handling in many primitives ml-explore/mlx
Every primitive was expected to call set_data on all output arrays, and empty outputs such as shape (16, 0) produced a malloc(0) buffer; when a primitive skipped it, results segfaulted on eval. The example in the report, mx.array_equal on an empty matmul result, crashed reliably and no longer does.
Action items
- → Update vLLM to a build containing the int64 pointer arithmetic, shared media byte budget, prompt-count bound, and roo... vllm-project/vllm [immediate]
- → Audit LoRA adapters loaded through sglang set_lora; on current main 116 of 826 sampled Hub LoRAs load zero layers and... sgl-project/sglang [plan]