$ the-wire · showcase
vLLM EngineCore can be killed by crafted stop_token_ids, llama.cpp reworks JSON schema handling
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
vLLM patched a cluster of input-validation gaps that let out-of-range or Rust-path stop_token_ids reach the EngineCore and terminate it, while llama.cpp landed a new internal JSON schema representation.
vLLM closed four breaking input-validation holes that could terminate the EngineCore. Out-of-range stop_token_ids combined with min_tokens can kill the engine [1], and the Rust HTTP/gRPC paths bypass the existing Python vocab-bound fix, offering the same route to engine termination [2]. Two multimodal-specific gaps also shipped: disaggregated generate skips decoder prompt-length validation for some multimodal processors [3], and a sampler subclass that shadows the counter bypasses the PyNvVideoCodec decoder limits and GPU memory accounting [4]. FunAudioChat and Tarsier2 had incomplete artifact pin propagation [5].
llama.cpp landed its common_schema internal representation for JSON schemas. The refactor gives json-schema-to-grammar a proper internal type, resolves refs up front instead of at grammar creation, and removes the common_schema_info workaround [6]. The commit series adds common_schema types with a schema optimizer, moves json-schema-to-grammar onto it, switches to common_trie, and drops common_chat_tool_parameters [7]. Separately, ggml-cuda added an AMD GCN per-arch MMQ config that handles wave64 (nthreads 512, 8 warps) instead of falling back to the RDNA2 config (wave32/nthreads 256) [8]. cpp-httplib was bumped to 0.56.0 [9], and the web UI gained asset caching for faster builds [10].
sglang aligned /v1/responses with Codex-class clients. Declaring a custom tool used to return no call, and tool_choice="required" returned HTTP 400, because _response_tools_to_chat_tools skipped every non-function tool and the required gate only accepted function [11]. include=["reasoning.encrypted_content"] was validated but never populated, so a store=false client could not replay reasoning [11]. In other sglang work, GraniteMoE now loads split per-expert quantized MoE weights; compressed-tensors checkpoints store experts as experts.<id>.{gate,up,down}_proj.{weight,weight_scale} while the unquantized HF checkpoint packs them into input_linear/output_linear, and the packed-only reader fell through to mixtral's warning, a silent failure where the server started and reported success [12]. Attention backend selection is now measured rather than assumed: allow_cudnn_sdp wraps the SDPA call in sdpa_kernel(_PYTORCH_DEFAULT_CUDA_SDP_BACKENDS), but torch's signature is sdpa_kernel(backends, set_priority=False), so without the priority flag it was an allow-set rather than an order and equivalent to nullcontext() for every caller [13]. The unused tokenwise QSA implementation and its tests were removed [14][15].
Action items
- → Upgrade vLLM to a release containing the stop_token_ids validation fixes before exposing an untrusted endpoint vllm-project/vllm [immediate]
- → Re-test custom tool declarations and include=["reasoning.encrypted_content"] against sglang /v1/responses after the Codex-client alignment sgl-project/sglang [plan]
- → Verify GraniteMoE compressed-tensors checkpoints load real expert weights rather than starting with a silent warning sgl-project/sglang [plan]
References
- [1] Out-of-range `stop_token_ids` with `min_tokens` can kill vLLM EngineCore vllm-project/vllm ↗
- [2] Rust HTTP/gRPC stop_token_ids bypass Python vocab-bound fix and can terminate EngineCore vllm-project/vllm ↗
- [3] Disaggregated generate skips decoder prompt-length validation for some multimodal processors vllm-project/vllm ↗
- [4] Sampler Subclass Counter Shadowing Bypasses PyNvVideoCodec Decoder Limits and GPU Memory Accounting vllm-project/vllm ↗
- [5] Incomplete artifact pin propagation in FunAudioChat and Tarsier2 vllm-project/vllm ↗
- [6] common : implement common_schema internal representation for JSON schemas ↗ ggml-org/llama.cpp
- [7] common : implement common_schema internal representation for JSON schemas (#28736) ↗ ggml-org/llama.cpp
- [8] ggml-cuda: hip: add missing AMD GCN MMQ config ↗ ggml-org/llama.cpp
- [9] vendor : update cpp-httplib to 0.56.0 ↗ ggml-org/llama.cpp
- [10] ui : add cache ↗ ggml-org/llama.cpp
- [11] [Feat][Responses API] Support custom tools, encrypted reasoning replay, developer tier and model validation ↗ sgl-project/sglang
- [12] [GraniteMoE] Load split per-expert quantized MoE weights ↗ sgl-project/sglang
- [13] [Diffusion] Pick the attention backend by measuring it ↗ sgl-project/sglang
- [14] [Qwen 3.8 Next] Remove unused tokenwise QSA implementation and tests ↗ sgl-project/sglang
- [15] [Qwen 3.8 Next] Remove unused tokenwise QSA implementation and tests (#38960) ↗ sgl-project/sglang
