$ the-wire · showcase
vLLM splits prefill and decode graph sizes, GLM MTP norm aligned
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
16 people shipped this
dhiltgen
@dhiltgen
1 cited
Aleksander Grygier
@allozaur
1 cited
pskrunner14
@pskrunner14
1 cited
Pb314314
@Pb314314
1 cited
spandantiwari
@spandantiwari
1 cited
ByronHsu
@ByronHsu
1 cited
hoyyeva
@hoyyeva
Todor Boinovski
@tboinovski1
randomized
@renewboy
ZhenchengLin
@ZhenchengLin
djmmoss
@djmmoss
hshrivastava-droid
@hshrivastava-droid
alphabetc1
@alphabetc1
yan-lgtm
@yan-lgtm
LinyuanLi0046
@LinyuanLi0046
ispobock
@ispobock
vLLM's two biggest changes today are both about correctness and throughput at the graph level, while llama.cpp's top stories this cycle land in new model support and the Windows-free Hexagon backend.
[Core][CUDA Graph] Separate prefill (token) and decode (request) CUDA graph capture sizes vllm-project/vllm
Model runner V2 used to capture prefill (token-sized) and uniform decode (request-sized, scaled by q = 1 + K) graphs from one shared `cudagraph_capture_sizes` list capped at 1024 tokens on GB200; it now keeps separate capture sizes per step kind. A decode graph must be a list entry divisible by q, so the old sharing was getting tighter as K grew.
[Bugfix][Spec Decode] Align speculator hidden norm in GLM-4.5/4.6 MoE and GLM-OCR MTP with GLM5Next to recycle post-final-hidden. vllm-project/vllm
`glm4_moe_mtp.py`, `glm4_moe_lite_mtp.py` and `glm_ocr_mtp.py` used to return the pre-final-norm hidden from the MTP layer, so the speculator was recycling a hidden one RMSNorm short into the next draft step's `hnorm`. The fix brings GLM-4.5/4.6, GLM-4.7-Flash and GLM-OCR in line with the GLM5Next contract already merged for DeepSeek-NextN and SGLang.
mtmd : add Nemotron 3.5 Transcribe support ggml-org/llama.cpp
llama.cpp's mtmd adds NVIDIA's Nemotron 3.5 Transcribe SALM with offline transcription in 40 languages, prompt context and speaker attribution. Word timestamps and streaming are explicitly out of scope, so pipelines that need those still have no path through this backend.
mlx: load bundled CUDA libraries in-process on Linux ollama/ollama
On Linux, MLX used to pick up the host's libcublas and libnvJitLink because libcusolver searches only its own directory and LD_LIBRARY_PATH does not affect the server's in-process load; the server now preloads the bundled copies by full path before loading MLX. The failure mode it removes is a mismatched host CUDA stack breaking MLX.
[Perf] R3: Keep the routed-expert host cache only on the output rank (1/3) sgl-project/sglang
With `--enable-return-routed-experts`, every rank used to keep a pinned host copy of routed experts while only the rank talking to the detokenizer ever sent them out; the host cache now lives only on that output rank. That is the CPU RAM line item that matters on GB300-class nodes.
ui: run local browser tests in installed Chrome with its sandbox on (#31596) ggml-org/llama.cpp
Also worth knowing: llama.cpp's local UI browser tests now run in the installed, auto-updated Chrome with the sandbox on rather than Playwright's bundled Chromium, which ships 141 at 1.56.1 and is launched with --no-sandbox by default. CI keeps the bundled browser, so self-hosted runners are unaffected.
