$ the-wire · showcase
llama.cpp fuses NVFP4 weight scales into MMQ prefill, sglang fixes zeroed MoE routing under pipeline parallelism
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
The day's work is almost entirely performance and correctness fixes inside existing kernels: fused scales for FP4 prefill, a Mamba-aware eviction policy, and a three-line class of bugs where the wrong layer's parameters reached the wrong block.
CUDA: Optimize prefil via fuse of w_s scale in epilogue MMQ for nvfp4 checkpoints ggml-org/llama.cpp
Prefill batches on FP4-quantized checkpoints used to run the weight scale as a separate multiply after MMQ; the epilogue now folds s_W directly into the fused kernel, so per-tensor scales on dense projections and per-expert scales on MoE expert projections come for free at prefill batch sizes. It matters wherever you serve NVFP4 weights with *_s tensors.
[Feat] mem_cache: Evict Mamba states by GreedyDual-Size (--radix-eviction-policy gds) sgl-project/sglang
The Mamba state pool is far smaller than the KV pool (419 state slots against 2,969,600 KV tokens on GLM-5.3-Flash NVFP4 across 2x B200 at TP2), and LRU of one state at a time drops the prefix a returning agent session needs. --radix-eviction-policy gds evicts by GreedyDual-Size instead.
[Feat] pp: Relay routed experts through pipeline-parallel stages sgl-project/sglang
--enable-return-routed-experts under --pp-size > 1 only ever captured the first stage's layers and returned zeros for the rest: on Qwen3-30B-A3B with --pp-size 2, layers 24 to 47 came back empty, 49.6% of all ids. Routed experts now relay across pipeline stages, which is the difference between a usable and a poisoned RL routing replay.
[Fix] minimax-h3: preserve per-layer AdaLN parameters with Cache-DiT sgl-project/sglang
Cache-DiT replaced the gathered AdaLN stack with a single-entry wrapper during forward, so every MiniMax H3 block received layer 0's modulation parameters, and a full-compute Cache-DiT step could diverge from the uncached model before any cache reuse at all. Each block now keeps its own layer parameters.
[Bugfix][Rust Frontend] Resolve auxiliary frames of add requests in the mock engine vllm-project/vllm
vllm-mock-engine silently dropped every add request carrying an auxiliary ZMQ frame, so any prompt of 64 tokens or more never got an answer. A new EngineCoreRequest::from_frames resolves every array sent that way. The quieter long tail: an Intel prefill FA shader and a Vulkan GGML_VULKAN_COORDINATE compile fix in llama.cpp, SpacemiT RVV and Q4_1 dispatch, a search_chats tool that ships disabled...