$ the-wire · showcase
GPU backends converge on sparse attention as vLLM wires DeepSeek V4.1 indexer
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
15 people shipped this
Zijun Yu
@wine99
2 cited
0cc4m
@0cc4m
1 cited
JaredforReal
@JaredforReal
1 cited
mgoin
@mgoin
1 cited
jiejingzhangamd
@jiejingzhangamd
1 cited
xiaobochen-amd
@xiaobochen-amd
1 cited
JinyanYi
@JinyanYi
1 cited
dhiltgen
@dhiltgen
MohamedElashri
@MohamedElashri
lhez
@lhez
zixi-qi
@zixi-qi
njhill
@njhill
TheEpicDolphin
@TheEpicDolphin
Yuxingwang-intel
@Yuxingwang-intel
hnyls2002
@hnyls2002
Sparse attention and MoE decode paths landed across llama.cpp, vLLM, and SGLang today, while a silent ROCm sampling bug in SGLang's EAGLE verify path turned out to be ignoring temperature entirely.
[DSA] Wire DeepGEMM sparse MQA logits into the DeepSeek V4.1 indexer vllm-project/vllm
DeepSeek V4.1 consumer indexer layers used to score the whole context with dense MQA logits and mask everything outside the top candidate blocks; they now score only the blocks DeepGEMM 2.8's sparse MQA-logits kernels hand them. In one line that is the difference between attention work proportional to context length and work proportional to the candidate set.
[ROCm] Fix EAGLE spec-decode verify silently sampling greedy on HIP sgl-project/sglang
SGLang's EAGLE spec-decode verify was committing argmax on ROCm regardless of temperature or top_p, because _is_hip sat next to is_all_greedy in the eagle_sample branch, and nothing in the log said so. At temp > 0 it degenerates into repetition loops; the fix restores real sampling on HIP.
vulkan: support sparse Flash Attention ggml-org/llama.cpp
Vulkan joins the sparse Flash Attention work tracked in the upstream CUDA effort, giving Vulkan backend users the same skipping of masked attention blocks rather than a full dense pass.
OpenVINO: optimize stateful decode and GPU MoE inference ggml-org/llama.cpp
OpenVINO gains compressed expert fusion with grouped 8-bit requantization for GPU MoE, plus weight spilling and a configurable K-quant requant target via GGML_OPENVINO_REQUANT_KQUANT and GGML_OPENVINO_SPILL_DIR. Stateful KV-cache handling now covers sliding-window and per-layer head layouts, and cacheless encoder models run on NPU.
[Perf][Kernel] Add sampled filtering for persistent top-k vllm-project/vllm
Sampled filtering estimates a cutoff, compacts survivors in shared memory, and finishes exact FP32 selection on those candidates, which is what makes long sparse-indexer decode top-k tractable. It only kicks in above 64 rows and at actual valid length >= 98,304 for k=512 (or >= 65,536 for k=1024/2048) with at least 144 KiB of opt-in shared memory; outside those bounds, full-row selection.
[AMD] GLM-5.2 NextN: cast draft fused MoE to per-channel FP8 sgl-project/sglang
GLM-5.2's layer 78 MTP draft reads its routed and shared experts in bf16 from the MXFP4 checkpoint, 71.7 MB per expert against 19.0 MB for a decoder layer, once per draft step. Casting the draft fused MoE to per-channel FP8 removes that repetition from bandwidth-bound decode.
fix(npu): fix hybrid KV transfer with PP prefill in PD disaggregation sgl-project/sglang
Kimi-K3's HybridLinearKVPool only gives KV buffers to sparse full-attention layers and Ascend exposes multiple buffer groups per layer, so positional PP slicing picked the wrong ones; Ascend hybrid-linear PP KV transfer now routes through global layer-id pairing with Mamba state slot sizes validated after pairing.
OpenVINO: optimize stateful decode and GPU MoE inference (#28638) ggml-org/llama.cpp
llama.cpp's OpenVINO backend picked up stateful decode fixes for Gemma-4 per-layer head sizes, structural sliding-window classification on interleaved-SWA models, and a fix for a GPU/NPU failing POOL_2D case.
Action items
