$ the-wire · showcase
CUDA fattn picks up head sizes 40 and 72, ROCm stack fills in FP8 and sparse MLA
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
llama.cpp closes two long-standing CUDA gaps (unusual attention head sizes and OCP FP8) while vLLM and SGLang spend the day making ROCm and Ascend backends actually keep up with their model families.
Cuda fattn mma head sizes 40 72 ggml-org/llama.cpp
Head sizes 40 and 72, the geometry used by the vision towers in Gemma 3/4 and Qwen3-VL, were excluded from every tensor-core dispatch branch in CUDA fattn and fell back to the tile kernel; the MMA fragment covers 16 halfs, so the kernel now pads geometry to 40 -> 48 and 72 -> 80 in registers, shared-memory strides and loop bounds while loading and storing the real size from global memory. Enabl...
[Attention][MiniMax-M3] Add FlashInfer MSA backend for SM90 and SM120/SM121 vllm-project/vllm
MiniMax-M3 sparse attention gets a FlashInfer msa_ops backend on Hopper and consumer Blackwell, GPUs that until now ran the Triton fallback because the vendored MSA kernels are SM100-only. It requires FlashInfer 0.7.3; on older builds the feature probes keep the Triton path, so merge order against the pin bump does not matter.
[ROCm][DSv4/4.1] Enable Tensor scale FP8 KV Cache for Sparse MLA vllm-project/vllm
DeepSeek V4 and V4.1 on ROCm gfx950 can now run --kv-cache-dtype fp8_e4m3 with AITER's sparse MLA kernel, storing each token as a plain 512-element FP8 row ([448 NoPE | 64 RoPE], one per-tensor scale) instead of the packed fp8_ds_mla record, which stays the default. The prerequisite is a hard one: AITER with ROCm/aiter#7429 (4e91c7ea6) or later.
[Perf][DSv4.1] DSpark draft on the specialized decode vllm-project/vllm
With decode_backend set to latency_specialized on GB300, the DSpark draft layers of DeepSeek-V4.1-Flash now run on the same decode executor as the target model, reusing its fused all-reduce + mHC kernel, attention and routed-expert GEMMs, and sampling through the model's sample_sequential hook with a Markov chain sharded by vocabulary across TP ranks. It is the speculative-decoding half of the ...
[Feat] paged-experts: a disk tier for experts that do not fit host memory sgl-project/sglang
Paged Experts kept every expert in a pinned host store, so a model that did not fit in RAM could not start at all; the new disk tier addresses exactly that case, sized from DeepSeek-V4-Flash-0731's 137.1 GiB of routed experts against a 62 GB host with roughly 45 GiB pinnable. Elsewhere the long tail holds a MoE stream-k fixup that read one element past its padding buffer, an FP8 MMQ gap deferre...