$ the-wire · showcase
Metal MoE/SSM fusion lands, MiMo-V2.5 fp8 sharding fixed
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
12 people shipped this
Georgi Gerganov
@ggerganov
1 cited
fadara01
@fadara01
1 cited
vllmellm
@vllmellm
1 cited
ScarWar
@ScarWar
1 cited
mjkvaak-amd
@mjkvaak-amd
1 cited
BBuf
@BBuf
1 cited
aparmp-quic
@aparmp-quic
aetherbird
@aetherbird
gty111
@gty111
DarkSharpness
@DarkSharpness
jiejingzhangamd
@jiejingzhangamd
shikicloud
@shikicloud
llama.cpp's Metal backend catches up to CUDA and Vulkan on MoE routing fusion, vLLM repairs a checkpoint that would not load at any tensor-parallel size, and sglang spends the day shaving DeepSeek-V4 overhead.
metal : add MoE and SSM_CONV fusion optimizations ggml-org/llama.cpp
Metal now fuses SOFT_MAX + ARGSORT + GET_ROWS with optional norm and scale so the kernel writes selected expert ids and routing weights directly, eliding the separate softmax, argsort, get-rows, sum-rows, clamp, div and scale passes, matching what CUDA and Vulkan already do. The same change adds MUL + expert-view + ADD reduction fusion, RMS_NORM + SCALE, SSM_CONV + UNARY (silu), and moves runti...
[Bugfix][Model] Fix MiMo-V2.5 fused fp8 qkv_proj sharding (pre-shard count is num_key_value_heads; MTP path too) vllm-project/vllm
XiaomiMiMo/MiMo-V2.5 could not load on main at any TP size, with TP4 on mi355x/mi300x hitting a tensor-size mismatch in _shard_fp8_qkv_proj at mimo_v2.py. The root cause is that the pre-shard count is num_key_value_heads, and the MTP path shares it; the fix is not platform specific, so every platform running that checkpoint is affected.
[CPU][Perf] refactor paged attention for Arm CPUs vllm-project/vllm
Arm CPU paged attention measured up to 25 percent faster in the PR's own numbers after faster GEMM kernels, quicker KV cache packing, prepacking Q during copy, and NEON enabled for block_size % 16. The refactor also removes roughly 100 lines overall and adds an interface letting the attention implementation write softmax probabilities directly in packed format.
[Bugfix] Fix SM100 fp8_ds_mla cache scales vllm-project/vllm
FlashMLA's fp8_ds_mla cache scale contract is now unified across SM90 and SM100: the 656-byte entry (512 FP8 NoPE values, four FP32 scale fields, a 128-byte BF16 RoPE tail) is unchanged, but each scale field now always stores 2 ** ceil(log2(max(amax / 448, 1e-4))). Power-of-two scales survive both reader conversions exactly, since SM90 converts to BF16 and SM100 to E8M0, which arbitrary FP32 va...
[ROCm][Perf] Enable the AITER GDN decode fast path for flat qkvz layouts vllm-project/vllm
AMD's AITER GDN decode fast path was gated on gqa_interleaved_layout, so only Qwen3-Next took it while flat-layout models such as Qwen3.5 and Qwen3.8 fell back to the generic path. A qkvz_layout parameter in ROCm/aiter now selects the source index mapping for both while outputs stay the same.
[Cleanup] Deduplicate kernel tests, diffusion fixtures and benchmark helpers sgl-project/sglang
Elsewhere: sglang removed 1,426 net lines across 26 files deduplicating kernel tests, diffusion fixtures and benchmark helpers while keeping the numerical checks and CI registrations, and it chunked the DeepSeek-V4 indexer MQA logits by query rows under a free-memory budget to avoid the single fp32 [query_rows, align256(max_compressed_seq_len)] allocation.
