$ the-wire · showcase
Faster Metal MXFP4 decode, a GLM-4.7 streaming parser stall fixed
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
20 people shipped this
dhiltgen
@dhiltgen
1 cited
am17an
@am17an
1 cited
Dao007forever
@Dao007forever
1 cited
vMaroon
@vMaroon
1 cited
zcnrex
@zcnrex
1 cited
hnyls2002
@hnyls2002
1 cited
aleroot
@aleroot
1 cited
lhez
@lhez
giuseppe
@giuseppe
max-krasnyansky
@max-krasnyansky
Juan Pérez de Algaba
@jperezdealgaba
zyongye
@zyongye
CJJ1008
@CJJ1008
AndyLi429
@AndyLi429
ispobock
@ispobock
SKRohit
@SKRohit
zcbenz
@zcbenz
arian262785-cell
@arian262785-cell
devYRPauli
@devYRPauli
ronaldmannak
@ronaldmannak
llama.cpp and ollama trimmed single-token decode on Apple hardware while vLLM plugged a parser loop that grew API-server cost per token with every closed argument pair.
[Bugfix][Parser] Parse closed GLM-4.7 tool arguments once while streaming vllm-project/vllm
Glm47MoeParser re-parsed every closed argument pair on each streaming conversion, so per-token cost climbed until the event loop saturated and liveness failed; it now parses each closed pair once. A follow-up to the earlier fix for a single long open value, so streamed GLM tool calls with repeated closed pairs stop degrading the server as the call grows.
metal : faster MXFP4 mat-vec, full threadgroups for unary/binary/GLU ggml-org/llama.cpp
Three Metal changes for single-token decode: MXFP4 mat-vec decodes fp4 via the fp16 bit trick instead of a threadgroup lookup table and reads packed nibbles as aligned 32-bit words, with kernel_mul_mv_id_mxfp4_f32 carrying the same path to MoE experts, plus full threadgroups for contiguous unary and row-broadcast binary and GLU ops. The work is upstreamed from QVAC Fabric.
mlx: sort long top-k rows in chunks ollama/ollama
mlx.TopKIndices sorts each 2048-value chunk in a single pass, then only the chunks' top k, returning the same indices in the same order. It replaces a full-vocabulary argpartition in the sampler's top_k and top_logprobs and in the DFlash2 candidate selector, which matters because MLX's argpartition sorts the whole row and a row over 2048 values on Metal takes a multi-pass merge sort, most of th...
Keep the existing file when save, save_safetensors or save_gguf fails ml-explore/mlx
save, save_safetensors and save_gguf opened the target with O_CREAT|O_WRONLY|O_TRUNC, so any failure after the open destroyed the existing file even though the open was moved after eval. The fix keeps the previous file intact on failure, which is the difference between a failed checkpoint write and a lost one.
[Fix] disaggregation: Keep output ids during a PD rebootstrap instead of popping and replaying the last one sgl-project/sglang
A PD decode rebootstrap popped the last output id before recompute and appended it back at transfer commit, overriding the token the prefill sampled and leaving output_ids one shorter than what the client had received. During that window an output row under overlap scheduling reported a shrunken completion_tokens.
[Bugfix][KV Connector][NIXL] Fix Mamba region stride on block-outermost layouts vllm-project/vllm
NIXL transfers of Mamba/SSM hybrid state were wrong on block-outermost KV cache layouts BLHNC and BLNHC; the fix adds a CPU test that moves one request between two NIXL workers on real model KV layouts and checks every byte.
[Perf] dsv4: Fix amax8_varlen and mega MoE pre-dispatch grids for 1024-thread SMs sgl-project/sglang
Two DeepSeek-V4.1 decode kernels lost time to grid shape rather than bandwidth on SMs holding 1024 resident threads (CC 10.7 / sm_107): amax8_varlen sized grid.y from max_compressed_seq_len and launched roughly 30 waves of empty CTAs at 128K context. Rounding out the day: IQ2_S/IQ2_XS/IQ2_XXS in the OpenCL backend, UPSCALE for Hexagon image-to-text, and Limite 1B - Violetto in llama.cpp; Moonca...
