$ the-wire · showcase
Metal grid truncation fixed, vLLM patches GLM-5.3 weight loss
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
18 people shipped this
dhiltgen
@dhiltgen
1 cited
anujj
@anujj
1 cited
anthony-wss
@anthony-wss
1 cited
trunk-trick
@trunk-trick
1 cited
pllopis
@pllopis
1 cited
ArupKantiDas
@ArupKantiDas
1 cited
Max Krasnyansky
@max-krasnyansky
am17an
@am17an
khluu
@khluu
WoosukKwon
@WoosukKwon
jeejeelee
@jeejeelee
esmeetu
@esmeetu
chien-an-chen
@chien-an-chen
kpham-sgl
@kpham-sgl
ShangmingCai
@ShangmingCai
hnyls2002
@hnyls2002
Satchitananda
@Satchitananda
Marcos Ashton
@MarcosAsh
MLX closed a silent correctness bug that returned zeros for most rows once a Metal grid crossed 2^27 rows, while vLLM fixed dropped FP8 indexer weights that made GLM-5.3 start in thinking loops.
Fix Metal grid truncation in softmax/logsumexp/rms_norm/layer_norm past 2^27 rows ml-explore/mlx
Past roughly 2^27 rows of 128 numbers or fewer, Metal silently truncated the grid, so most rows of softmax, logsumexp, rms_norm and layer_norm came back as 0 rather than erroring. The layer_norm gradient and the long-row rms_norm gradient carried the same defect; a new grid helper now launches work in bounded pieces.
[Bugfix] Preserve FP8 indexer WK pairs across load_weights calls vllm-project/vllm
The loader parked the first half of each FP8 indexer wk pair (values plus scale) in a dict created fresh on every load_weights call, so non-deterministic I/O completion order under runai_streamer could drop weights and leave GLM-5.3 / DeepSeek-V3.2 (deepseek_v32) starting in thinking loops.
CUDA: fuse FFN gate/up matmuls and GLU in MMQ with interleaved gate/up rows ggml-org/llama.cpp
Gate/up matmul fusion and the following GLU now run on the MMQ path for prompt-processing batch sizes, not just the small batches covered by MMVQ/MMVF. It keeps a single accumulator and applies the GLU in write-back instead of carrying a second gate tile.
llama-quant : fix 4D tensor quantization using ggml_nrows() and name-based expert detection ggml-org/llama.cpp
Quantization counted rows from ne[1] and ne[2] and ignored ne[3], so a tensor like a.conv1d.0.weight [3, 3, 1, 128] got the wrong size and tripped the gguf_get_tensor_size(...) == new_size assert. The fix uses ggml_nrows() and identifies expert tensors by name instead of shape.
fix exhausting CPU and MEM caused by dry sequence breaker ggml-org/llama.cpp
Separately, dry_sequence_breakers with a large breaker count (the report used 10000) could burn CPU and memory for minutes just initializing, because the duplicate check in the sampler was leaky and incomplete; count checks in two places cap the redundant work.
[Metal] Add sdpa_split kernels for mid rows for speculative decoding ml-explore/mlx
The rest of the day is smaller: MLX added sdpa_split kernels filling the gap between the 8-token vector path and full attention for 33 to 128 rows, ollama bumped MLX/MLX-C to pick up SDPA window_size for gemma, flint and laguna prefill, and sglang landed AMD mamba chunk-grid cutting, an SM107 split-K FP32 GEMM for Kimi-K3 and block-FP8 replication under --dcp-replicate-q-proj.
Action items
- → Update vllm past the load_weights fix if you serve GLM-5.3 or DeepSeek-V3.2 checkpoints with runai_streamer vllm-project/vllm [plan]
- → Update mlx past the grid truncation fix before running softmax, rms_norm or layer_norm on more than 2^27 rows ml-explore/mlx [plan]
- → Re-quantize 4D-tensor models once llama.cpp includes the ggml_nrows() fix, since earlier runs could fail the tensor-s... ggml-org/llama.cpp [plan]
