$ the-wire · showcase
Ollama honors YaRN in Qwen3, llama.cpp fixes quantized KV state copies
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
17 people shipped this
pdevine
@pdevine
1 cited
dhiltgen
@dhiltgen
1 cited
anujj
@anujj
1 cited
Hongqiang Wang
@wanghqc
1 cited
AndreasKaratzas
@AndreasKaratzas
1 cited
netrunnereve
@netrunnereve
shaofeiqi
@shaofeiqi
allozaur
@allozaur
Pb314314
@Pb314314
gcanlin
@gcanlin
Misha Goin
@mgoin
Kevin H. Luu
@khluu
chang-wenbin
@chang-wenbin
jiaryang
@jiaryang
bingxche
@bingxche
DarkSharpness
@DarkSharpness
CaoE
@CaoE
A correctness day across the local inference stack: the two fixes that matter most are state copy sizes for quantized KV caches in llama.cpp and a startup-time reject for unsupported CUDA devices in Ollama's MLX runner.
llama : fix on-device state copy sizes for quantized KV caches (#30541) ggml-org/llama.cpp
The on-device sequence state copy computed its element count as size / ggml_element_size(), which for q8_0 or q4_0 is the number of blocks rather than elements, so only part of each range was copied: restored state came back wrong, or the copy crashed. The count now divides by ggml_type_size() and scales by the block size, so sequence state save and restore over quantized KV caches behaves as i...
mlx: reject unsupported CUDA devices ollama/ollama
The MLX CUDA runner used to initialize on a GPU whose architecture the build had no compatible code for, then fail later with a backend/load error; it now reads cuda.get_arch_list() (merged upstream in ml-explore/mlx#4761) and rejects the device at startup. The MLX pin moves to 54a70511 with one temporary compatibility patch carried until MLX-C picks it up.
mlx: support YaRN scaling in Qwen3 and Qwen3.5 ollama/ollama
Ollama's MLX path now honors explicit RoPE scaling metadata in both supported config schemas, applying YaRN frequencies and amplitude scaling to text and mRoPE while leaving non-rotary dimensions and unscaled defaults untouched. Reference fixtures cover prefill, decode and partial rotation, so it fixes the two reported issues where extended-context Qwen3 configs were loaded without their scaling.
[Bugfix][UMBP] Copy every KV cache layout's per-head runs vllm-project/vllm
UMBP was taking cache.stride(0) as the block stride and copying one contiguous page, which is wrong once a page is split per head; the embedded bulk path now registers per-head runs from compute_layout_strides and falls back to per-range copies where the stride does not match. It also fails at startup instead of corrupting KV when a page is split and kv_cache_layout is unset, and all 6 layouts ...
Title: CUDA: fuse NVFP4 gate/up matmuls and GLU in MMQ ggml-org/llama.cpp
On the performance side, the fused gate/up plus GLU MMQ kernel now runs for NVFP4 FFNs, both dense and MoE via MUL_MAT_ID, across the W4A4 native FP4 MMA path (default on Blackwell) and W4A8, with the per-tensor or per-expert .scale applied in the fused write-back. Elsewhere: a version bump to llama.cpp b12111 drops the superseded 002-metal-unary-cast-constexpr.patch, right-click actions landed...
