$ the-wire · showcase
MLX engine graduates out of x/, ROCm fixes silent attention corruption
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
14 people shipped this
jessegross
@jessegross
2 cited
rjtokenring
@rjtokenring
1 cited
Titaniumtown
@Titaniumtown
1 cited
dongluw
@dongluw
1 cited
hoyyeva
@hoyyeva
noctrex
@noctrex
am17an
@am17an
Flora Feng
@sfeng33
BugenZhao
@BugenZhao
Rohan138
@Rohan138
divakar-amd
@divakar-amd
Kangyan-Zhou
@Kangyan-Zhou
alphabetc1
@alphabetc1
hnyls2002
@hnyls2002
Ollama promoted its MLX runner to the top level and finished clearing the dead Go engine, while vLLM fixed a ROCm bug that could zero out a model's answers without raising an error.
Move the MLX engine out of x/ ollama/ollama
The MLX runner is now Ollama's only Go engine and no longer experimental, so it moves from x/ to the top level: bindings become mlx/ (mirroring llama/), the runner becomes mlxrunner/, and the import pipeline and safetensors reader move to create/ and fs/safetensors. Package names are unchanged, so importing code sees only new paths.
cmd: launch the MLX runner without the engine dispatcher ollama/ollama
Ollama now calls mlxrunner.Execute directly from the hidden runner subcommand and drops the --mlx-engine argument the MLX client spawned, deleting the single-arm engine dispatcher and its stale README. If you script against the runner subcommand or --mlx-engine, that flag no longer exists.
[ROCm][Bugfix] Ignore descales for unquantized AITER caches vllm-project/vllm
ROCM_AITER_FA was passing an attention layer's unused K/V scale tensors to AITER decode kernels; when those tensors held non-neutral values, such as stale dummy-weight initialization after a weight reload, AITER read them as descales and corrupted attention output. Their MI300X reproduction dropped the post-reload InfoVQA score from 0.952 to 0.000, and the fix passes null descales for unquantiz...
hexagon: Support for K-Quants Q4_K and Q6_K ggml-org/llama.cpp
llama.cpp added Q4_K and Q6_K support to the Hexagon backend, reusing Q4_1 infrastructure for Q4_K and adding new kernels for Q6_K, which unlocks Q4_K_M models (typically Q4_K and Q6_K mixed). The maintainer reports validation across multiple runs on the IQ8 (VentunoQ) and IQ9 platforms.
sycl: ssm_conv: fuse the SiLU epilogue into the ssm_conv kernel ggml-org/llama.cpp
The SYCL ssm_conv kernel fuses the SiLU epilogue rather than running it as a separate pass; Titaniumtown notes ssm_conv "is an op that is called very frequently during the prefill phase of many models." The long tail this cycle is otherwise housekeeping: Ollama removed dead tokenizer and ml/backend.go code, added an ollama://apps deep link for macOS and Windows, vLLM refactored the Rust fronten...
Action items
- → Upgrade vllm to a build containing the AITER decode fix if you serve quantized-free KV cache on ROCm with ROCM_AITER_FA vllm-project/vllm [immediate]
- → Drop --mlx-engine from any scripts or launch configs that invoke the Ollama runner subcommand ollama/ollama [plan]
