$ the-wire · showcase
Ollama fixes cloud metadata for every launcher, MLX speeds 6-16 row decode ~3x
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
19 people shipped this
dhiltgen
@dhiltgen
1 cited
jberg5
@jberg5
1 cited
hoyyeva
@hoyyeva
1 cited
Tim Wang
@timothywang21
1 cited
lishuoshuo-amd
@lishuoshuo-amd
1 cited
michalk8
@michalk8
1 cited
kurquhar
@kurquhar
budiony
@budiony
Max Krasnyansky
@max-krasnyansky
malsbat
@malsbat
Mritunjay Kumar Sharma
@mritunjaysharma394
benenzhu
@benenzhu
shen-shanshan
@shen-shanshan
esmeetu
@esmeetu
wookjeHan
@wookjeHan
Fridge003
@Fridge003
chenyang08056032
@chenyang08056032
hnyls2002
@hnyls2002
mickqian
@mickqian
Today's work is mostly repair at the edges: a hard crash in MLX autodiff, a silently dropped failure event in Ollama's Responses stream, and a genuinely large Metal decode win for batched and speculative generation.
[Metal] Add qmv_mma/qmv_nax kernels for 6-16 rows with large N/K ml-explore/mlx
New qmv_mma (M1-M4) and qmv_nax Metal kernels read each weight block once and multiply rows on the matrix units, avoiding the 32-row Split-K padding waste; the PR reports ~3x speedup on M1/M2 generation and ~2x on newer systems for fp quant models generating 6-16 sequences. The same work landed on main, with the commit noting co-authorship by Cheng.
Make sure all launchers know about cloud model metadata ollama/ollama
`ollama launch <harness>` now sees cloud model inventory: Codex's catalog context_window no longer falls back to 128,000 for untabled models (glm-5.2 gets 1,048,576), and CLAUDE_CODE_MAX_CONTEXT_TOKENS plus the auto-compact window are set from the inventory for the 15 models that previously used Claude's own defaults.
openai: preserve failures in Responses streams ollama/ollama
Once /v1/responses begins streaming, HTTP 200 is committed and a native error envelope was being decoded as an empty chat response, so clients saw EOF with no terminal failure event. It now detects in-band generation errors and emits one response.failed, preserving partial text, reasoning, and completed tool output.
server : support typed content input for RERANK endpoint (#29625) ggml-org/llama.cpp
The RERANK endpoint accepts typed content input, and review hardening now rejects multimodal rerank input when the model has no rerank chat template (images were silently dropped) and returns 400 rather than 500 for media download or base64 failures, and 400 for a negative top_n instead of letting resize throw.
[Bugfix][ROCm][MiniMax-M3] Reject IndexCache overrides that miss text_config vllm-project/vllm
MiniMax-M3 reads IndexCache settings from text_config, so the documented top-level --hf-overrides form was silently ignored and the server started with IndexCache off and no log line. On ROCm, top-level use_index_cache, index_topk_freq, or index_skip_topk_offset now raise a ValueError when ModelConfig is built, with the nested override spelled out in the error.
Fix/padding jvp vjp ml-explore/mlx
The VJP returned only one element (the one w.r.t. x) while callers indexed vjps[1], and the guard assert is absent in release builds, so the path segfaulted; JVP is now allowed w.r.t. the padded value. Elsewhere in MLX, mx.take now handles negative indices and normal loc/scale shape is corrected.
