$ the-wire · showcase
Ollama makes Metal history adaptive, llama.cpp widens M5 conv2d
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
Ollama replaced its fixed 8 GiB Metal history allowance with memory-pressure-driven reclamation, while llama.cpp shipped an M5 tensor-API CONV_2D kernel and a 3x faster IQ2 creation path.
mlxrunner: adapt saved history to memory pressure (#19678) ollama/ollama
Saved conversation history in Metal runners no longer gets a fixed 8 GiB budget split across request slots; the runner now retains prefix history using observed memory headroom and releases cold buffers when system compression grows, keeping active requests intact. The old behavior could discard useful conversations while memory was still free, forcing later requests to reprocess their prompts ...
metal : CONV_2D tensor API for M5, split-K for all chips ggml-org/llama.cpp
On M5, CONV_2D runs as a matmul on the tensor cores, gathering 64 output positions of patches into threadgroup memory and reading kernel rows directly from device memory; M1 through M4 keep the simdgroup path, which this PR also speeds up. Depends on the earlier simdgroup CONV_2D kernel landing for all Apple Silicon.
IQ2 creation speedup through caching ggml-org/llama.cpp
Caching repeated patterns during quant creation is byte-identical across all IQ2 types, verified on 8 models across 5 architectures. On Qwen3.6-35B-A3B at 32 threads, the PR reports 3.04x for IQ2_XS and 2.99x for IQ2_S end-to-end quant speedup, which matters if you build quants locally rather than pull them.
[Perf][MoE] Enable SM90 MXFP4 MegaMoE via FlashInfer MoE-EP vllm-project/vllm
Passing --moe-backend flashinfer_moe_ep_cutedsl now runs MXFP4 MoE checkpoints such as DeepSeek-V4.1-Flash on Hopper via FlashInfer's SM90 MXFP4 x FP8 MegaMoE, with packed weights going to the kernel without a bf16 round trip. Previously that path was not available on SM90.
[Fix] sgl-router: match the engine's Responses summary events, argument coercion, and error type sgl-project/sglang
The router streamed response.reasoning_text.* where the engine emits response.reasoning_summary_part/_text.*, so Codex clients saw no live reasoning through the router; the fix mirrors serving_responses.py, and closes three further parity gaps in argument coercion and error type. Anyone putting sgl-router in front of /v1/responses should retest reasoning display.
Do not wire user provided buffers ml-explore/mlx
With a wired limit set, MLX was adding caller-owned buffers from make_buffer to a residency set, so Metal wired the entire mapping; for an mmap-backed file that read the whole file from disk even when only a few rows were touched on the CPU. Measured on an Apple M5 Max with a 1 GiB file outside the page cache.
[Security] Reject ICNS image inputs before decoding (#68245) vllm-project/vllm
Smaller items: vLLM now rejects ICNS image inputs before decoding, the vllm preload source daemon keeps running and advertises no bundle when its compiled cache exceeds the 8 GiB limit, and MLX fixed the ring backend build on FreeBSD.
Action items
- → Retest live reasoning streams through sgl-router after upgrading past the responses parity fix sgl-project/sglang [plan]