RepoJournal
Local LLMs Local LLMs
59 wires and counting

$ follow Local LLMs

Keep up with Local LLMs in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.

or

fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.

Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.

$ status

wire 2026-09-11
stories 260
RSS Archive Reviews Topics

© 2026 RepoJournal Home Showcase Explore How it works Privacy

$ the-wire · showcase

Ollama fixes unbounded KV memory growth, vLLM clarifies CVE timing

By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology

jessegross dhiltgen ggerganov Iggy Jackson shaofeiqi Juan Pérez de Algaba juliendenize
15 people shipped this
jessegross @jessegross 4 cited dhiltgen @dhiltgen 1 cited ggerganov @ggerganov 1 cited Iggy Jackson @iggy 1 cited shaofeiqi @shaofeiqi 1 cited Juan Pérez de Algaba @jperezdealgaba 1 cited juliendenize @juliendenize 1 cited simondanielsson @simondanielsson 1 cited clintg6 @clintg6 1 cited mmangkad @mmangkad 1 cited liupeng374 @liupeng374 1 cited SalikovAlex @SalikovAlex njhill @njhill Jialin @Jialin jamesjxliu @jamesjxliu

Ollama lands a scope-based rewrite of its MLX memory bindings and prefix-cache eviction fixes that previously let long conversations grow memory without bound, while vLLM documents when it publishes CVEs.

Ollama's MLX bindings used to free arrays by sweeping everything not pinned, so no caller could release memory without knowing what every other caller still held, and code that never swept accumulated until memory ran out. Every array now belongs to a scope, entered with Scoped or one of its variants [1]. The prefix cache previously protected every node on the active path from eviction, so a conversation's own snapshots were never freed no matter how far over budget the cache was; models with sliding window or recurrent layers store a complete copy of that state per snapshot (about 800 MB per turn on gemma4:31b) and memory grew without bound over a long conversation. Snapshots are now evicted from the active conversation [2], and the reused head of a cached edge is kept safe from eviction when a request resumes partway through that edge [3]. The empty-handle checks outside the bindings are gone too: a missing tensor is nil, and a handle only loses its array when its scope frees it [4].

Separately, GGUF metadata is now extracted once per blob into a file at <OLLAMA_MODELS>/metadata/sha256-<hex>.json, with only arrays over 4096 elements and non-finite floats left out. Two caches had evolved to mitigate the expensive load, and the two capability implementations produced inconsistent results for some models; Capabilities() discovery now costs microseconds instead of milliseconds, and /api/tags can build directly from manifests and the extracted metadata [5].

In llama.cpp, the SYCL-specific changes to test-backend-ops.cpp from an earlier PR are partially reverted, since they were specific to the SYCL backend [6]. Dead switch branches copypasted in a former refactor were cleaned out of the old model code; the only remaining shared code is llama_model_bert::graph::graph [7]. An optimized Q4_0 non-MoE GEMM bin kernel plus the matching GEMV kernel for Adreno targets OpenCL [8].

vLLM's docs now clarify how the project credits reporters and when it publishes CVEs [9]. On the backend, tokenizer_mode="hf" now forces mistral_format=False so MistralCommonBackend no longer reaches HF processors and structured-output backends, removing the need for processor-level compatibility wrappers and deleting the now-obsolete MistralCommonBackend tokenizer caching and structured-output tests [10]. KV connectors now work with ROCM_AITER_UNIFIED_ATTN; the flag previously inherited False from ROCM_ATTN, which is not a block-first layout, and was validated on MI300 across MoRI, DecodeBenchConnector, and NIXL [11].

In SGLang, DSA prefill and decode can now be pinned to a pure Triton sparse MLA backend on ROCm via --dsa-prefill-backend triton and --dsa-decode-backend triton, validated on MI355X (gfx950) and MI300X (gfx942) and replacing the SGLANG_DSA_TRITON_PREFILL env-var gate [12]. The expert_pack_mxfp4 kernel moved to load_jit: its old torch.utils.cpp_extension.load path waited on a lock file released only in a finally, so a build killed mid-flight left the file behind and every later run against that cache spun forever with no output, burning a full CI timeout on a 7s test [13]. A pp=2 hang on NPU is fixed by issuing paired send/recv via batch_isend_irecv, which prevents simultaneous pp0->pp1 proxy and pp1->pp0 output sends from deadlocking [14].

Action items

References

  1. [1] mlx: scope array lifetimes instead of pinning and sweeping ↗ ollama/ollama
  2. [2] mlxrunner: Evict prefix cache snapshots from the active conversation ↗ ollama/ollama
  3. [3] mlxrunner: keep the reused head of a cached edge safe from eviction ↗ ollama/ollama
  4. [4] mlx: drop the empty-handle checks outside the bindings ↗ ollama/ollama
  5. [5] server: extract GGUF metadata and unify capabilities ↗ ollama/ollama
  6. [6] tests : drop SYCL special-casing in test-backend-ops.cpp ↗ ggml-org/llama.cpp
  7. [7] models: clean up some dead switch branches in old models (#28669) ↗ ggml-org/llama.cpp
  8. [8] opencl: add bin kernel `kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a8_bin` ↗ ggml-org/llama.cpp
  9. [9] [Docs][Security] Clarify reporter credit and CVE publication timing (#55476) ↗ vllm-project/vllm
  10. [10] [Bugfix] Avoid MistralCommonBackend for HF tokenizers ↗ vllm-project/vllm
  11. [11] [ROCm][Feature] Support KV connectors with ROCM_AITER_UNIFIED_ATTN ↗ vllm-project/vllm
  12. [12] [AMD] Enable Fast Triton Sparse MLA backend ↗ sgl-project/sglang
  13. [13] [JIT] Port the expert-pack MXFP4 kernels to load_jit and fix their launch limits ↗ sgl-project/sglang
  14. [14] [NPU] fix pp 2 hang on npu ↗ sgl-project/sglang

$ ls local-llm/reviews/ # the step-back reads

the week in review Sparse attention lands across vLLM, SGLang, llama.cpp the month in review Speculative decoding consolidates across the stack as DeepSeek V4 lands
Topics: Local LLMs AI / ML Full archive → Weekly reviews →

Quick answers

What shipped in Local LLMs on September 11, 2026?
Ollama lands a scope-based rewrite of its MLX memory bindings and prefix-cache eviction fixes that previously let long conversations grow memory without bound, while vLLM documents when it publishes CVEs. In total, 128 commits, 122 pull requests, and 10 releases landed.
Who contributed to Local LLMs on September 11, 2026?
15 developers shipped this update, including jessegross, dhiltgen, ggerganov, Iggy Jackson, shaofeiqi, Juan Pérez de Algaba, SalikovAlex, and juliendenize, and 7 more.
What were the notable Local LLMs updates?
mlx: scope array lifetimes instead of pinning and sweeping, mlxrunner: Evict prefix cache snapshots from the active conversation, and mlxrunner: keep the reused head of a cached edge safe from eviction.
♥

$ log Local LLMs --recent

Metal MoE/SSM fusion lands, MiMo-V2.5 fp8 sharding fixed 158 shipped Ollama exposes thinking levels, llama.cpp widens Hexagon and architecture coverage 250 shipped GGUF offset corruption fixed, vLLM restores Dynamo KV metadata 244 shipped MLX engine graduates out of x/, ROCm fixes silent attention corruption 268 shipped

$ log --all # elsewhere on the wire

@openclaw Cold-start Gateway diagnostics stop crying wolf, doctor repairs plugin drift Agentic Coding Cline desktop bundles shrink, goose unblocks its own clicks

$ ls showcases/

Rails Django Laravel Node.js Go Vercel Vue.js Kubernetes HashiCorp Arch Linux Linux Anthropic

$ init your-desk

RepoJournal can write one of these for any repo you care about. Sign in with GitHub, pick your repos and sources, and your desk gets a wire like this one every morning.