$ the-wire · showcase
OLLAMA FIXES KV CACHE CORRUPTION, LLAMA.CPP HARDENS FILE SEARCH, VLLM KILLS MOE LEGACY CODE
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
14 people shipped this
jessegross
@jessegross
1 cited
Pascal
@ServeurpersoCom
1 cited
SilenNaihin
@SilenNaihin
1 cited
aoshen02
@aoshen02
1 cited
BBuf
@BBuf
1 cited
mmangkad
@mmangkad
1 cited
Daniel Hiltgen
@dhiltgen
Xuan-Son Nguyen
@ngxson
Niklas Wenzel
@nikwen
bnellnm
@bnellnm
Lin-z-w
@Lin-z-w
Tianyu Guo
@gty111
DevashishLal-CB
@DevashishLal-CB
hnyls2002
@hnyls2002
Ollama patched a critical cache corruption bug that corrupts lazy snapshots during multi-request prefills, while llama.cpp hardened Windows junction traversal and vLLM removed 14 hardcoded MoE shards across quantization methods.
Ollama's mlxrunner was silently overwriting lazy KV snapshots when later appends reached their buffer slots, corrupting cached conversations on request reuse [1]. The fix ensures snapshots are copied before any append that touches their range, not just the first one. In parallel, llama.cpp tightened file_glob_search to skip Windows junctions that std::filesystem reports as directories, preventing infinite loops when a junction points to an ancestor [2]. The patch also adds case-insensitive junk directory matching on Windows and surfaces incomplete results instead of silent failures. Over on vLLM, a major MoE refactor removes 14 quantization methods that hardcoded w13 shard counts to 2, where non-gated MoE (e.g. NemotronH) only fuses one shard [3]. Separately, vLLM fixed level-2 sleep/wake/reload for LoRA models by restructuring BaseLayerWithLoRA to preserve checkpoint semantics after memory discard [4]. SGLang pushed forward on diffusion kernels, preferring cuDNN SDPA over FlashAttention-4 on Blackwell for 1.81x speedups [5], while adding ModelOpt MXFP8 checkpoint support and unifying BaseFusedOp dispatch across backends [6]. Across 4 repos, 115 commits and 10 releases shipped this period.
Action items
- → Update Ollama immediately if you cache long conversations with short prefixes ollama/ollama [immediate]
- → Upgrade llama.cpp if serving from Windows directories with deep hierarchies ggml-org/llama.cpp [plan]
- → Patch vLLM MoE quantization methods if using non-gated MoE models vllm-project/vllm [plan]
- → Monitor SGLang B200 performance gains on your Blackwell hardware sgl-project/sglang [monitor]
References
- [1] mlxrunner: stop cache rewind refills from corrupting later lazy snapshots ↗ ollama/ollama
- [2] server: harden the file_glob_search directory walk ↗ ggml-org/llama.cpp
- [3] [Bugfix] Size and iterate w13 by shard count for non-gated MoE ↗ vllm-project/vllm
- [4] [Bugfix] Fix level-2 sleep/wake/reload with enable_lora=True ↗ vllm-project/vllm
- [5] [diffusion] Prefer cuDNN SDPA over FA4 for dense attention on sm_100 (B200) ↗ sgl-project/sglang
- [6] Support ModelOpt MXFP8 checkpoints ↗ sgl-project/sglang
