$ the-wire · showcase
llama.cpp swaps in a smarter expert cache, Ollama ships a leaner Qwen3.8 Flash Next
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
13 people shipped this
dhiltgen
@dhiltgen
1 cited
sanmai
@sanmai
1 cited
Juan Pérez de Algaba
@jperezdealgaba
1 cited
aoshen02
@aoshen02
1 cited
nvpohanh
@nvpohanh
1 cited
hoyyeva
@hoyyeva
sam-india-007
@sam-india-007
Masashi Yoshimura
@yomaytk
khluu
@khluu
divakar-amd
@divakar-amd
ShangmingCai
@ShangmingCai
Baizhou Zhang
@Fridge003
metamergebot
@metamergebot
Two MoE memory wins landed the same day: llama.cpp replaces its SLRU expert cache with a classed, decaying-usage replacement that measures faster decode, and Ollama reworks Qwen3.8 Flash Next off blanket NVFP4 so long generations stop degrading.
moe-cache : LRFU/MQ-like hybrid cache, ClassLRU ggml-org/llama.cpp
ClassLRU keeps frequently used experts in classes ranked by decaying usage history, replacing SLRU and enabling the auto mode for llama_moe_cache. The author measured +6% decode on QFN and +7% on GLM versus SLRU with a warm cache, plus 8% fewer expert uploads.
MLX: make Qwen3.8 Flash Next memory efficient ollama/ollama
Blanket NVFP4 quantization was hurting long generations, so QSA projections and the non-expert MTP path stay in BF16 while other eligible dense and control weights move to MXFP8. The 26.8 GiB NVFP4 PLE table is now memory-mapped from the safetensor shards, with only the selected hashed rows pulled into MLX for dequantization; qwen3.8-flash-next:125b-a6b-nvfp4 and qwen3.8-flash-next:125b-mlx bot...
[Sleep Mode] Level 2 discards only parameters vllm-project/vllm
Level 2 was discarding everything allocated during model construction because the weights pool wraps it, and buffers had to be rescued by name one at a time. Now sleep(level=2) states what the caller rewrites and keeps the rest at the same address, narrowing the discard to parameters.
[Perf] kimi-k3.1: TP-shard the decode MoE front and latent-up tail sgl-project/sglang
On main every TP rank of a Kimi-K3.1 MoE layer read the full 12.9 MB router and 51.4 MB latent down-proj rows plus the 51.4 MB latent up_proj; this port shards both so each rank reads only its own row block. It matters where DFLASH verify batches are bound by weight reads.
[Security] Raise patched runtime and test dependency floors (#63574) vllm-project/vllm
The long tail: vLLM raised its runtime and test dependency floors for patched versions, llama.cpp moved CUDA 13 release builds to 13.5 (dropping GGML_CUDA_CCCL_VERSION, since the 13.5 rows ship CCCL 3.5.0), Ollama's launch path no longer lets inherited provider flags route Claude Code away from the selected model and surfaces HTTP 410 retirement messages, sgl-router now logs per-request image, ...
