$ the-wire · showcase
llama.cpp 0.7.0 ships MoE host cache, ollama reaps orphaned Windows runners
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4)
21 people shipped this
dhiltgen
@dhiltgen
1 cited
github-actions[bot]
@github-actions[bot]
1 cited
Juntian777
@Juntian777
1 cited
stecasta
@stecasta
1 cited
shodoco
@shodoco
1 cited
Jeffrey Morgan
@jmorganca
jberg5
@jberg5
Max Krasnyansky
@max-krasnyansky
ngxson
@ngxson
arthw
@arthw
FINAL-Bench
@final-bench
WoosukKwon
@WoosukKwon
omuhamma
@omuhamma
njhill
@njhill
ckvermaAI
@ckvermaAI
Jiminator
@Jiminator
avjves
@avjves
metamergebot
@metamergebot
nav
@navthings
Haojin Yang
@yanghaojin
Anastasiia Filippova
@nastya236
llama.cpp's v0.7.0 release lands a host-memory expert cache that roughly doubles decode speed when experts are offloaded, while ollama closes a Windows subprocess leak and vLLM trims Python overhead off every long sliding-window request.
v0.7.0 ggml-org/llama.cpp
A new GPU cache for MoE experts kept in host memory, exposed as --moe-cache-mib, is the headline perf change: up to ~2.2x decode speedup when experts are offloaded. The release also adds LiliCorr speculative decoding, MTP for GLM5-Next, and support for K2 Horizon, Prism Bonsai 2 27B, EmbeddingGemma-2, LiquidAI d1/d1-omni and vLLM decision 2/d3 decision, pplx-decider, MiniCPM-V 4.7, Nemotron Sup...
win: prevent orphaned runners ollama/ollama
Runner processes now belong to a singleton Windows Job Object configured with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, so Windows terminates owned llama-server and MLX subprocesses when Ollama exits, including forced termination where graceful cleanup cannot run. The patch notes desktop users are unaffected because the app already shuts the server down gracefully; the gap was for people running olla...
[Perf][Core] Skip the null head of sliding-window block tables vllm-project/vllm
A sliding-window KV group keeps one null block per out-of-window block, so a 128K-token prompt with a 128-token window and 16-token blocks leaves 8,183 of 8,192 entries null, and vLLM walked every one of them in Python inside KVCacheBlocks.get_unhashed_block_ids_all_groups and SingleTypeKVCacheManager.free. This change skips that null head, finding the leading null run by binary search and chec...
[MRV2][Spec Decode] Quantized draft lm_head for drafters that share the target head vllm-project/vllm
An opt-in key gives an MTP or EAGLE drafter a private quantized copy of the lm_head instead of re-reading the shared target head on every draft token. With an NVFP4 draft head on Qwen3.8-Flash-Next-NVFP4 + MTP (k=3), decode at c=1 on SPEED-Bench goes from 39.4 to 46.5 tok/s, +18% with average AL unchanged at -0.4%, measured on three nodes on one DGX Spark with the SM121 skinny-GEMM plans.
[Fix] sglang-server: Keep the matched stop token in output_ids sgl-project/sglang
The Rust detokenizer popped the matched stop token from the chunk's token IDs before decoding, dropping it from both the decoded text and output_ids in /generate and gRPC responses, while the Python detokenizer trims it only from the IDs it decodes and passes output_ids through unchanged. The two servers returned different token IDs for the same request, which breaks clients that count or repla...
