$ the-wire · showcase
vLLM 0.34.0 drops Model Runner V1, Ollama batches speculative decode
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
vLLM shipped 0.34.0 with extensible KV cache on by default and Model Runner V1 removed, while Ollama and llama.cpp spent the day making concurrent serving faster and more reproducible.
v0.34.0 vllm-project/vllm
The release carries 1,061 commits from 412 contributors, turns on an extensible KV cache sized from memory actually free after warmup (more cache and throughput for most models), and removes Model Runner V1 entirely, leaving V2 as the only GPU runner. Model Runner V1 is gone: anything pinned to GPUModelRunner V1 via the V0 path stops working on this version.
mlxrunner: batch speculative verification ollama/ollama
Ollama's mlxrunner used to disable drafting whenever several generation requests overlapped, so models with MTP, DFlash or DFlash2 lost their speculative speedup under concurrency. Drafting now stays on and proposed tokens are verified in batches, at the cost of more temporary GPU memory and sometimes a slightly later first token; per-request sampling and output constraints stay separate.
mlxrunner: reuse saved prefixes across request slots ollama/ollama
The runner already reused a conversation held by an idle slot, but a request arriving when that slot was busy reprocessed the same prompt from scratch. A new request can now reuse a compatible saved prefix from a busy slot, picking the longest complete saved prefix, so overlapping conversations with shared prompts start responding sooner.
vulkan: make the radix top-k tie selection deterministic ggml-org/llama.cpp
Radix top-k in the Vulkan backend (k >= 1024 and the fused TOPK_QSA path) broke ties with a shared atomic counter, so the kept columns changed from run to run when ties outnumbered open slots; it now follows the Metal kernel with each subgroup owning a contiguous chunk of the row. The nary and bitonic paths were already deterministic, and with this the Vulkan output matches the CUDA and Metal f...
[Perf][Muse Spark] FP8 KV cache for the specialized decode vllm-project/vllm
FP8 (E4M3) KV cache support lands for the Muse Spark latency_specialized decode on Blackwell, including the --kv-cache-dtype-skip-layers sliding_window option, measured on a 4x B300 TP4 setup with DSpark 7 speculative tokens at concurrency 1. vLLM reports TPOT down 6.0% versus the stock decode at 150k context, or 5.6% with SWA layers kept in BF16, with gsm8k unchanged in both cases.
ui: move the web UI to ui/apps/chat and call it Llama Chat (#32032) ggml-org/llama.cpp
Also landed: llama.cpp moved the web UI from tools/ui to ui/apps/chat and renamed it Llama Chat with no code change, a step toward splitting ui/lib from the apps built on it, so any CMake, Docker, Nix or workflow path pointing at tools/ui needs updating. Elsewhere, kv-cells replaced its per-removal O(n_cells) scan with a sorted position index, YuE2-3B support arrived in mtmd, and sglang fixed W...
Action items
- → Migrate off Model Runner V1 before upgrading vLLM to 0.34.0 vllm-project/vllm [plan]
- → Update tools/ui paths to ui/apps/chat in CMake, Docker, Nix and workflow files ggml-org/llama.cpp [plan]