Speculative decoding consolidates across the stack as DeepSeek V4 lands

♥

August 2026 saw the local LLM stack converge on speculative decoding and DeepSeek V4, with every major engine shipping optimized paths.

3071 commits 3041 PRs merged 252 releases 30 briefings covered

local-llm wire →

The month's dominant thread was speculative decoding becoming a standard optimization across engines. Ollama and llama.cpp both shipped Multi-Token Prediction (MTP) speculative decoding for Qwen3.5, with llama.cpp loading MTP tensors only when actually used [1][2]. Ollama reported a 70% speedup on llama.cpp [1], while DeepSeek V4 inference got a 50% speedup across llama.cpp and vLLM [3][4]. SGLang shipped v0.5.18 with 710 PRs and DeepSeek V4 FP4 auto-routing , and vLLM 0.28.0 landed with 584 commits and a Kimi K3 performance push [5]. The pattern is clear: speculative decoding is no longer experimental, it is the baseline expectation for new model support.

DeepSeek V4 was the other through-line, with engines racing to optimize its inference. vLLM implemented sequence parallelism for DeepSeek V4 [4], llama.cpp added MTP and DSpark support plus a converter option for separate dspark GGUFs [3][6], and the Metal backend implemented DeepSeek V4 hyper-connections [10]. SGLang added session-reference-aware unified radix cache for agentic multi-turn workloads [8], a response to real usage patterns that pairs with the Kimi K3 day-0 support earlier in the month [5].

Correctness and stability fixes mattered as much as new features. vLLM universally aligned block table width to 128 tokens [9], fixing a class of bugs that could corrupt generation. Ollama fixed a deadlock that hung chat and generate [10] (date), and patched a critical SSRF vulnerability in manifest handling [11]. llama.cpp hardened its Vulkan stack and later fixed Vulkan memory bloat [12]. For production users, these fixes are the difference between a demo and a reliable service.

The month closed with a wave of platform-specific optimizations. llama.cpp shipped Vulkan top-k radix sort and HIP Q2_0 optimizations , while vLLM landed a CUDA-graph memory fix and deprecated PyAV [13]. Ollama's MLX runner gained structured output support , and the OpenVINO backend landed Qwen3.5 on NPU with Whisper.cpp support . Each engine is carving out its niche on specialized hardware, but the shared priorities of speculative decoding, memory efficiency, and correctness suggest the next several months will see these optimizations fold into each other as the ecosystem consolidates.

References

[1] mlx: qwen3.5 MTP speculative decoding ↗ ollama/ollama
[2] llama : load MTP tensors only if they are really used ↗ ggml-org/llama.cpp
[3] DeepseekV4 MTP + DSpark ↗ ggml-org/llama.cpp
[4] [DSV4] Implement Sequence Parallelism ↗ vllm-project/vllm
[5] [Kimi K3] Add reasoning, tool-call, and OpenAI serving support ↗ sgl-project/sglang
[6] convert: add option to create separate dspark GGUF ↗ ggml-org/llama.cpp
[7] metal: implement DeepSeek V4 hyper-connections ↗ ggml-org/llama.cpp
[8] feat: Session-reference-aware Unified Radix Cache for agentic multi-turn workloads ↗ sgl-project/sglang
[9] [Bugfix] Universally align block table width to 128 tokens ↗ vllm-project/vllm
[10] metal: implement DeepSeek V4 hyper-connections ↗ ggml-org/llama.cpp
[11] [ROCm][MLA] Mask the AITER MLA small-head verify flatten causally ↗ vllm-project/vllm
[12] b10223 ↗ ggml-org/llama.cpp
[13] [1/N] Unify multiple-path encoder cuda graph support ↗ vllm-project/vllm

$ ls local-llm/month/ # the briefings behind this review

Sat Aug 1 QWEN3.5 SPECULATIVE DECODING HITS 70% SPEEDUP ON LLAMA.CPP, OLLAMA ROUTES CLOUD MODELS Sun Aug 2 LLAMA.CPP SHIPS FOUR BUILDS IN 24 HOURS AS VLLM AND SGLANG RACE ON SEQUENCE PARALLELISM Mon Aug 3 DEEPSEEK V4 INFERENCE GETS 50% SPEEDUP; LLAMA.CPP AND VLLM BOTH SHIPPING OPTIMIZED PATHS Tue Aug 4 OLLAMA REWIRES OPENAI COMPATIBILITY, LLAMA.CPP HARDENS SAMPLING Wed Aug 5 OLLAMA FIXES RACE CONDITIONS AS LLAMA.CPP SHIPS BREAKING TTS CHANGES Thu Aug 6 OLLAMA FIXES KV CACHE CORRUPTION, LLAMA.CPP HARDENS FILE SEARCH, VLLM KILLS MOE LEGACY CODE Fri Aug 7 LLAMA.CPP HARDENS VULKAN STACK, VLLM CUTS ATTENTION LATENCY, SGLANG SCALES MLA Sat Aug 8 OLLAMA SHIPS BLOCK-DIFFUSION SPECULATIVE DECODING, SGLANG LAUNCHES KIMI K3 DAY-0 Sun Aug 9 OLLAMA CUTS MODEL CONFIG BLOAT 100KB+ PER MODEL, LLAMA.CPP SHIPS TOOL ISOLATION AND CUDA FUSION GAINS Mon Aug 10 OLLAMA SHIPS VISION SUPPORT ACROSS MLX AND QWEN3.5 AS LLAMA.CPP FIXES DISPATCH BUGS Tue Aug 11 OLLAMA PATCHES CRITICAL PARSER BUG, NEMOTRON 3 LANDS WITH SPECULATIVE DECODING Wed Aug 12 OLLAMA CUTS REPEAT PENALTY BLOAT, LLAMA.CPP SHIPS POCKET-TTS SUPPORT Thu Aug 13 OLLAMA PATCHES CRITICAL SSRF VULNERABILITY IN MANIFEST HANDLING Fri Aug 14 OLLAMA LAUNCHES DEEPSEEK AND MUSE INTEGRATIONS WHILE VLLM SHIPS HARDWARE-AGNOSTIC MODELING Sat Aug 15 QWEN3.8 LANDS IN OLLAMA WITH DEVELOPER INSTRUCTION SUPPORT Sun Aug 16 LLAMA.CPP OVERHAULS INFERENCE FLAGS, VLLM PATCHES DEEPSEEK SPARSE MLA Mon Aug 17 LLAMA.CPP SHIPS MODEL OPTIMIZATIONS WHILE VLLM AND SGLANG CHASE MEMORY AND SCHEDULING GAINS Tue Aug 18 LLAMA.CPP SHIPS BAILINGMOE3 SPEC-DECODING SUPPORT Thu Aug 20 OLLAMA FIXES DEADLOCK THAT HUNG CHAT AND GENERATE Fri Aug 21 DeepSeek V4 fuses shared experts, as vLLM ships a security guard Sat Aug 22 SGLang ships v0.5.18 with 710 PRs and DeepSeek V4 FP4 auto-routing Sun Aug 23 SGLang Fixes Pixtral 500s as vLLM Cuts TTFT 25% Mon Aug 24 VLLM'S P2P WEIGHT SYNC CUTS RL MOE TRANSFER COSTS Tue Aug 25 OLLAMA BOLSTERS CLAUDE DESKTOP, VLLM HARDENS AUDIO PATHS Wed Aug 26 OLLAMA'S MLX RUNNER NOW SPEAKS STRUCTURED OUTPUT Thu Aug 27 vLLM 0.28.0 LANDED WITH 584 COMMITS AND A KIMI-K3 PERFORMANCE PUSH Fri Aug 28 LLAMA.CPP UNLEASHES QWEN3.8-FLASH-NEXT SUPPORT Sat Aug 29 OpenVINO backend lands Qwen3.5 on NPU, Whisper.cpp support Sun Aug 30 llama.cpp Vulkan memory bloat fixed, vLLM deprecates PyAV Mon Aug 31 Vulkan top-k radix sort, HIP Q2_0 optimizations, and vLLM CUDA-graph memory fix

local-llm wire →