$ the-wire · showcase
QWEN3.5 SPECULATIVE DECODING HITS 70% SPEEDUP ON LLAMA.CPP, OLLAMA ROUTES CLOUD MODELS
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
16 people shipped this
drifkin
@drifkin
1 cited
jessegross
@jessegross
1 cited
fairydreaming
@fairydreaming
1 cited
MatthewBonanni
@MatthewBonanni
1 cited
ch-wan
@ch-wan
1 cited
nanjiangwill
@nanjiangwill
1 cited
dhiltgen
@dhiltgen
arthw
@arthw
johnkarlhill
@johnkarlhill
Masato Nakasaka
@rillomas
fxmarty-amd
@fxmarty-amd
yewentao256
@yewentao256
NickLucche
@NickLucche
kpham-sgl
@kpham-sgl
mickqian
@mickqian
luoroger37
@luoroger37
Qwen3.5's MTP head is now live across both Ollama and llama.cpp, delivering 24-70% inference speedup on dense models while Ollama adds smart cloud-model routing to end user confusion.
The speculative decoding win spans two major inference engines. Ollama loaded the MTP head for qwen3.5 and wired it as an automatic draft model in the MLX engine [1], pushing the 35B MoE from 131 to 162 tokens/s on code (24% faster) and the dense 27B from 31 to 53 tokens/s (70% speedup). Llama.cpp followed with its own MTP tensor loading [3], but added a load_mtp flag to skip MTP overhead when speculative decoding isn't requested. In parallel, Ollama shipped a UX fix [2] that probes for :cloud variants when a model pull fails, suggesting kimi-k3:cloud to users at the terminal instead of leaving them with a cryptic file-not-found error. On the vLLM side, the team landed a block-table alignment fix [4] that unifies width calculation across three indexer paths, preventing buffer mismatches under deep context packing. Across 4 repos, the stack logged 110 commits, 104 PRs, and 10 releases in 24 hours. SGLang is now routing Kimi Linear through paged MLA backends [5] and shipping fastsafetensors without GDS for hosts lacking the NVIDIA driver [6].
Action items
- → Pull Ollama latest and test qwen3.5 inference on M-series; measure your token/s before and after ollama/ollama [monitor]
- → Rebuild llama.cpp with MTP support and verify --spec-type draft-mtp on Qwen models in your pipeline ggml-org/llama.cpp [plan]
- → If running vLLM with deep context packing, upgrade to pick up block-table alignment fix immediately vllm-project/vllm [immediate]
- → SGLang users: test fastsafetensors no-GDS loading if your host lacks NVIDIA GPUDirect Storage driver sgl-project/sglang [monitor]
References
- [1] mlx: qwen3.5 MTP speculative decoding ↗ ollama/ollama
- [2] cmd: suggest :cloud when a model has no default tag ↗ ollama/ollama
- [3] llama : load MTP tensors only if they are really used ↗ ggml-org/llama.cpp
- [4] [Bugfix] Universally align block table width to 128 tokens ↗ vllm-project/vllm
- [5] [unified-memory] Let Kimi-Linear use the paged MLA attention backends ↗ sgl-project/sglang
- [6] Support fastsafetensors no-GDS loading and page-cache release ↗ sgl-project/sglang
