$ the-wire · showcase
LLAMA.CPP SHIPS BAILINGMOE3 SPEC-DECODING SUPPORT
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
14 people shipped this
Toby
@aetherbird
1 cited
王金旭
@wjinxu
1 cited
ZJY0516
@ZJY0516
1 cited
tlrmchlsmth
@tlrmchlsmth
1 cited
mmangkad
@mmangkad
1 cited
Xuan-Son Nguyen
@ngxson
Pascal
@ServeurpersoCom
Eve
@netrunnereve
030611
@030611
mgoin
@mgoin
LopezCastroRoberto
@LopezCastroRoberto
zhihengy
@zhihengy
Lianmin Zheng
@merrymercy
ch-wan
@ch-wan
llama.cpp lands full BailingMoE3 support with speculative decoding, giving developers a native path for a key emerging sparse MoE family.
llama.cpp's new BailingMoE3 backend [1] is the story to beat this cycle: full weight loading, speculative decoding, Q-LoRA for the Ling-3.0-tiny variant, and a fix for SwiGLU clamps. If you serve sparse experts, this is the upgrade. Right behind it, DSpark checkpoints in the speculators format now load directly in llama.cpp [2], which pairs with vLLM's new RecoverSSM path for Kimi-K3 [3] to make speculative decode on MoE targets dramatically cheaper. Meanwhile, vLLM turns one of its most battle-tested env knobs into a first-class argument: prefix_cache_retention_interval is now a CLI flag, and its default drops to 0 [4], which the team says is "an extremely important setting especially for agentic workloads." That one change could reshape how you size prefix caches for long-horizon agents. sglang is quieter but ships a meaningful accuracy win: with DSA skip-topk layer handling [5], a GLM-5.2-NVFP4 run hits 90.83% pass@1 on AIME25, so if you're chasing reasoning benchmarks, that's the config to copy. With 116 commits and 2 releases across the stack, the headline is clear: MoE speculative decoding just got a whole lot more practical.
Action items
- → Review BailingMoE3 support in llama.cpp and test against your sparse-MoE checkpoints ggml-org/llama.cpp [plan]
- → Migrate VLLM_PREFIX_CACHE_RETENTION_INTERVAL to the new --prefix-cache-retention-interval argument vllm-project/vllm [immediate]
- → Benchmark vLLM RecoverSSM on Kimi-K3 spec decoding to measure state-materialization savings vllm-project/vllm [monitor]
References
- [1] model : BailingMoE3 Support (#26608) ggml-org/llama.cpp ↗
- [2] model: support speculators-format checkpoints for DSpark (#26275) ggml-org/llama.cpp ↗
- [3] [K3] support recoverssm for K3 ↗ vllm-project/vllm
- [4] Promote `prefix_cache_retention_interval` to an argument and change the default to 0 ↗ vllm-project/vllm
- [5] [DSA] Skip indexer KV cache for skip-topk layers ↗ sgl-project/sglang
