$ the-wire · showcase
Ollama enables speculative decoding under structured output, vLLM adds LoRA and fusion
By RepoJournal · Filed · About Local LLMs
17 people shipped this
jessegross
@jessegross
1 cited
dhiltgen
@dhiltgen
1 cited
mgoin
@mgoin
1 cited
BugenZhao
@BugenZhao
1 cited
wseaton
@wseaton
1 cited
mickqian
@mickqian
1 cited
Jiminator
@Jiminator
1 cited
kediwu0331
@kediwu0331
1 cited
ngxson
@ngxson
tboinovski1
@tboinovski1
max-krasnyansky
@max-krasnyansky
wanghqc
@wanghqc
Daniel Bevenius
@danbev
Fangzhou-Ai
@Fangzhou-Ai
AndreasKaratzas
@AndreasKaratzas
wplf
@wplf
hzh0425
@hzh0425
Ollama's MLX runner now runs speculative decoding under structured output constraints, promising near-full draft throughput on MLX models, while vLLM lands static LoRA loading in its Rust frontend and begins manual activation-quant fusion.
Ollama unblocked speculative decoding for structured-output requests on MLX. Previously, a structured-output request decoded one token at a time, roughly halving throughput on a dense 27B MTP model. [1] The runner now enforces the grammar during verification: each draft position's logits are masked before rejection sampling, so an invalid draft is rejected and every emitted token obeys the grammar. Drafts stay unconstrained, preserving pipelined forwards. [1] Separately, the MLX CI now rebuilds stale test payloads when the release dylib lacks symbols or MLX pins drift, preventing silent test skips. [2]
vLLM is fleshing out its Rust frontend and manual fusion work. The Rust frontend now supports `--lora-modules` for static adapter loading, matching Python's `name=path` or JSON forms, with startup failing on any load error. [3] The first `ActivationQuantFusionPass` application adds `maybe_fused_act_quant`, which emits a `QuantizedActivation` via the fused `silu_and_mul_quant` kernel when the linear advertises a consumable input key, otherwise falling back to plain activation. [4] For the reasoning parser, token-attributed text from the incremental detokenizer is now threaded through, but no emitted text changes. [5]
SGLang's diffusion support fixes several interoperability gaps. The diffuser now completes exact files from partially cached Hugging Face snapshots, keeps LoRA dynamic when quantization owns the base weight, and accepts MiniMax-H3 hybrid filenames and native VAE overrides. [6] A new cookbook recipe moves DeepSeek-V4 DGX Spark to a v2 image enabling NVFP4 and FP4 multi-node configs. [7] Cosmos3 FP8 video quality improves by parsing the diffusion step policy from modelopt checkpoints and using W8A16 for specified steps. [8]
Action items
- → Upgrade ollama to include the MLX speculative decoding under structured output fix to recover draft-head throughput on MLX models. ollama/ollama [plan]
- → If you run vLLM's Rust frontend with static adapters, upgrade to the next release containing the `--lora-modules` support. vllm-project/vllm [plan]
- → Monitor the manual fusion migration in vLLM if you depend on quantized activation kernels. vllm-project/vllm [monitor]
- → Review SGLang's third-party component bundle fixes if you use diffusion or MiniMax-H3 components. sgl-project/sglang [plan]
References
- [1] mlxrunner: enable speculative decoding under structured output ↗ ollama/ollama
- [2] ci: rebuild MLX macOS test payloads the release can't supply ↗ ollama/ollama
- [3] [Rust Frontend] Support `--lora-modules` for static adapter loading ↗ vllm-project/vllm
- [4] [Fusion] Manual `ActivationQuantFusionPass` initial application ↗ vllm-project/vllm
- [5] [Rust Frontend] Use token-attributed text in reasoning and unified parsers ↗ vllm-project/vllm
- [6] [diffusion] Compose third-party component bundles safely ↗ sgl-project/sglang
- [7] [Cookbook] DeepSeek-V4 DGX Spark: v2 image + Flash Official NVFP4 and Flash Vision FP4 cells ↗ sgl-project/sglang
- [8] Cosmos3 fp8 mixed precision ↗ sgl-project/sglang
