$ the-wire · showcase
Deterministic top-p under TP, KV constants stay on the GPU
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
22 people shipped this
angt
@angt
1 cited
anton-iermolenko
@anton-iermolenko
1 cited
njhill
@njhill
1 cited
spandantiwari
@spandantiwari
1 cited
gilfordting
@gilfordting
1 cited
STRML
@STRML
1 cited
Lucius
@luciusacarr
1 cited
hoyyeva
@hoyyeva
ParthSareen
@ParthSareen
cabelo
@cabelo
danbev
@danbev
shaofeiqi
@shaofeiqi
Dao007forever
@Dao007forever
akii96
@akii96
hshrivastava-droid
@hshrivastava-droid
axx-ty911
@axx-ty911
hnyls2002
@hnyls2002
yuan-luo
@yuan-luo
vincentbryan
@vincentbryan
MarcosAsh
@MarcosAsh
ipazarbasi
@ipazarbasi
ArupKantiDas
@ArupKantiDas
SGLang pins down a cross-rank sampling divergence that hangs NCCL, llama.cpp stops shipping rotation matrices across the PCIe bus, and vLLM fixes multi-module MTP drafting for the Qwen3.5 family.
[Fix] sampling: deterministic top-p / top-k renorm when tp_size > 1 (TP rank divergence in DFlash/DSpark) sgl-project/sglang
flashinfer's default top-p/top-k renorm kernels use float atomicAdd, so TP ranks renormalizing identical logits can land on different coins, commit divergent KV/radix state, and hang in NCCL collectives. This makes the renorm deterministic so ranks agree before sampling; the report calls the divergence out for DFlash/DSpark specifically.
kv-cache : keep rotation constants on the GPU ggml-org/llama.cpp
The Haramard static matrices now stay resident on the GPU instead of being re-transferred from the host each step. Measured against master at edc353842c1b on Linux with an RTX 3090 and Qwen3.5 9B Q4_K_M, Q8_0 K/V and flash attention, where the author says host-to-GPU trips were the bottleneck on his PCIe 2.0 link.
[Bugfix] Fix multi-layer MTP module selection for Qwen3.5 and Qwen3-Next vllm-project/vllm
Qwen3_5MTP, Qwen3_5MoeMTP (also used by Qwen3.6, Qwen3.8 and Intern-S2 checkpoints) and Qwen3NextMTP ran MTP module 0 for every draft step whenever a checkpoint carried more than one module. Each draft step now uses the module trained for it on Model Runner V2, including the default compiled configuration.
[Config] Move V2-only checks into V1 model runner unsupported features vllm-project/vllm
HiSparse, custom speculative methods and watermarking were special-cased in use_v2_model_runner; they are now listed in _get_v1_model_runner_unsupported_features() and rejected by _validate_v1_model_runner(). Behavior change: watermarking with VLLM_USE_V2_MODEL_RUNNER=0 raises instead of silently forcing V2.
cmake : embed an app manifest in all Windows exes ggml-org/llama.cpp
llama.cpp embeds a manifest selecting UTF-8 as the Windows process code page and opting into long-path support, replacing reliance on u8path() conversions that compiled clean but broke Unicode paths. Legacy child-process output decoding is preserved.
Cache generated source code and hash in custom kernels ml-explore/mlx
CustomKernel takes an optional precomputed source hash rather than hashing the full source on every dispatch, a path the author found consuming 14% of host thread time in a Time Profiler capture on M5 Ultra. Graph build drops from 1.42 to 1.21 ms per round.
[Metal] Fix fill loop in `layer_norm` (#4772) ml-explore/mlx
Elsewhere around the stack: MLX fixed the fill loop in Metal layer_norm, llama.cpp is moving all Apple CI jobs except macos-latest-arm64 into a daily workflow to cut runner load, and Ollama now leaves a Retry disconnect action in place when ChatGPT cleanup fails.
Action items
- → Pin SGLang past the deterministic top-p/top-k renorm fix if you serve DFlash or DSpark with tp_size > 1 sgl-project/sglang [plan]
- → Drop VLLM_USE_V2_MODEL_RUNNER=0 from watermarking deployments in vLLM; it now raises vllm-project/vllm [plan]
