$ the-wire · showcase
Same-host NIXL falls back to TCP on the extensible KV cache
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
11 people shipped this
vLLM's default extensible KV cache has been silently routing same-host NIXL prefill/decode transfers over TCP instead of NVLink since it landed, and the day's other fixes are quieter: determinism in llama.cpp's top-k, ROCm VMM undercounting, and CRIU restore crashes in sglang metrics.
[Bugfix][KV Connector] Keep same-host NIXL on NVLink with the extensible KV cache vllm-project/vllm
Since the extensible KV cache became the default, same-host NIXL P/D moved KV over TCP rather than NVLink; per-transfer time went from 9.9 s to about 24 ms once the fix restores the FABRIC handle with an IMEX channel. Without NIXL on device memory nothing changes, and without an IMEX channel it stays off by default.
[Bugfix][ROCm] Size the extensible KV cache from exact VRAM use, not 2 MB-rounded free memory vllm-project/vllm
hipMemGetInfo rounds every allocation up to 2 MB, so on ROCm the extensible KV cache sized itself from free memory that looked a few GB smaller than it was. The fix adds that memory back once at final sizing using amdgpu's exact VRAM count; only unpartitioned discrete ROCm GPUs are affected.
[Fix] metrics: Handle process CPU clock resets after restore sgl-project/sglang
After CRIU restore the process CPU clock can sit below the scheduler's saved baseline, and the negative delta made the Prometheus counter raise ValueError, killing the restored scheduler with metrics enabled. Without a time namespace, restoring on a host with less uptime also pushes the monotonic clock backwards until accounting catches up.
ggml : update CUDA/Metal radix Top-K to make tied selections deterministic (#31084) ggml-org/llama.cpp
Radix Top-K on CUDA now keys on an ordered 64-bit pair, so tied selections resolve deterministically; the author's own commits describe it as a fix for a thread timing bug. Metal, ROCm and Vulkan are excluded from the change, so determinism here is CUDA-only for now.
[Transformers backend] Migrate Gemma and StableLM vllm-project/vllm
Gemma (v1) and StableLM now run on the Transformers modeling backend, with gemma.py and stablelm.py deleted; the PR claims accuracy and throughput matching native. PaliGemma v1 keeps working because its Gemma text config sets use_bidirectional_attention, which the backend now checks before picking EncoderOnlyAttention.
