RepoJournal

$ cat local-llm/week/2026-11-09.log

Local LLMs

Local LLMs

the week in review · Nov 9 – Nov 15, 2026

vLLM restores NVLink for same-host NIXL KV transfer

By RepoJournal · composed from the cited sources · human-reviewed weekly · methodology

♥

The extensible KV cache had been silently routing same-host prefill-to-decode traffic over TCP since it became the default; ROCm sizing and sglang pause bugs got fixes too.

900 commits 898 PRs merged 69 releases 2 security advisories 7 briefings covered

all local-llm reviews →

[Bugfix][KV Connector] Keep same-host NIXL on NVLink with the extensible KV cache vllm-project/vllm

by gcanlin

Since the extensible KV cache became the default, same-host NIXL prefill/decode pairs were quietly moving KV over TCP rather than NVLink. With an IMEX channel the cache now gets a FABRIC handle; without one it stays off for NIXL by default, so check whether your host actually exposes IMEX before assuming you got the bandwidth back.

ggml : update CUDA/Metal radix Top-K to make tied selections deterministic (#31084) ggml-org/llama.cpp

by Alexey Kopytko

CUDA and Metal radix Top-K now key on an ordered 64-bit pair with a warp-contiguous layout, so selections with tied scores resolve identically run to run. The rewrite also fixes a thread-timing bug in the old path, which matters if you sample once and replay the seed.

Add FP8 KV-cache support ggml-org/llama.cpp

by ORippler

OCP-FP8 KV caches land in llama.cpp with optional run-time, static per-tensor scales, using graph-level scaling. Models with non-unit scales, such as the NVFP4 Mistral-Medium-3.5-128B example the PR cites, are the ones to test first.

mlx: support concurrent generation ollama/ollama

by dhiltgen

MLX models in Ollama can serve concurrent generations via OLLAMA_NUM_PARALLEL; the default stays at one request and each extra slot needs its own memory. Ready requests share projection and feed-forward work, which is where the first-token wait comes down.

RPC: Improve loading times ggml-org/llama.cpp

by am17an

On two DGX Spark machines over RDMA with a cold page cache, loading DeepSeek-V4-Flash (145.6 GiB) got 5x faster in load_tensors. Multi-node RPC startup is the case this targets.

ggml-rpc : fail gracefully on unreachable endpoints and failed handshakes (#30415) ggml-org/llama.cpp

by figkcoci-debug

rpc_dispatcher::start() used to abort the whole process on an unparseable endpoint, a failed transport, or a failed handshake. It now fails gracefully, so one bad node no longer takes down the run.

[Bugfix][ROCm] Size the extensible KV cache from exact VRAM use, not 2 MB-rounded free memory vllm-project/vllm

by GodRishUniverse

hipMemGetInfo rounds every allocation up to 2 MB, so after warmup the extensible KV cache sized itself from free memory that looked a few GB smaller than it was. The final sizing now adds that back once from amdgpu's exact VRAM count; partial allocations still round.

[Fix] metrics: Handle process CPU clock resets after restore sgl-project/sglang

by rst0git

After a CRIU restore the process CPU clock can sit below the scheduler's saved baseline, and the resulting negative delta made the Prometheus counter raise ValueError and kill the restored scheduler. Now metrics can stay enabled across restore. Alongside: sglang retracts every KV holder on retract-mode pause, Metal rejects MUL_MAT for src1 precisions it cannot honor, and Ollama stopped dropping...

$ ls local-llm/week/ # the briefings behind this review

Keep up with Local LLMs in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.

or

fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.

Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.

all local-llm reviews →