RepoJournal
Local LLMs Local LLMs
47 wires and counting

$ follow Local LLMs

Keep up with Local LLMs in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.

or

fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.

Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.

$ status

wire 2026-09-08
stories 242
RSS Archive Topics

© 2026 RepoJournal Home Showcase Explore How it works Privacy

$ the-wire · showcase

SGLang fixes TP hangs and GLM-5.2 routing

By RepoJournal · Filed · About Local LLMs

DevVexus Jeff Bolz Pranesh Gonegandla Kevin Hopper Woosuk Kwon AndreasKaratzas gau-nernst
15 people shipped this
DevVexus @devvexus 1 cited Jeff Bolz @jeffbolznv 1 cited Pranesh Gonegandla @praneshgo 1 cited Kevin Hopper @kh0pper 1 cited Woosuk Kwon @WoosukKwon 1 cited AndreasKaratzas @AndreasKaratzas 1 cited gau-nernst @gau-nernst 1 cited kamb-code @kamb-code 1 cited franciscojavierarceo @franciscojavierarceo 1 cited sagearc @sagearc 1 cited xiaobochen-amd @xiaobochen-amd 1 cited nogumanov @nogumanov 1 cited hnyls2002 @hnyls2002 1 cited Mohammad Miadh Angkad @mmangkad 1 cited Zhaolun Yin @ZhaolunYin

SGLang patches two correctness bugs that caused hangs and wrong expert selection, while llama.cpp and vLLM ship backend improvements.

SGLang fixed a rank-consistency bug in request-timeout aborts that could hang TP servers [1]. Timeout enforcement previously ran independently on each TP rank, so a request at the timeout boundary could be aborted on one rank before another, diverging the waiting-queue composition and leading to mismatched collectives. SGLang also reverted a fused silu+mul+FP8-quant kernel that was "not numerically equivalent to the two-kernel path it replaces" [2], which was hurting speculative-decoding acceptance on GLM-5.2-FP8 EAGLE MTP and failing a test asserting `accept_length > 4.0`.

On AMD, SGLang keeps GLM-5.2's MoE `e_score_correction_bias` in fp32 instead of casting it to bf16 [3]. The bias values cluster around 7 with a spread of 0.246, and bf16 can only represent about 8 distinct values near 7, collapsing 238 distinct biases and picking wrong experts. Additionally, SGLang dropped its vendored dense BF16 GEMM port in favor of FlashInfer 0.6.18 [4] and fixed p/d bootstrap across DP listeners, which previously returned `503 Prefill server not fully registered` when a request used a local listener without topology [5].

llama.cpp's Vulkan backend added the "DeepSeek-V4 hyper-connection fused ops (DSV4_HC_COMB/PRE/POST)" [6], closing the last major backend gap; on DeepSeek-V4-Flash the unfused chain took about 32% of decode op time on gfx1151. Vulkan also now falls back to CPU for GET_ROWS with misaligned offsets, previously a hard crash on Qwen3-TTS and Qwen3-VL [7]. Additionally, llama.cpp writes explicit `recurrent_layers` for Qwen3-Next/Qwen3.5, fixing silent wrong-layer reconstruction for non-uniform layer types [8], and CUDA Q4_K/Q5_K unpack is now branchless to speed up mmvq [9].

vLLM added a stateless `POST /v1/responses/render` endpoint for GPU-less preprocessing in disaggregated serving [10], fixed OffloadingConnector zeroing offload hits under MTP/EAGLE spec decode [11], added FP8 support for QSA indexer with up to 1.3x decode speedup on GB300 [12], and removed an obsolete TPU Dockerfile [13]. ROCm AITER sparse MLA now supports attention sinks introduced by HY-V4 [14].

Action items

References

  1. [1] [Scheduler] Make request-timeout aborts rank-consistent to fix TP collective hangs ↗ sgl-project/sglang
  2. [2] Revert "[kernel] add fused silu mul quant fp8" ↗ sgl-project/sglang
  3. [3] [AMD][GLM-5.2] Keep GlmMoeDsa MoE e_score_correction_bias in fp32 ↗ sgl-project/sglang
  4. [4] [Kernel] Drop the vendored dense BF16 GEMM port in favor of FlashInfer 0.6.18 (#38124) ↗ sgl-project/sglang
  5. [5] [rust-server] fix p/d bootstrap across dp listeners ↗ sgl-project/sglang
  6. [6] vulkan: add DeepSeek-V4 hyper-connection fused ops (DSV4_HC_COMB/PRE/POST) (#26578) ↗ ggml-org/llama.cpp
  7. [7] vulkan: support type-aligned GET_ROWS (#28253) ↗ ggml-org/llama.cpp
  8. [8] convert : write explicit recurrent_layers for Qwen3-Next / Qwen3.5 (#28208) ↗ ggml-org/llama.cpp
  9. [9] CUDA: branchless Q4_K/Q5_K unpack to speed up mmvq, L2 prefetch on DGX Spark (#26705) ↗ ggml-org/llama.cpp
  10. [10] [Frontend] Add stateless /v1/responses/render endpoint ↗ vllm-project/vllm
  11. [11] [Bugfix] OffloadingConnector: stop zeroing offload hits under MTP/EAGLE spec decode ↗ vllm-project/vllm
  12. [12] [Qwen3.8-Flash-Next] Support FP8 indexer cache for QSA ↗ vllm-project/vllm
  13. [13] [Build] Remove obsolete TPU Dockerfile (#55376) ↗ vllm-project/vllm
  14. [14] [ROCm][CI] Add attention-sink support to ROCm AITER sparse MLA ↗ vllm-project/vllm

$ ls local-llm/reviews/ # the step-back reads

the week in review Vulkan top_k radix select arrives for long rows the month in review Speculative decoding consolidates across the stack as DeepSeek V4 lands
Topics: Local LLMs AI / ML Full archive →

Quick answers

What shipped in Local LLMs on September 8, 2026?
SGLang patches two correctness bugs that caused hangs and wrong expert selection, while llama.cpp and vLLM ship backend improvements. In total, 116 commits, 116 pull requests, and 10 releases landed.
Who contributed to Local LLMs on September 8, 2026?
15 developers shipped this update, including DevVexus, Jeff Bolz, Pranesh Gonegandla, Kevin Hopper, Zhaolun Yin, Woosuk Kwon, AndreasKaratzas, and gau-nernst, and 7 more.
What were the notable Local LLMs updates?
[Scheduler] Make request-timeout aborts rank-consistent to fix TP collective hangs, Revert "[kernel] add fused silu mul quant fp8", and [AMD][GLM-5.2] Keep GlmMoeDsa MoE e_score_correction_bias in fp32.
♥

$ log Local LLMs --recent

Vulkan adds TQ1_0, Spark2.5 lands in llama.cpp 180 shipped vLLM closes scale-out multimodal handoff hole 86 shipped vLLM Fast Start maps weights zero-copy; GLMGA sampling capped 256 shipped Ollama enables speculative decoding under structured output, vLLM adds LoRA and fusion 259 shipped

$ log --all # elsewhere on the wire

@openclaw Selective evidence collection and truthful reviewer prompts land in ClawSweeper AI Agents Cline desktop commands unblocked from slow backend startup

$ ls showcases/

Rails Django Laravel Node.js Go Vercel Vue.js Kubernetes HashiCorp Arch Linux Linux Anthropic

$ init your-desk

RepoJournal can write one of these for any repo you care about. Sign in with GitHub, pick your repos and sources, and your desk gets a wire like this one every morning.