$ the-wire · showcase
Metal MoE fix for duplicate ids, sglang stops crashing on retracted DeepSeek-V4
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
15 people shipped this
ggerganov
@ggerganov
1 cited
akshayv
@akshayv
1 cited
venkywonka
@venkywonka
1 cited
hnyls2002
@hnyls2002
1 cited
Vegetog
@Vegetog
1 cited
Anna Rahn
@anra8571
dhiltgen
@dhiltgen
jmorganca
@jmorganca
ngxson
@ngxson
xinnywinne
@xinnywinne
tomasruizt
@tomasruizt
ieBoytsov
@ieBoytsov
AMD-yanfeiwang
@AMD-yanfeiwang
1am9trash
@1am9trash
Oasis-Git
@Oasis-Git
Two correctness and crash fixes land: llama.cpp's Metal mul_mat_id path silently computed wrong expert outputs whenever ids repeated, and sglang's decode scheduler raised NotImplementedError on the first retraction for DeepSeek-V4.
metal : fix mul_mat_id with duplicate ids (#30373) ggml-org/llama.cpp
The old kernel_mul_mm_id_map0 assumed at most one match per token and a fixed capacity of ne21 entries per expert, so duplicate ids produced wrong or dropped output rows and could overflow into the next expert's slice. The prepass now builds a compact per-expert list of (token, slot) entries, counting each expert's entries in pass 1 before thread 0 computes the slice offsets, so Metal MoE infer...
[PD] Recompute retracted DeepSeek-V4 requests on prefill instead of crashing decode sgl-project/sglang
dsv41_dspark_needs_rebootstrap is replaced by dsv4_needs_rebootstrap, so every DeepSeek-V4 decode server recomputes a retracted request's prefix on the prefill rather than falling back to the cpu_tensor path that DeepSeekV4TokenToKVPool does not implement. Before this, V4 and V4.1 without DSpark crashed the decode scheduler with NotImplementedError on their first retraction; backup_kv_cache als...
[Bugfix][KVConnector] Fix MoRI-IO WRITE-mode requests stranded in WAITING_FOR_REMOTE_KVS vllm-project/vllm
In MoRI-IO WRITE (push) mode, decode requests could sit in WAITING_FOR_REMOTE_KVS forever under decode overload while the engine stayed healthy and kept accepting new work, showing up as a benchmark that never finishes rather than an error. This fixes two of the three causes at 1024 in / 1024 out, 128 requests at concurrency 64 with decode limited to 20 running requests; the third, decode preem...
[Attention][Spec Decode] Multi-query Triton split-K for spec-decode verify, with empty-segment/padding hardening vllm-project/vllm
Spec-decode verify runs 1+K query tokens per request, but on main only max_seqlen_q == 1 entered the Triton 3D split-K kernel, pushing verify batches to the 2D kernel that parallelizes only over Q blocks and slows down at long context and low batch. Split-K now runs under a shape-only gate in unified_attention when max_seqlen_q <= 8, with empty-segment and padding hardening alongside.
[Fix] Preserve HTTP errors for non-streaming /v1/messages sgl-project/sglang
Non-streaming /v1/messages turned an upstream HTTPException(503) into 500 api_error because the Anthropic handler caught it as a generic exception, so queue-full rejections lost their overload status. HTTPException is now caught ahead of the generic handler and its status code preserved.
