$ the-wire · showcase
llama.cpp's RPC layer gets faster and stops aborting on bad nodes
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
13 people shipped this
ngxson
@ngxson
1 cited
figkcoci-debug
@figkcoci-debug
1 cited
am17an
@am17an
1 cited
nliu365
@nliu365
1 cited
mickqian
@mickqian
1 cited
dorado269
@dorado269
1 cited
hnyls2002
@hnyls2002
1 cited
hoyyeva
@hoyyeva
allozaur
@allozaur
Juan Pérez de Algaba
@jperezdealgaba
Hotragn
@Hotragn
mani1106
@mani1106
DarkSharpness
@DarkSharpness
llama.cpp reworked its RPC path for both speed and durability, vllm-project/vllm corrected an MoE scaling bug that degenerates FP16 GLM-4.7 output, and sglang fixed a pause that never released its KV cache.
RPC: Improve loading times ggml-org/llama.cpp
The RPC tensor-loading path got substantially faster in one reported configuration: with DeepSeek-V4-Flash at 145.6 GiB across two DGX Spark boxes over RDMA and a cold page cache, load_tensors dropped from 1484 s to 166 s under -sm tensor -lm dio. The author reports a 5x overall improvement; the smallest gain in the table, mmap with -sm layer, was 495 s to 326 s.
ggml-rpc : fail gracefully on unreachable endpoints and failed handshakes (#30415) ggml-org/llama.cpp
rpc_dispatcher::start() previously aborted the whole host process via GGML_ABORT, sending SIGABRT, when the endpoint failed to parse, the transport failed to initialize, the socket could not connect, or the HELLO handshake failed. That meant one typo in --rpc or one down node killed your application and the reason went to a log stream you might never see. It now fails gracefully, so an unreacha...
[Bugfix][MoE] Only divide shared expert output in FP16 for models that compensate vllm-project/vllm
Since the earlier MoE runner change, the shared expert output was divided by routed_scaling_factor in FP16 whenever a model set apply_routed_scale_to_output=True, which is only correct for DeepseekV2-style decoders that compensate for it. GLM-4 MoE does not, so FP16 GLM-4.7 produced degenerate output on v0.20.0 and later. The divide-shared path is now opt-in; the report reproduces it with FP16 ...
[Fix] scheduler: Retract every KV holder on pause and order a few releases behind in-flight steps sgl-project/sglang
pause_generation(mode="retract") was missing requests that still held KV cache and, under pipeline parallelism, could retract one request twice. The effect compounded: a paused scheduler never reached idle, so release_memory_occupation failed for the duration of the pause, and a dLLM request crashed the scheduler on resume. Two other operations also failed to wait on a step that was already lau...
mtmd: support gemma 4 input audio stream ggml-org/llama.cpp
llama.cpp's mtmd now streams gemma 4 audio input through the existing mtmd_audio_stream infrastructure, since the gemma 4 audio encoder is causal. The landing value is step_ms=480ms (up from an initial 320ms for speed), with worst-row cosine between streaming and non-streaming measured at about 0.993 to 0.997.
[Fix] [AMD] dsa: Align Aiter paged rows with DP-attention metadata sgl-project/sglang
On AMD, indexer._get_topk_paged could receive physical query and weight rows past the real query count q_offset once DP-attention padding was in play: Aiter sized logits from the query tensor while top-k sized from real-row metadata, so padded inputs failed the row check, and the post-top-k padding restoration excluded HIP. Reproduced on MI355X with DeepSeek-V3.2, TP8 and --attn-dp-size 8.
[Fix] diffusion: Count reclaimable page cache as available inside a cgroup cap sgl-project/sglang
In the remaining fixes: sglang's physical_host_memory_available_bytes() now counts reclaimable page cache inside a cgroup cap, matching what psutil's MemAvailable reports outside one, after the two paths disagreed on an RTX 5090 with MiniMax-H3 under a 56G MemoryMax scope; the Ollama launch CLI now builds OpenCode thinking variants from shared model metadata instead of advertising named levels ...
Action items
