$ the-wire · showcase
Ollama streams real errors, llama.cpp forwards sampling params on /v1/messages
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
16 people shipped this
hoyyeva
@hoyyeva
1 cited
dhiltgen
@dhiltgen
1 cited
David Garcia
@Dgarc359
1 cited
bzsuni
@bzsuni
1 cited
Fridge003
@Fridge003
1 cited
bosh
@boshjerns
Maphist0
@Maphist0
Hoang Vien Duy
@vieduy
Aleksander Grygier
@allozaur
micah-wil
@micah-wil
WoosukKwon
@WoosukKwon
wenjinhust
@wenjinhust
wutao
@wutaosang
Mick
@mickqian
brandonfzhang
@brandonfzhang
metamergebot
@metamergebot
Two Anthropic-compatible endpoints got materially less lossy today: Ollama stopped swallowing mid-stream runner failures, and llama.cpp's server now carries llama.cpp's own sampling parameters through /v1/messages instead of copying a fixed key list.
anthropic: surface errors after streaming starts ollama/ollama
Streaming /v1/messages responses previously lost in-band runner failures after the HTTP 200, so clients saw truncated text with no cause. The fix detects those errors before chat-response conversion and emits a flushed Anthropic error event in both the ordinary and web-search passthrough writers, preserves pre-stream HTTP/JSON errors, and ends failed streams without a success message_stop; the ...
server : forward llama.cpp sampling params on /v1/messages (#30856) ggml-org/llama.cpp
The server no longer copies a fixed list of keys: the request passes through, and llama.cpp parameters are accepted under extra_params on /v1/messages, with the README example updated to show it. If you have been unable to set llama.cpp-specific sampling knobs through the Anthropic-shaped endpoint, use extra_params instead of the standard fields.
[Bugfix] Resolve disabled thinking effort in Anthropic render vllm-project/vllm
/v1/messages/render bypassed the disabled-thinking effort resolution that /v1/messages and token counting already applied, so render requests with thinking.type="disabled" failed on GPT-OSS and ignored --anthropic-disabled-thinking-effort. The same resolution now runs in both the API server and the standalone render server; requests without disabled thinking are unchanged.
[Feat] cp: Enable GLM-5.3 Flash interleave BCG with batched prefill and DFlash2 sgl-project/sglang
Breakable prefill CUDA graphs are now on by default for GLM-5.3 Flash interleave context parallelism, replaying captured token shapes across multiple prefix-free requests, with the CP helpers consolidated under layers/cp/bcg/. The CUDA release image nightlies for x86_64 ROCm 10.0 needed a revert of the parallel paged-attention kernel build to link again.
[Metal] Add fast route for dequantizing fp4 to half in fp_qmm_t ml-explore/mlx
fp_qmm_t now decodes fp4 to f16 tiles and applies the group scale in one op instead of dequantizing to f32, scaling, then converting to bf16, and it uses bigger tiles for long prompts. Ross Hileman at ml-explore reports no precision loss for nvfp4 and fp16-based mxfp4, while bf16-based mxfp4 models stay on the old path; benchmark output for nvfp4 Qwen3.8-27B prompt throughput is in the PR. Roun...
Action items
- → Drop kitchen_int8 from diffusion configs in sglang, the alias is removed sgl-project/sglang [plan]
