$ the-wire · showcase
Ollama stops silently truncating Responses inputs; llama.cpp cuts checkpoint copies
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
21 people shipped this
marlonferreira1800-hue
@marlonferreira1800-hue
1 cited
cwriter
@cwriter
1 cited
fuscof-ibm
@fuscof-ibm
1 cited
BBuf
@BBuf
1 cited
erwinzhang7
@erwinzhang7
1 cited
jberg5
@jberg5
ParthSareen
@ParthSareen
liangliang205
@liangliang205
Adrien Gallouët
@angt
tokinasin
@tokinasin
JohannesGaessler
@JohannesGaessler
jadenmach2
@jadenmach2
Fangzhou-Ai
@Fangzhou-Ai
aoshen02
@aoshen02
Etelis
@Etelis
SKRohit
@SKRohit
Shunkangz
@Shunkangz
jybsuper
@jybsuper
Liangsheng Yin
@hnyls2002
nastya236
@nastya236
ArupKantiDas
@ArupKantiDas
The most consequential fix of the day is a quiet one: Ollama's OpenAI-compatible endpoint had been ignoring the `truncation` field entirely, so clients asking for `"disabled"` still got their oversized prompts shortened by the runner.
openai: forward truncation option in responses requests ollama/ollama
`FromResponsesRequest` never mapped `Truncation` onto `api.ChatRequest.Truncate`, leaving it nil so the chat handler defaulted `truncate` to true even when `truncation` was set to `"disabled"`. It now maps `"disabled"` to `Truncate: &false`, `"auto"` to `&true`, and defaults to `Truncate` otherwise, so long inputs that previously vanished mid-prompt reach the model intact.
llama-server: Use reference-counted checkpoints instead of deep-copies ggml-org/llama.cpp
Context checkpoints in `llama-server` were created and reused by vector copy; since they are immutable by design, they are now shared through `std::shared_ptr` with zero-initialization skipped, guarded by the patch's own invariants after the author flagged the security angle. The change is cheap ref-counted copies instead of full deep copies on each checkpoint.
[Bugfix][Spec Decode] Avoid catastrophic cancellation in the acceptance estimator vllm-project/vllm
The adaptive-verification acceptance feature collapses to the constant `40.0` for every draft row more confident than `1 - 6e-8`, losing its whole signal to catastrophic cancellation; the estimator recovers the complement mass by subtracting 1 from a sum that no longer holds the precision. Any spec-decode setup relying on that feature for adaptive verification was working from a flat number.
[CPU] Fix accumulation precision in affine quantized matmul ml-explore/mlx
CPU affine quantized matmul accumulated in the output dtype, round-tripping every intermediate sum; `_qmm_t` and `_qmm` now accumulate in float32 and convert once, with a deterministic regression test added. Errors were largest for big reduction dimensions, exactly where quantized CPU inference is slow enough that you now notice it is also wrong.
[Refactor] kernels/diffusion/tests: Prune new dead code and make silently skipped tests run sgl-project/sglang
SGLang's second dead-code pass over `kernels`, `multimodal_gen` and `test/registered` lands at 433 files, +5,551 / −13,052, and the part that matters is not the deletions: it fixes tests that were green in CI without exercising anything, promoted about 1.4K lines of assertions to `test/manual/` because no runner had them, plus runtime bugs found along the way. The long tail: Ollama restores the...
Action items
