$ the-wire · showcase
Ollama stops leaking canceled requests upstream, llama.cpp hardens big-endian paths
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
15 people shipped this
hoyyeva
@hoyyeva
2 cited
taronaeo
@taronaeo
1 cited
lhez
@lhez
1 cited
aoshen02
@aoshen02
1 cited
jberg5
@jberg5
cabelo
@cabelo
kashif
@kashif
Mritunjay Kumar Sharma
@mritunjaysharma394
noooop
@noooop
QHarshil
@QHarshil
Misaka9468
@Misaka9468
CaoE
@CaoE
huangtingwei9988
@huangtingwei9988
kpjeeja
@kpjeeja
wutaosang
@wutaosang
Ollama's remote-model path now propagates client cancellation to the upstream HTTP request, while llama.cpp adds regression coverage for big-endian model conversion and OpenCL picks up new quantized GEMM kernels.
server: cancel remote inference when clients disconnect ollama/ollama
Canceling a local request that targeted a configured remote model could leave the upstream chat or generate call running; the fix passes c.Request.Context() into both API client calls so client cancellation actually reaches the outbound HTTP request. Long-lived or abandoned remote generations stop consuming backend capacity they no longer have a caller for.
openai: finish reasoning before client tool calls ollama/ollama
With a streaming Responses request containing web_search, client tool calls could complete before an open reasoning item even when no search ran, and the stream reused the reasoning item's output index. EmitFunctionCallItems now finishes reasoning before emitting client calls, matching the normal tool path while still permitting subsequent text.
tests: add big-endian conversion tests ggml-org/llama.cpp
New conversion tests cover most data types other than 1-bit and 2-bit against the stories15M model, scoped to s390x CI runners, because conversion script changes used to leave byteswapped Big-Endian models incoherent. Big-endian users get an early warning on regressions instead of discovering them at inference time.
opencl: add cooperative-k dp4a gemm for q4_0, q4_k and q6_k ggml-org/llama.cpp
The OpenCL backend gains cooperative-k dp4a GEMM for q4_0, q4_k and q6_k, with the PR reporting improvement on pp2 through pp4. Extracted from a larger change and refactored, it lands as a contained kernel addition for GPU prompt processing.
[Bugfix][AuxOutput] Send routed-experts block hashes only for computed or scheduled blocks vllm-project/vllm
The scheduler sent a request's block hashes to the worker as soon as they existed, so under chunked prefill it pinned a store object for every prompt block including ones whose KV was not computed yet; hashes are now sent only for computed or scheduled blocks. This is the scheduler-side half of the store eviction fix, which ships separately.
