$ the-wire · showcase
Streamed tool calls get 200x faster in vLLM; llama.cpp learns proxies
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
18 people shipped this
hoyyeva
@hoyyeva
1 cited
texasich
@texasich
1 cited
wangxiyuan
@wangxiyuan
1 cited
deadstrobe5
@deadstrobe5
1 cited
metamergebot
@metamergebot
1 cited
drifkin
@drifkin
pdevine
@pdevine
pwilkin
@pwilkin
IMbackK
@IMbackK
danielhanchen
@danielhanchen
mctylr-gh
@mctylr-gh
mottopanikeiku
@mottopanikeiku
omuhamma
@omuhamma
WoosukKwon
@WoosukKwon
Oasis-Git
@Oasis-Git
AaravMehta-07
@AaravMehta-07
caihuali95
@caihuali95
XXXXRT666
@XXXXRT666
vLLM's MiniMax M2 parser stopped rescanning the whole argument on every delta, and llama.cpp finally reads HTTPS_PROXY on its download path, so both long tool-call streaming and proxied model fetches stop being the thing that breaks.
[Perf][Parser] Stream MiniMax M2 tool argument values incrementally (16 KB call: 5.9 s to 26 ms) vllm-project/vllm
The MiniMax M2 parser re-ran its argument converter and the engine's safe-prefix scan over the entire argument on every streamed delta, making long parameter values quadratic; it now uses an arg_value_extender, the same mechanism glm47 and gemma4 already had. Parser CPU for a streamed write_file call dropped from 5.9 s to 26 ms at 16 KB and from 387 ms to 6.7 ms at 4 KB, which is the difference...
common: honor HTTP_PROXY/HTTPS_PROXY env vars in http client ggml-org/llama.cpp
common_http_client() built the httplib client without ever checking proxy environment variables, so downloads through that path, including the -hf flag and model presets, failed silently in proxied environments. It now reads HTTPS_PROXY/https_proxy for https:// URLs and HTTP_PROXY/http_proxy for http:// ones, with scheme-specific values taking precedence, and passes them to cli.set_proxy().
openai: allow Responses effort on non-thinking models ollama/ollama
Switching a Responses client from a thinking model to llama3.2 could carry a saved reasoning.effort=high into the next turn, which Ollama rejected with does not support thinking. The fix applies the existing non-thinking capability fallback to effort-derived settings and reuses the resolved think value for web-search follow-ups, while explicit think overrides keep precedence.
[Core][Feat] Pluggable KVCacheConfigBuilder for platform/model-specific KV cache planning vllm-project/vllm
vLLM's single get_kv_cache_groups() heuristic now gives way to an experimental pluggable KVCacheConfigBuilder, with core keeping one planning flow and validating every plan. It is a follow-up to existing work replacing the monkey-patching vllm-ascend does to core planning for its DSV4 plan, so new hybrid models and cache precisions stop requiring edits to core.
[Perf] unified-memory: Prewarm the compaction move at startup and copy whole page envelopes by page id sgl-project/sglang
The long tail, as usual: sglang prewarms its unified-memory compaction at startup to avoid a first-use Triton out-of-memory that takes down the scheduler, skips discarded KV rows during DSv4.1 window commit, and swaps gloo all_gather for gather plus broadcast on DP attention; Ollama defers redirect-target DNS to the proxy in the same PR that fixes the lm_head-to-embed_tokens rename on the mtp f...
