$ the-wire · showcase
Ollama stops reloading on shift flips, vLLM drops the doubling copy path
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
Today's work is mostly startup and scheduling overhead: Ollama shares one runner across context-shift settings, vLLM deletes the IPC copy mode that doubled VRAM, and llama.cpp's WebUI stops showing newcomers a models manager.
server, llm: share one runner across `shift` settings ollama/ollama
`--context-shift` is a llama-server launch flag, so any request whose `shift` differed from the loaded runner forced a model reload; clients alternating `shift: false` with the default paid that on every request. The scheduler now launches with context shift whenever the model supports it and enforces `shift: false` per request by capping `n_predict` at `num_ctx - prompt_tokens`, so only a runn...
[Fast Start] Remove copy mode for fast start IPC loader vllm-project/vllm
`copy` mode for the fast-start IPC loader is gone. vLLM's own description says it doubles VRAM usage and significantly increases startup time, making cold start the faster path; anyone who selected copy mode for startup now gets the shared-memory path instead.
ui: hide Models and MCP behind settings, off by default ggml-org/llama.cpp
The WebUI hides Models and MCP by default behind two new General settings, "Show Models" and "Show MCP". Existing MCP servers and the Discover Models toggle are preserved, and MCP servers keep working while hidden: the setting only removes the UI for managing them.
[CUDA] : Add SKU-aware FP16/BF16 accumulation precision gating for NVIDIA GPUs ggml-org/llama.cpp
cuBLAS precision selection is now keyed to the GPU. Eligible quantized ops keep FP16 inputs and accumulation on GeForce, while supported professional GPUs and Spark move to BF16 inputs with FP32 accumulation, classified via NVML brand information and Spark's compute capability, with existing per-op precision requirements and manual overrides intact.
[Bugfix][NIXL] Push: stop WRITE submission stalls and early block release; lock the NIXL agent vllm-project/vllm
NIXL push mode in P/D disaggregation now defaults to `num_threads=0`, so WRITEs are posted from the writer thread instead of waiting on the UCX backend's posting threads, which stalled large writes under sustained load; the NIXL agent is also created with `NIXL_THREAD_SYNC_RW`, since without a sync mode its lock is a no-op. Elsewhere: sglang added Intel XPU to the EPD encoder path after CUDA-ha...