Local LLMs
Vulkan adds TQ1_0, Spark2.5 lands in llama.cpp
llama.cpp and its ecosystem shipped new hardware support and a model architecture, while vLLM and SGLang fixed bugs in prefill and warmup paths.
read --wire →
$ tail -f topics/llms.log
Local LLMs news, every morning.
Daily updates from the local-LLM inference stack - Ollama, llama.cpp, vLLM, and SGLang. What shipped this week in running and serving open models on your own hardware.
11 updates across 2 projects this week.
Your week in Local LLMs, every Monday.
One calm review of what shipped across Local LLMs - the commits, releases, and security advisories that matter. Every Monday, with security advisories same-day. Free, unsubscribe in one click.
We'll start you on the top Local LLMs projects - refine anytime. · Read a sample issue →
Recent wires
Open WebUI
Open WebUI fixes MCP cleanup cancel-scope crash and expands language support
A fix removes asyncio.wait_for/shield from MCP cleanup in the chat handler, preventing a BaseException crash on anyio cancel-scope violations.
read --wire →
Open WebUI
Open WebUI Desktop fixes blank Linux webviews, decouples local server
Open WebUI Desktop 0.0.19 ships with a critical Linux fix that restores blank webviews by replacing --in-process-gpu with SwiftShader.
read --wire →
Open WebUI
Desktop patches Python bootstrap, GPU rendering, and model discovery
Open WebUI Desktop shipped three fixes that repair broken terminal startup, blank webview content on Linux, and hidden Hugging Face models.
read --wire →
Open WebUI
Open WebUI desktop fixes external link handling
Open WebUI desktop release 0.0.17 fixes external links so they open in the default browser instead of inside the app.
read --wire →
Local LLMs
vLLM closes scale-out multimodal handoff hole
vLLM now validates scale-out multimodal data before engine handoff, closing a trust gap in the generate route.
read --wire →
Local LLMs
vLLM Fast Start maps weights zero-copy; GLMGA sampling capped
Two security patches and a zero-copy weight cache land across vLLM and llama.cpp, while sglang ships v0.5.19 with 786 PRs.
read --wire →
Local LLMs
Ollama enables speculative decoding under structured output, vLLM adds LoRA and fusion
Ollama's MLX runner now runs speculative decoding under structured output constraints, promising near-full draft throughput on MLX models, while vLLM lands static LoRA loading in its Rust frontend and begins manual activation-quant fusion.
read --wire →
Local LLMs
Error propagation hardened in ollama's MLX bindings, gemma4 gains multimodal support
ollama's MLX runner now treats every fallible MLX call as an error it can surface, ending silent failures that corrupted outputs.
read --wire →
Local LLMs
Ollama honors model generation defaults; llama.cpp and vLLM ship MoE performance fixes
Ollama now respects model-authored sampler defaults from GGUF metadata, and both llama.cpp and vLLM land performance and correctness fixes for MoE kernels.
read --wire →
Local LLMs
vLLM bounds validation errors, AutoRound FP8 arrives
vLLM shipped a security fix bounding validation-error response bodies and added AutoRound block-wise FP8 support, while llama.cpp and sglang pushed kernel fusions for Blackwell and ROCm.
read --wire →
Local LLMs
Vulkan top-k radix sort, HIP Q2_0 optimizations, and vLLM CUDA-graph memory fix
Across three repos, the most consequential changes tune performance on AMD and Vulkan, fix a vLLM memory regression, and extend speculative decoding support.
read --wire →
Local LLMs
llama.cpp Vulkan memory bloat fixed, vLLM deprecates PyAV
llama.cpp fixes a Vulkan memory bloat bug that caused extreme VRAM usage, while vLLM begins deprecating its PyAV video decoder backend over performance concerns.
read --wire →
Local LLMs
OpenVINO backend lands Qwen3.5 on NPU, Whisper.cpp support
llama.cpp's OpenVINO backend now runs Qwen3.5 on Intel NPUs and supports Whisper.cpp, while sglang fixes a memory threshold that was causing Cosmos3 Nano to offload on 96 GB GPUs.
read --wire →
Local LLMs
LLAMA.CPP UNLEASHES QWEN3.8-FLASH-NEXT SUPPORT
The biggest local-model news in weeks: llama.cpp just landed draft support for Qwen3.8-Flash-Next, and it ships with three quantizer fixes the model needs.
read --wire →
Local LLMs
vLLM 0.28.0 LANDED WITH 584 COMMITS AND A KIMI-K3 PERFORMANCE PUSH
vLLM shipped its biggest release in months, and Ollama quietly fixed a macOS bug that could leave stale app processes running.
read --wire →
Local LLMs
OLLAMA'S MLX RUNNER NOW SPEAKS STRUCTURED OUTPUT
Ollama closed a silent correctness gap on Apple Silicon, and the LLM inference stack shipped a wave of fixes that touch everything from tool calls to native ARM support.
read --wire →
Local LLMs
OLLAMA BOLSTERS CLAUDE DESKTOP, VLLM HARDENS AUDIO PATHS
Ollama's desktop app just made Claude Desktop integration faster and safer to manage, while vLLM closed a security gap on audio file size limits.
read --wire →
Local LLMs
VLLM'S P2P WEIGHT SYNC CUTS RL MOE TRANSFER COSTS
Forget all-gathering experts during RL rollout: vLLM's new sharded_rdt backend lets each worker pull only the slice it needs, turning a total_bytes transfer into a total_bytes divided by num_workers one.
read --wire →
Local LLMs
SGLang Fixes Pixtral 500s as vLLM Cuts TTFT 25%
The biggest news overnight: SGLang is patching a production-breaking Pixtral bug, and vLLM engineers found a way to shave up to 25% off time-to-first-token on Mamba models.
read --wire →