$ the-wire · showcase
OLLAMA CUTS MODEL CONFIG BLOAT 100KB+ PER MODEL, LLAMA.CPP SHIPS TOOL ISOLATION AND CUDA FUSION GAINS
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4)
11 people shipped this
Ollama eliminated redundant manifest data ballooning config blobs on safetensors models, while llama.cpp landed docker-backed tool isolation and fused three GPU kernels into one.
Ollama's manifest layer deduplication [1] strips unused rootfs.diff_ids that were copying every layer digest into the config blob, cutting config size by over 100KB on per-tensor safetensors models without changing model identity. Llama.cpp shipped four releases in 24 hours rolling out production-grade tool isolation via docker [2], proper working directory reporting for isolated runtimes [3], and UI controls that only surface when tools actually need them [4]. On the GPU side, a new CUDA kernel fuses rms_norm plus mul plus rope operations, combining existing fusions and driving 1% throughput gain on affected models [5]. VLLm migrated bitsandbytes support to an out-of-tree plugin [6], trimming core dependencies while keeping BNB models working via simple package install. SGLang hardened KV cache freeing against aliasing bugs [7], unified scheduler write-after-read barriers across speculat and non-speculative paths [8], and cut 5.5 GPU-hours from nightly CI by removing redundant test registrations [9].
Action items
- → Test Ollama deployments against new manifest format - config blob sizes will drop immediately on model pulls ollama/ollama [plan]
- → If using llama.cpp server with tools, upgrade to b10328+ to get isolation and proper cwd reporting ggml-org/llama.cpp [plan]
- → BNB users: after next vllm release, install vllm-bnb-plugin separately instead of relying on core support vllm-project/vllm [monitor]
- → SGLang users on paged KV allocators should pull the KV free aliasing fix to prevent request row corruption sgl-project/sglang [plan]
References
- [1] manifests: remove OCI rootfs from the model config (#17619) ↗ ollama/ollama
- [2] b10328 ↗ ggml-org/llama.cpp
- [3] b10331 ↗ ggml-org/llama.cpp
- [4] b10329 ↗ ggml-org/llama.cpp
- [5] b10330 ↗ ggml-org/llama.cpp
- [6] [Migration] Migrate bitsandbytes support to OOT plugin ↗ vllm-project/vllm
- [7] [Bugfix] Fix batched KV free aliasing ↗ sgl-project/sglang
- [8] [Scheduler] Unify WAR read-done gating behind shared-read boundary declarations ↗ sgl-project/sglang
- [9] [CI] Trim redundant nightly test registrations ↗ sgl-project/sglang
