$ the-wire · showcase
Ollama brings parallel generation to MLX, llama.cpp adds FP8 KV-cache
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
Ollama's MLX path can now serve several generation requests at once, and llama.cpp grew an FP8 KV cache, while the serving engines spent the day on scheduler and connector plumbing rather than headline throughput.
mlx: support concurrent generation ollama/ollama
MLX models can now serve concurrent generation through OLLAMA_NUM_PARALLEL, with the default still at one request and each additional in-flight request costing its own memory. Ready requests share projection and feed-forward work while attention, caches, sampling and response streams stay independent, and the scheduler interleaves 2048-token prompt chunks with decode rounds.
Add FP8 KV-cache support ggml-org/llama.cpp
llama.cpp accepts OCP-FP8 KV caches, with optional run-time static per-tensor scales; scaling is applied at graph level rather than inside ggml_flash_attn_ext, matching the NVFP4/FP8 ownership model and letting the same logic serve flash and standard attention. Models with non-unit-scaled FP8 KV caches, such as nvidia/Mistral-Medium-3.5-128B-NVFP4, are what the path is built for.
ggml : add MUL_MAT src1 precision tests and gating ggml-org/llama.cpp
Backends that cannot honor a requested MUL_MAT src1 precision now fall back to the CPU instead of silently computing at the wrong width. Metal rejects MUL_MAT and MUL_MAT_ID for GGML_PREC_F32 and MUL_MAT for GGML_PREC_BF16 unless the tensor API is present and src0 is not F32; CUDA rejects GGML_PREC_F32 for MUL_MAT on Ampere+ (TF32) and for MUL_MAT_ID, and test-backend-ops gains BF16/F32 src1 ca...
[Bugfix][Rust Frontend] Fix cancellation during request submission vllm-project/vllm
A race in the Rust frontend could send Abort before Add when a caller cancelled mid-submission, letting the engine process a request that was already cancelled; the output stream now stays alive in the spawned Add send task until the send completes. A deterministic regression test cancels after the task is spawned and before it runs.
[Refactor] moe: Remove the triton_kernel MoE runner backend sgl-project/sglang
The triton_kernel MoE runner backend is deleted, the largest step in a four-PR stack to drop the vendored triton_kernels package from the sglang-kernel wheel; --moe-runner-backend triton_kernel no longer resolves.
Action items
- → Replace --moe-runner-backend triton_kernel with a supported MoE runner backend in sglang configs sgl-project/sglang [plan]