$ the-wire · showcase
Ollama picks parallel slots from memory, llama.cpp fuses Gemma 4 prompt rows
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
The day's work is mostly about squeezing more out of the hardware you already have: Ollama's MLX runner now sizes its slot count from available memory and llama.cpp skips work it never needed to do, while vLLM closes a Rust-frontend gap that returned a hard 400.
mlxrunner: choose default parallelism from available memory ollama/ollama
When OLLAMA_NUM_PARALLEL is unset, the MLX runner now picks 1, 2, 4 or 8 slots from available memory and per-client storage estimated at the model's context limit. Explicit settings still win, and it keeps the serial default under memory pressure, for unknown cache layouts and on other backends.
gemma4: Optimized prompt processing for multi-seq queries ggml-org/llama.cpp
Gemma 4 E4B and E2B can now skip logits computation for prompt rows that do not need it during multi-sequence processing, continuing the work started in an earlier PR. It targets multi-seq prompt processing specifically, so single-stream decode is unaffected.
mlx: fuse Inkling residual mixing on Metal ollama/ollama
Inkling residual mixing on Metal now computes normalized residuals, query scores and the weighted sum in a single dispatch, keeping intermediates in the threadgroup while preserving the existing reduction order and rounding. On an M5 Max with HumanEval at 2048 prompt / 256 generated tokens and a cold cache, generation went from 54.75 to 59.90 TPS (+9.4%) for one user.
[Fix] speculative: Feed every accepted token to the penalizers sgl-project/sglang
Speculative decoding was feeding the penalizers only the last output token per request per verify round, so frequency, presence and repetition penalties plus min_new_tokens counted one token per round instead of every committed token. A new Req.penalty_cumulated_len tracks what the penalizers have seen and cumulate_penalty_output_tokens() feeds each token past it with its batch row as req_indices.
[Rust Frontend] Support `additional_tools` input items in the Responses API vllm-project/vllm
The Rust frontend rejected additional_tools input items with a 400 unknown variant error; it now hoists their tools into the request tools before anything reads them, so they reach the model, tool_choice resolution and the echoed tools like top-level tools. Duplicates and redefinitions merge as they do in Python's _merge_add, and the Rust skip is removed from the e2e test.
Correct gradients and batching for DivMod ml-explore/mlx
Elsewhere in the long tail: mx.divmod now produces real vjp and jvp for its remainder output instead of zeros (so differentiating divmod(x, y)[1] no longer silently yields 0) and its vmap rule reports the second output axis, while llama.cpp added Inkling 2 Mini to mlxrunner, wired Drama 3 voice cloning to the shared Fish DAC encoder, and vLLM published an Inkling 2 Mini recipe that serves from ...