$ the-wire · showcase
llama.cpp server stack overflow, Ollama brings Metal kernels to CUDA
By RepoJournal · Filed · About Local LLMs · Composed from the cited sources · methodology
18 people shipped this
jmorganca
@jmorganca
2 cited
dhiltgen
@dhiltgen
1 cited
ykhrustalev
@ykhrustalev
1 cited
mickqian
@mickqian
1 cited
Ronald Mannak
@ronaldmannak
1 cited
anra8571
@anra8571
ParthSareen
@ParthSareen
yomaytk
@yomaytk
ngxson
@ngxson
taronaeo
@taronaeo
AndreasKaratzas
@AndreasKaratzas
Taimys
@Taimys
MatthewBonanni
@MatthewBonanni
lucamotz
@lucamotz
GuanLuo
@GuanLuo
OnePunchMonk
@OnePunchMonk
ch-wan
@ch-wan
DarkSharpness
@DarkSharpness
Ollama ported its Metal custom kernels to CUDA to give inference models real GPU paths, while llama.cpp fixed a remotely triggerable stack overflow in llama-server and mlx closed a Metal lifetime bug that could corrupt fill values under timing.
server : escape the special tokens without std::regex ggml-org/llama.cpp
A /v1/systemone request against a d1-omni or kev model could crash llama-server on Linux: escape of <|name|> used std::regex_replace, whose libstdc++ backtrace-matched one recursive call per character, so a state of "<|" followed by 30,000 letters overflows the 8 MB HTTP thread stack before any size check runs. macOS via libc++ is unaffected, and the fix drops std::regex entirely.
mlx: port the Metal custom kernels to CUDA ollama/ollama
mamba2_scan, depthwise_conv_silu and gated_delta (with their _states and _bias variants) had Metal sources only, so CUDA fell back to graph ops on every call. dhiltgen's port keeps the Metal algorithms with CUDA-specific deviations: early scan return past D, per-warp registers replacing threadgroup q/k caches, and a shfl butterfly in place of simd_sum.
cli: add decision model support to `ollama run` ollama/ollama
ollama run previously failed on decision models with "nimble" does not support generate; it now accepts them, with --ask for yes/no, --choice for labels and --level for scored answers. State is read from stdin or the arguments after the model name, and a JSON object or array is sent as JSON.
decision: fix JSON state prompts for MLX decision models ollama/ollama
Ollama had been compacting object and array states before building Nimble, Laya and Tev1 prompts, so MLX decision models scored JSON differently from the same models on GGUF, with answers differing by up to 0.085 in earlier comparisons. Prompts now match the publishers' code and llama.cpp's json.dumps output, and Tev1 gets its state nested in the payload instead of escaped as a string.
Fix missing add_temporary in pad_gpu/fill_gpu ml-explore/mlx
Metal fill_gpu never kept the fill value alive, so the GPU can read overwritten values depending on timing; 12 call sites in conv.cpp and gated_delta_update.cpp leave the value unretained and are potentially exposed (confirmed with 3D Conv), against 8 that call add_temporary correctly. A companion commit fixes the same omission in pad_gpu and fill_gpu.
[diffusion] Replay small Triton wrappers' launches per operand signature sgl-project/sglang
On the long tail: sglang cut per-call host overhead in small Triton kernels by replaying launches per operand signature, noting that rmsnorm_scale at a Z-Image 512² shape costs about 28 µs of host time on H200 against a 9 µs launch, and sglang also added a PR title format check requiring a Type of Feat, Fix, Refactor, Perf, Doc, Test, CI or Chore.
