$ the-wire · showcase
OpenVINO backend lands Qwen3.5 on NPU, Whisper.cpp support
By RepoJournal · Filed · About Local LLMs
12 people shipped this
wine99
@wine99
1 cited
Strongtut
@Strongtut
1 cited
ravel7524
@ravel7524
1 cited
zyongye
@zyongye
1 cited
CHIPMUNK-T0T
@CHIPMUNK-T0T
1 cited
xuebwang-amd
@xuebwang-amd
1 cited
BBuf
@BBuf
1 cited
RunFMe
@RunFMe
1 cited
aoshen02
@aoshen02
BugenZhao
@BugenZhao
zhaozx-cn
@zhaozx-cn
Qiaolin-Yu
@Qiaolin-Yu
llama.cpp's OpenVINO backend now runs Qwen3.5 on Intel NPUs and supports Whisper.cpp, while sglang fixes a memory threshold that was causing Cosmos3 Nano to offload on 96 GB GPUs.
llama.cpp's OpenVINO backend updated to OpenVINO 2026.3.1, enabling Qwen3.5 on NPU via static shapes, chunked prefill, and gated-delta-net state handling, and adding Whisper.cpp support [1]. The same work fuses IM2COL plus MatMul into a native OpenVINO convolution and adds RELU, POOL_2D, QUICK_GEGLU, and ROLL ops [1]. A new `GGML_OPENVINO_NPU_COMPILE_CONFIG` env var forwards NPU compilation modes like `optimization-level=3` [1]. Also in llama.cpp, M4 gains `fa_vec_tuned_table` tunings for F16, Q4_0, Q4_1, Q5_0, Q5_1, and Q8_0 on the 10 GPU core part [2]. Vulkan `MUL_MAT_ID` now hoists token-row IDs and expert counts for routed MoE prompt processing, removing redundant routing table searches per workgroup; dense and decode paths are unchanged [3]. In vllm, the Kimi-K3 latent-MoE decode tail gets optimized on SM100 by deferring BF16 top-16 finalization and launching rank-sharded up-projection earlier with PDL [4]. Idefics3 and SmolVLM now support full CUDA graph capture for their ViT encoders by precomputing position IDs [5]. On ROCm, fused shared experts now work with block-quantized FP8, enabling models like DeepSeek-R1-0528 [6]. sglang lowered the Cosmos3 resident-memory threshold so Nano stays on 96 GB GPUs; "Cosmos3 Nano unnecessarily selects component offload even though the measured peak with both components resident is only 36.1 GiB" [7]. Large SM90 row/column-scaled FP8 GEMMs now route to Torch `_scaled_mm` when shapes meet `(K >= 5376 and N >= 3584) or (K >= 3584 and M >= 8192)` [8].
Action items
- → Review OpenVINO backend changes if you target NPUs; new ops and `GGML_OPENVINO_NPU_COMPILE_CONFIG` shipped ggml-org/llama.cpp [monitor]
- → Test MoE routing performance on Vulkan; hoisting may reduce prompt-processing latency ggml-org/llama.cpp [plan]
- → Check if your Cosmos3 Nano workload hits the offload path on 96 GB GPUs; threshold now keeps it resident sgl-project/sglang [plan]
- → Verify FP8 shared-expert fusion if you run DeepSeek-R1-0528 on ROCm vllm-project/vllm [monitor]
References
- [1] OpenVINO: Update OV to 2026.3.1, whisper.cpp support, Qwen3.5 on NPU, and new ops ↗ ggml-org/llama.cpp
- [2] metal : add fa-vec tunings for M4 ↗ ggml-org/llama.cpp
- [3] Vulkan: add hoisting support for row IDs and expert count in shaders ↗ ggml-org/llama.cpp
- [4] [Kimi-K3][Kernel] Optimize the low-M fused latent MoE tail ↗ vllm-project/vllm
- [5] [MM][CG] Support ViT full CUDA graph for Idefics3 and SmolVLM ↗ vllm-project/vllm
- [6] [ROCm][Quantization][MOE] Enable fused shared experts for block-quantized FP8 ↗ vllm-project/vllm
- [7] [diffusion] Keep Cosmos3 Nano resident on 96 GB GPUs ↗ sgl-project/sglang
- [8] [Kernel] Route large SM90 row/column-scaled FP8 GEMMs to Torch ↗ sgl-project/sglang
