Transformers v5.16.0 ships Qwen4-Exp, MoE loss cut 99.7%, TRL unblocks MoE LoRA

Transformers shipped v5.16.0 [1], a release that manages to be both routine and consequential. It includes Qwen4-Exp [1], a new model architecture with expanded support, and a number of fixes: nemotron_h save_pretrained now emits singular backbone.embedding.weight instead of the duplicated form [2], Cpmant use_cache handling is corrected [3], and the Qwen3-VL video processor gains an opt-in per-frame pixel cap via cap_pixels_per_frame [4]. The release also carries a 99.7% memory reduction in the MoE load-balancing loss [5]. The loss was computed via a massive one-hot materialization across all tokens; it is now computed per layer, meaning the loss at 128k sequence length no longer requires a giant intermediate tensor. This is a memory win for any MoE training or fine-tuning job that uses load-balancing loss.

TRL had a busy week that centers on removing roadblocks in the MoE + LoRA path. The revert of an xfail for MoE LoRA tests, now that peft#3602 fixed the autocast dtype bug [6], means that MoE LoRA training is unblocked for real. Even better, the same thread shows that the original 99.7% MoE loss memory cut [5] actually enables MoE LoRA training at all, because the loss calculation would otherwise OOM. TRL also gained tool calling support in the DistillationTrainer refactor [7], aligning AsyncDistillationTrainer with AsyncGRPOTrainer on pre-init args [8], and the SFT collator now emits position_ids when sequence parallelism is enabled [9]. Separately, TRL fixed vLLM weight-sync hangs (as noted in the daily briefing for 2026-08-29) . If you train or fine-tune MoE models with LoRA on TRL, upgrade both TRL and transformers to pick up these fixes.

The kernels ecosystem moved forward on two fronts. huggingface/kernels shipped v0.16.1 [10], a hotfix release, and also documented its telemetry env var better [11]. The more interesting work is in kernels-community: sage-blackwell ports SageAttention3 Blackwell FP4 attention [12], giving Sage Attention a second wind on the newest hardware. Elsewhere, optimum-intel returns Gemma4 tests for OpenVINO [13], and Diffusers refactored the deepfloyd if img2img superresolution pipeline tests to the new mixin structure [14], which is a sign the test suite is getting cleaner even if it doesn't change user-facing behavior.

The rest of the week is smaller, but not unimportant. huggingface_hub v1.26.1 ships [15] and adds support for updating bucket visibility [16]. Transformers also started a linter that hunts nn.Buffer misuse [17], which should catch a class of subtle bugs where buffers are accidentally treated as parameters or vice versa. TRL reorganized its examples into self-contained per-example folders [18] and removed a stale detector exclusion comment [19]. None of these change runtime behavior, but they reduce the chance of future breakage.

For those tracking the transformers core, documentation also caught up: links were fixed and TokenizerFast is removed from the docs [20]. The removal marks the end of a deprecation cycle that started long ago; if you still see TokenizerFast in your own code, the docs say it's gone, and the class has been undocumented for a while.

"Compute MoE load-balancing loss per layer to avoid giant one-hot materialization −99.7% @ 128k" [5]. That PR title, quoted verbatim, captures the week's biggest engineering win.

The through-line this week is that the MoE training stack finally has its memory and dtype problems solved. Transformers cut the load-balancing loss memory by three orders of magnitude, and TRL unblocked MoE LoRA by reverting the xfail once peft fixed the autocast bug. If you had been avoiding MoE LoRA because of OOMs or dtype issues, this is the week to revisit your setup. Upgrade transformers to v5.16.0 and TRL to the latest (which includes the vLLM weight-sync fix) before your next MoE training run.