$ the-wire · showcase
Fix expert-parallel NaN gradients and honor shift_labels
By RepoJournal · Filed · About Hugging Face
9 people shipped this
Transformers fixes NaN gradients in expert-parallel training and makes decoder-only LLM/VLM losses actually honor shift_labels.
The most consequential change lands in transformers: expert-parallel training could produce NaN/Inf gradients because torch._grouped_mm left sentinel token-expert output rows uninitialized, and the forward relied on masks that let the garbage transit activation and backward passes. The fix masks those rows, so EP runs no longer corrupt gradients [1].
Quentin Gallouédec also shipped the `shift_labels` fix: decoder-only LLM/VLM losses previously ignored the `shift_labels` argument, forcing a manual shift that renamed the target from `labels` to `shift_labels`. Generated files for qwen4_exp and blip were regenerated or allowlisted [2].
Over in serge, a one-line deploy regression is now guarded: on 2026-09-04, a deploy from a stale `deploy/helm/env/prod.yaml` inside the repo silently dropped live settings like `VERIFY_ON_GPU` and `VERIFY_REPRODUC...`, though helm reported `Upgrade complete` and health checks stayed green. New logic refuses a deploy whose values omit settings present in the live release, preventing that silent config loss [3]. Separately, serge now reports peak and cached input tokens after discovering `LLM_MAX_INPUT_TOKENS` caps a cumulative billing figure that grows quadratically in turns because the agent loop re-sends the full conversation every turn [4].
Funes also shipped three fixes: bumping lance from 7 to 11 (four major versions behind) [5], a workaround for corrupt cache entries where a failed download's error response gets saved as the object and later produces a backwards byte range [6], and replacing a giant `id IN (…)` filter in push with an in-Rust match that scans only the id column, fixing what looked like a memory leak on memories with many pending ids [7].
Finally, TRL raised dependency floors: it now requires peft>=0.13.0 and deepspeed>=0.18.6, dropping all the old version guards and the imports they made dead [8] [9].
Action items
- → Upgrade transformers to include the expert-parallel NaN fix before running EP training huggingface/transformers [immediate]
- → Verify your serge deploy pipeline uses the canonical values file, not a stale repo copy huggingface/serge [immediate]
- → Review funes push behavior after the id-filter replacement to confirm RAM usage drops huggingface/funes [plan]
- → Update TRL dependencies to meet peft>=0.13.0 and deepspeed>=0.18.6 huggingface/trl [plan]
References
- [1] Fix expert-parallel training: NaN gradients and missing gradient contributions (#48205) ↗ huggingface/transformers
- [2] Honor `shift_labels` in decoder-only LLM/VLM losses ↗ huggingface/transformers
- [3] Refuse a deploy whose values drop settings the live release has ↗ huggingface/serge
- [4] Report peak and cached input tokens, and add an opt-in transcript window ↗ huggingface/serge
- [5] Bump lance version to 11 ↗ huggingface/funes
- [6] Workaround corrupt cache entries ↗ huggingface/funes
- [7] fix(push): select pending rows without a giant id filter ↗ huggingface/funes
- [8] Require peft>=0.13.0 and drop the 0.12 version guards (#7102) ↗ huggingface/trl
- [9] Require deepspeed>=0.18.6 and drop the 0.16.4 guard (#7118) ↗ huggingface/trl
