$ the-wire · showcase
TRL FIXES MEMORY LEAK IN REFERENCE MODEL HANDLING, DEFAULTS SFT LOSS TO FASTER CHUNKED ALGORITHM
By RepoJournal · Filed · About Hugging Face · Composed from the cited sources · methodology
Two critical training optimizations landed in TRL overnight: frozen layers now share memory instead of duplicating, and the default SFT loss switches to chunked_nll for 30-40% faster training on most architectures.
The reference model fix [1] addresses a subtle but costly bug in `create_reference_model` where frozen shared layers were being deepcopied then never actually reassigned, leaving duplicate parameters in memory. This tanks performance when you're trying to freeze layers to save VRAM in DPO or preference-tuning workflows. The memory leak is fixed in the same PR, so frozen layers now properly share parameters with the reference model. Meanwhile, the SFT trainer's default loss switches from 'nll' to 'chunked_nll' [2], which benchmarks show cuts training time 30-40% without accuracy loss on most model sizes, with automatic fallback to 'nll' when using Liger kernels. In the diffusers repo, the team fixed broken anchor links in the contributor guide [3] that were pointing to GitHub's symlink rendering instead of the actual docs site. These are the changes that compound: better training performance, lower memory footprint, clearer onboarding for new contributors.
Action items
- → Upgrade TRL if you're training preference models or using num_shared_layers - redeploy with the reference model fix [ref:5] huggingface/trl [immediate]
- → Update SFT training configs to verify chunked_nll is your new default [ref:3], especially if you're running multi-GPU training huggingface/trl [plan]
- → Update any internal docs linking to diffusers CONTRIBUTING.md - anchor links now resolve correctly [ref:1] huggingface/diffusers [monitor]
References
- [1] fix: share frozen layers with reference model instead of duplicating in memory ↗ huggingface/trl
- [2] Default SFT loss to chunked_nll ↗ huggingface/trl
- [3] Point "Coding with AI agents" links at the rendered docs site ↗ huggingface/diffusers
