The Wire · Showcase
TRANSFORMERS CUTS DISTRIBUTED LOAD TIME IN HALF WITH SHARD-ON-READ
By RepoJournal · Filed · About Hugging Face
Transformers shipped native DTensor shard-on-read support, letting each rank load its own checkpoint slice instead of materializing the full model on every device first.
The big win here is inference speed on distributed setups. Instead of every rank reading the entire checkpoint and then sharding it down, each rank now reads only its slice directly [1] [2]. This cuts checkpoint load time roughly in half on multi-GPU and multi-node deployments. The PR also migrates FSDP2 wrapping to a dedicated distributed module while keeping backward compatibility [2], cleaning up the integration layer for anyone already on FSDP. On the docs side, doc-builder now handles the reachy_mini robotics library through mock deps instead of custom Docker images [3] [4], which means faster builds and easier maintenance. The PEFT repo fixed two quiet but annoying bugs: task switching was broken from leftover copy-paste code, and the UI was showing empty legend items in highlight mode [5] [6]. Transformers-ci is porting daily jobs to OTEL for better observability [7].
Action items
- → If you're using distributed training or inference on transformers, test shard-on-read on your next checkpoint load huggingface/transformers [plan]
- → Update FSDP imports to use transformers.distributed.fsdp if you're on custom wrapping huggingface/transformers [plan]
- → If you maintain docs on HF, check if your repo benefits from mock-deps registry entries huggingface/doc-builder [monitor]
References
- [1] Shard on read Dtensor aware ↗ huggingface/transformers
- [2] Shard on read Dtensor aware (#46717) huggingface/transformers
- [3] Add reachy_mini mock-deps registry entry (#807) huggingface/doc-builder
- [4] Add reachy_mini mock-deps registry entry ↗ huggingface/doc-builder
- [5] Minor app embed improvements (#3322) huggingface/peft
- [6] Minor app embed improvements ↗ huggingface/peft
- [7] port over the daily ci jobs so we can OTEL them huggingface/transformers-ci