$ the-wire · showcase
FSDP2 plus expert parallelism, and a unified device mesh for PP + TP inference
By RepoJournal · Filed · About Hugging Face · Composed from the cited sources · methodology
Transformers landed two distributed-training changes: a 2-D (fsdp, tp) device mesh that enables FSDP2 with expert parallelism, and a unified device_mesh initialization that enables pipeline plus tensor parallel inference.
FSDP2 and expert parallelism now compose through a 2-D (fsdp, tp) device mesh [1]. The same work masks uninitialized grouped_mm rows left by sentinel token-expert slots that sit beyond offsets[-1], where torch._grouped_mm leaves forward output and backward d_input uninitialized [2]. Per the commit, the forward "relied on a single post-mask plus a single pre-mask," which let NaN/Inf from uninitialized memory transit the activation and down-projection backward [2]. That is the NaN-gradient failure expert-parallel training was hitting.
Separately, device_mesh initialization is now unified to enable pipeline plus tensor parallel inference [3]. The commit removes an unused device_mesh and is co-authored by Arthur Zucker and guarin [4].
A pending change moves Transformers onto the huggingface_hub httpx export [5]. It is explicitly marked not to merge before Saturday Sep 12, so treat it as scheduled, not shipped.
On the docs desk, the Inference Providers documentation was regenerated twice by the automated workflow, upgrading @huggingface/tasks and @huggingface/inference to latest and re-running generation [6][7]. Both landed with no manual edits: the second is the merge commit of the first [7].
Action items
- → Use the 2-D (fsdp, tp) device mesh if you train with FSDP2 plus expert parallelism huggingface/transformers [plan]
- → Adopt the unified device_mesh init for PP + TP inference configurations huggingface/transformers [plan]
- → Do not merge the huggingface_hub httpx export before Saturday Sep 12 huggingface/transformers [monitor]
References
- [1] Enable FSDP2 + expert parallelism via a 2-D (fsdp, tp) device mesh ↗ huggingface/transformers
- [2] Enable FSDP2 + expert parallelism via a 2-D (fsdp, tp) device mesh (#48516) ↗ huggingface/transformers
- [3] unifying device_mesh init to enable PP + TP inference ↗ huggingface/transformers
- [4] unifying device_mesh init to enable PP + TP inference (#48155) ↗ huggingface/transformers
- [5] [DON'T MERGE BEFORE Sep 12 Saturday] Use huggingface_hub httpx export ↗ huggingface/transformers
- [6] [Bot] Update Inference Providers documentation ↗ huggingface/hub-docs
- [7] Update Inference Providers documentation (automated) (#2780) ↗ huggingface/hub-docs
