$ the-wire · showcase
TorchTitan unifies graph execution and makes grouped MoE projections the default
By RepoJournal · Filed · About PyTorch · Composed from the cited sources · methodology
14 people shipped this
sanketpurandare
@sanketpurandare
2 cited
Pearu Peterson
@pearu
1 cited
Ivan Kobzarev
@IvanKobzarev
1 cited
jinsooihm
@jinsooihm
1 cited
rraminen_amdeng
@rraminen
Albert Malewski
@albmalamd
Natalia Gimelshein
@ngimel
Apostolos Kokolis
@A-Kokolis
wwwjn
@wwwjn
neuriv
@neuriv
theap06
@theap06
atalman
@atalman
can-gaa-hou
@can-gaa-hou
izaitsevfb
@izaitsevfb
TorchTitan's training stack got its biggest structural changes of the week, while the core repo kept to kernel-level tuning and a batch of ROCm test unskips.
[MoE] Make structured fused grouped-expert projection the default pytorch/torchtitan
Structured fused grouped-expert projection is now the default for routed experts, storing W13 as `[E, 2, F, D]` so the gate and up axes stay explicit and the feature axis stays independently shardable. Grouped GEMM consumes a zero-copy `[E, 2F, D]` view and returns `[R, 2, F]` before activation, so checkpoints and sharding plans built on the old layout need re-examination.
[graph_trainer] Unify graph execution with pipeline runtime (#4666) pytorch/torchtitan
GraphTrainer now expresses non-pipeline AOT execution as the smallest possible pipeline (`PP=1, VPP=1, one model stage`), reusing the pipeline schedule machinery for gradient accumulation and deferred FSDP gradient synchronization instead of a separate graph runner. Anyone relying on the old non-pipeline runner path should expect its behavior to route through the pipeline scheduler.
[MoE] Make routed expert projections first-class grouped linears pytorch/torchtitan
The opaque `RoutedExperts.inner_experts` owner is replaced by explicit `w13`, `w2`, `activation_fn` and `token_dispatcher` children, with a reusable bias-free `GroupedLinear` whose public configuration matches `Linear` (`out_features` names one projection's output width, `num_linears=2` keeps both). This is the plumbing that the new default layout sits on.
Fix FSDP hanging for multimodal models pytorch/torchtitan
A multimodal model that only runs its vision encoder when an image is present could hang FSDP: with DP=2 and an image on rank 0 only, rank 0 issues an all-gather that rank 1 never meets. The fix is implemented for Muse Glimmer, and the cc12m dataset never exposed the bug because its samples are always text plus image.
torch/_native: less per-call work in the chunked linear_cross_entropy loop (#196134) pytorch/pytorch
Three changes independent of the kernel arithmetic cut roughly 6% off the chunked `linear_cross_entropy` loop on an H1: `grad_linear_weight` is no longer zeroed before the loop because the first chunk writes it outright, the kernel emits each row's loss contribution instead of the two statistics it was built from, and the staging default becomes `(512, 8)`. Meanwhile the long tail stayed routin...
