$ the-wire · showcase
TorchTitan drops pure TP on routed experts, restacks fused FFN weights
By RepoJournal · Filed · About PyTorch · Composed from the cited sources · methodology
TorchTitan's MoE path now requires expert parallelism at least as wide as dense TP, while the fused W13 projection moves to a stacked layout so block quantization stops forcing W1 and W3 to share scales.
[MoE] Deprecate pure TP on routed experts pytorch/torchtitan
Routing experts through EP of at least the dense tensor-parallel degree removes duplicated routed-expert computation, keeps token activations sequence-sharded, and leaves shared experts and dense layers tensor-parallel, with the shared-expert path gathering its input and reduce-scattering its output into the routed-expert layout. Token-count reductions, the affected model layouts, and the docs ...
Use stacked layout for fused feed-forward projections pytorch/torchtitan
W13 now stores W1 and W3 as a [2, F, D] tensor with a generalized Linear taking num_linears, instead of interleaved rows in a [2F, D] tensor. The old layout worked well for TP but let block-quantization tiles span both projections and force them to share scales; the stacked layout keeps each projection separate. The change is the reviewed #4676 rebased onto main, after the original accidentally...
Route all triton_meta device properties through a CooR-aware helper (#196149) (#196149) pytorch/pytorch
compile_on_one_rank requires byte-identical generated code across ranks, so triton_meta must not carry a rank-specific device index; that drop existed only in TritonKernel while three other construction sites in select_algorithm.py and triton_combo_kernel.py baked in DeviceProperties(type='cuda', index=N). A CooR-aware helper now routes all of them. This matters mainly if you rely on compile_on...
[precompile] Complete the guard-type classification and pin it with a totality test (#197268) pytorch/pytorch
The guard-type classification is now complete and pinned by a totality test: _IDENTITY_GUARD_TYPES freezes CheckFunctionManager.UNSUPPORTED_SERIALIZATION_GUARD_TYPES, and _INVARIANT_DROPPABLE_GUARD_TYPES is the only set an invariance policy may drop from a serialized artifact. Together with the shape-bearing set from the previous commit, the two properties enforced are never dropped and never c...
[precompile] Add _is_risky_drop, the lint over identity guards serialization dropped (#197266) pytorch/pytorch
Serialization drops every identity guard, so precompiling gives up on noticing a guarded object was rebound; _is_risky_drop(entry, namespaces) is the lint that flags which drops can plausibly change results on a capture-here, serve-there deployment, where the same source diverges because a config, flag, or environment variable picks a different path. The other tail of the day is small: test_exp...
