The Wire · Showcase
PYTORCH FUSES MATMUL REDUCTION INTO TRITON, FIXES HYDRA 1.4 COMPATIBILITY
By RepoJournal · Filed · About PyTorch
PyTorch's distributed matmul just got faster with a kernel fusion that eliminates the final memory-bandwidth bottleneck, while TorchRL and TensorDict harden their infrastructure across the stack.
The big move: a Triton kernel fusion in `fused_matmul_reduce_scatter` eliminates the separate reduction kernel that was blocking communication [1]. Previously, the op computed per-destination GEMM partials, exchanged them all-to-all, then hit a bandwidth wall with a final `torch.sum` or `torch.mean` over stacked partials. Now that reduction happens inline. TorchRL is cutting ties with Hydra 1.1 and bumping the floor to version 1.3 to prepare for Hydra 1.4 [2], with all entry points now passing `version_base="1.3"` and configs gaining explicit `hydra.job.chdir: true` directives. TensorDict hardened memmap path handling across nested directories and dynamic subtrees, applying robust filesystem encoding consistently while preserving backward compatibility for single-component names [3]. ExecutorTorch fixed a CI image pollution bug where PEP 517 build requirements were being permanently installed and changing downstream builds, breaking all 17 RISC-V jobs [4]. AOTAutograd fixed CSE deduplication of NaN constant tensors by normalizing float and complex hashing using IEEE 754 bit patterns [5], and TensorDict locked down GitHub automation inputs to distinguish repository-owned runs from fork branches [6].
One email a day. Unsubscribe in one click.
Keep up with PyTorch in about 3 minutes a day: what actually shipped — the commits, pull requests, releases, and security advisories that matter.
One email a day. Unsubscribe in one click. Read a past issue →
Action items
- → Upgrade TorchRL dependencies to Hydra >=1.3 before next release pytorch/rl [plan]
- → Test matmul_reduce_scatter fusion in your distributed training pipeline for bandwidth wins pytorch/pytorch [monitor]
- → Verify ExecutorTorch CI environment after PEP 517 fix lands pytorch/executorch [plan]
References
- [1] Fuse the final reduction in matmul_reduce_scatter into one Triton kernel (#191803) pytorch/pytorch
- [2] [Versioning] Migrate Hydra entry points to version_base 1.3 for Hydra 1.4 readiness ↗ pytorch/rl
- [3] [BugFix] Harden memmap path handling ↗ pytorch/tensordict
- [4] Build PyTorch wheel via PEP 517 without poisoning the CI image (#21685) pytorch/executorch
- [5] [AOTAutograd] Fix CSE to deduplicate NaN constant tensors by normalizing float/complex hashing and comparison (#191173) pytorch/pytorch
- [6] [CI] Harden GitHub automation inputs ↗ pytorch/tensordict