$ the-wire · showcase
NVFP4 kernel pipeline gets PDL, fused scaling, and tuned autotuning; torchtitan drops stale train command
By RepoJournal · Filed · About PyTorch · Composed from the cited sources · methodology
11 people shipped this
Michael Lazos
@mlazos
4 cited
Yeonwoo Sung
@YeonwooSung
1 cited
Utkarsh Bahuguna
@u7k4rs6
1 cited
Sherlock Huang
@SherlockNoMad
1 cited
aniketpandey05
@aniketpandey05
1 cited
Ivan Kobzarev
@IvanKobzarev
sanketpurandare
@sanketpurandare
N0AHZACH
@N0AHZACH
Huy Do
@huydhn
Jeff Daily
@jeffdaily
Benson Ma
@q10
Inductor's NVFP4 path picked up four coordinated changes that together cut launch overhead and make autotune decisions reflect production CUDA-graph execution, while torchtitan quietly retired its Tyro-era entry point and CohenKappa's silent checkpoint bug got a fix.
[inductor][NVGEMM] Chain dependent NVFP4 launches into Triton consumers (#197485) pytorch/pytorch
NVGEMM kernels can now take part in CUDA programmatic dependent launch, so direct Triton consumers overlap their launch setup with the producer tail instead of waiting for ordinary kernel completion. Selective PDL is enabled for measured NVFP4 decode shapes.
[inductor][NVGEMM] Fuse NVFP4 output scaling (#194651) pytorch/pytorch
ModelOpt NVFP4 linear layers used to materialize a separate pointwise multiply after `_scaled_mm`; the dequantization factor is now folded before the QKV fan-out, removing one standalone scale launch per layer and widening the GEMM implementations autotuning can consider.
[inductor][NVGEMM] Improve NVFP4 autotuning (#194652) pytorch/pytorch
`nvMatmulHeuristics` did not cover several Blackwell NVFP4 decode winners, while an unrestricted supplemental pool added compile and profiling cost. Short inference GEMMs now use a small automatic candidate set measured to reflect transformer decode execution.
[inductor] Make CUDA graph autotuning measurements consistent (#194649) pytorch/pytorch
Autotune requests created under one Inductor configuration and executed later in another process could compare candidates measured under different launch policies. The effective CUDA-graph policy now travels on the request, so every candidate in one decision is measured the same way.
Split WANDB_RUN_TAGS into a list before passing to wandb (#4965) pytorch/torchtitan
`WANDB_RUN_TAGS` was passed straight into `wandb.init` as a plain string, and wandb's `tuple(tags)` turned `"exp1,baseline"` into one tag per character. The env var is now split on commas with empties and whitespace dropped; unset or empty still means no tags.
[graph_trainer] Drop "schedule" FSDP modes for SPMD gradient accumulation (#5024) pytorch/torchtitan
The `"schedule"` values for `fsdp_param_unshard_mode` and `fsdp_grad_reduce_mode` extracted FSDP all-gather and reduce-scatter out of the PP=1 joint graph into standalone schedule actions. They are now dropped for SPMD gradient accumulation, so any config still setting them will need to move off schedule mode.
[docs] Replace the removed torchtitan_train command (#5013) pytorch/torchtitan
Override examples still pointed at `torchtitan_train`, which stopped being a script, entry point, or module after the config loader landed, so copy-pasted examples fail today. The working invocation is `python -m torchtitan.train`, with the same `--module`, `--config`, and `--override` flags including the quoted JSON override form.
Fix CohenKappa losing its state on checkpoint resume pytorch/ignite
`CohenKappa` delegates all state to `self._impl` but never declared it in `_state_dict_all_req_keys`, so `state_dict()` returned an empty mapping and `load_state_dict()` appeared to succeed before `compute()` raised `NotComputableError`. The fix restores the accumulated state on resume, and two tests cover it.
Action items
- → Migrate torchtitan override examples and scripts off `torchtitan_train` to `python -m torchtitan.train` pytorch/torchtitan [plan]
- → Drop `fsdp_param_unshard_mode="schedule"` and `fsdp_grad_reduce_mode="schedule"` from torchtitan configs using SPMD g... pytorch/torchtitan [plan]
