$ the-wire · showcase
ROCm test timeouts traced to duplicated runs, and distributed/test_store drops 15 minutes
By RepoJournal · Filed · About PyTorch · Composed from the cited sources · methodology
14 people shipped this
Jeff Daily
@jeffdaily
3 cited
Huy Do
@huydhn
1 cited
SongyuanZhao
@SongyuanZhao
1 cited
Christian Aurich
@christianaurichzm
Bob Ren
@bobrenjc93
Oscar Andersson
@oscarandersson8218
Sebastian Larsson
@Sebastian-Larsson
mcremon-meta
@mcremon-meta
Gasoonjia
@Gasoonjia
yinli-systems
@yinli-systems
IvanKobzarev
@IvanKobzarev
JavaZero
@JavaZeroo
jeanschmidt
@jeanschmidt
pytorchbot
@pytorchbot
The day's PyTorch work is mostly CI economics: shaving a 15-minute shard cost off test_store, stopping inductor files from being discovered twice on ROCm default shards, and unskipping ROCm tests that no longer fail.
[CI] Fix retry teardown and run distributed/test_store in-process (#197324) pytorch/pytorch
distributed/test_store.py was paying Python startup plus import torch once per test case, roughly 15 minutes of wall clock per shard for 140 tests that take 24 seconds in-process; the fix restores in-process execution only after repairing the test-level state leaks that justified the isolation.
Stop double-running inductor tests on ROCm default shards (#197313) pytorch/pytorch
The linux-noble-rocm-py3.11-mi300 default shards were running near their 270-minute step timeout and main-branch timeout failures rose from about 1 percent through August to 5.6 percent the week of 2026-09-06. Test counts per shard have been flat since July: the default config selects files by discovery and so picks up every inductor/* file that the inductor config already runs.
[ROCm] Unskip assorted GPU tests and fix two ROCm arch skips (#197340) pytorch/pytorch
Five ShardedTensor tests that do not use RPC lost their skip (it was carried over from the RPC-based tests when the skips were added), along with test_float8_scale_result, the semi-structured cutlass_mm functionalization test that only exercises FakeTensorMode, and test_var_mean and test_cpu_scalar_with_gpu_tensor_cpp in test_torchinductor.
Fix SimpleFSDP metadata for uneven composed sharding pytorch/torchtitan
SimpleFSDP applied DP sharding using full global tensor metadata on a TP/EP-local tensor, and implicit shape inference could not recover uneven shard sizes: a TP-local dimension of 5 across two DP ranks yields shards of 3 and 2, but inferring the global size from either shard gives 6 or 4. The fix uses the local tensor's shape and stride for the temporary DP redistribution.
Right-size the 1-GPU GPU runners, and add 4xlarge-sized ones pytorch/ci-infra
The 1-GPU g5/g6 runners l-x86aavx2-29-113-a10g, l-x86aavx2-29-113-l4 and rel-l-x86aavx2-29-113-l4 requested a 113Gi job that does not fit the 8xlarge they name, leaving a release runner Pending and pytorch/vision nightly CUDA wheels queued, and new 4xlarge-sized runners are added alongside the correction.
