$ the-wire · showcase
NCCL split/shrink init failures now surface, torchtitan lands an RL KV-cache fix
By RepoJournal · Filed · About PyTorch · Composed from the cited sources · methodology
11 people shipped this
Tristan Rice
@d4l3k
2 cited
Sherlock Huang
@SherlockNoMad
1 cited
wwwjn
@wwwjn
1 cited
acisseJZhong
@acisseJZhong
1 cited
eyad qasim
@eyad6789
1 cited
theap06
@theap06
1 cited
IvanKobzarev
@IvanKobzarev
zhaiyaxing001
@zhaiyaxing001
Jean Schmidt
@jeanschmidt
morrison-turnansky
@morrison-turnansky
Huy Do
@huydhn
The day's work is mostly correctness plumbing: a c10d race that hid communicator init failures, torchtitan's first real answer to stale KV after policy updates, and a silent trajectory-id collision in torchrl's Collector.
[c10d][nccl2] Wait for nonblocking split/shrink child initialization (#199263) pytorch/pytorch
Nonblocking ncclCommShrink and ncclCommSplit return ncclSuccess while the child communicator is still initializing on NCCL's async thread, and the child's own error state is not visible through the API, so waitForNcclChildComm only polled the parent. That gap is closed; distributed jobs that split or shrink communicators can now observe an init failure instead of proceeding against a half-initi...
[RL] Preserve per-group KV cache across policy updates (#4761) pytorch/torchtitan
Policy updates change the weights but do not invalidate vLLM's cached KV, and clearing everything preempts active requests and forces long multi-turn rollouts to re-prefill accumulated context. torchtitan adds two explicit KV-cache lifecycle modes so newly started rollouts can steer clear of stale prefix state without discarding active work.
[BugFix] Collector: fresh trajectory ids on reset_at_each_iter and reset() pytorch/rl
reset_at_each_iter=True reset the env each batch without assigning new ids, so with fixed-size batches separate episodes merged into one trajectory in a replay buffer, and with trajs_per_batch the leftover step from an interrupted episode was glued onto the next one. Collector now issues fresh trajectory ids on reset_at_each_iter and reset().
[Perf][Compile] Add local compilation for OffsetRMSNorm pytorch/torchtitan
An offset_rmsnorm local-compile target for Qwen3.5-compatible models is enabled by default alongside gated_rmsnorm, compiling the existing FP32 OffsetRMSNorm expression with TorchInductor while keeping the eager implementation and state dict intact. Batch-invariant mode still falls back to eager, and the Triton override remains available.
[graph_trainer] Pass one config through GraphRuntime construction (#4983) pytorch/torchtitan
GraphRuntime construction threaded compile_config, model_config, parallelism and trainer_config side by side, even though on PP=1 the first three are always derived from trainer_config. That duplication is gone; PP>1 still enters through core's generic model.pipeline(**kwargs) API with only the decomposed fields.
[elastic] Fix lost-wakeup races that truncate tailed log output (#195211) pytorch/pytorch
tail_logfile() sampled the finished event after inspecting the file in both of its loops, so a file created and exited inside that window was never tailed at all and a last line written inside it was dropped. The elastic tailer now checks the event before acting.
[distributed] Drop redundant per-test supervisors in run_test.py (#198059) pytorch/pytorch
The test-infra long tail: --subprocess in run_test.py is dropped where suites already spawn fresh rank processes per test, and TestComputeCommReorderingBucketing and TestManualOverlapBucketing stop re-running ten inherited tests three times, cutting collection from 65 to 45 cases and about 310s of reported test time in sampled runs.
