$ the-wire · showcase
ROCm handle reuse, silent losses in pytorch/rl, and lazy device guards on HIP 7.14
By RepoJournal · Filed · About PyTorch · Composed from the cited sources · methodology
12 people shipped this
theap06
@theap06
2 cited
Joshua Su
@joshuuuasu
1 cited
Chinmay Kuchinad
@chinmaydk99
1 cited
Andrey Talman
@atalman
1 cited
Felipe Mello
@felipemello1
1 cited
peterdsharpe
@peterdsharpe
1 cited
Bita Bayestehparvin
@bbayeste
Michael Lazos
@mlazos
jeanschmidt
@jeanschmidt
Siddartha Pothapragada
@psiddh
arun-elr
@arun-elr
francesco-bertolotti
@francesco-bertolotti
The day's work is concentrated in correctness: a HIP-specific device guard regression, six pytorch/rl bugs that corrupt or bias training without raising, and a tensordict fix that silently turns parameters into buffers.
[ROCm] Use the lazy MaybeSetDevice path on ROCm (#203037) pytorch/pytorch
c10's lazy device restore (MaybeSetDevice / MaybeExchangeDevice) is gated on CUDA_VERSION >= 12000, which hipify turns into an always-false TORCH_HIP_VERSION >= 12000, so ROCm always took the eager path; with HIP 7.14+ that means a device guard restore marks the target device's primary context active. The fix enables the lazy path on ROCm. No BC break.
[ROCm] Reuse a per-stream handle only on the stream it ran on (#202678) (#202678) pytorch/pytorch
hipBLASLt keeps the synchronizer state for single-kernel split-K solutions (GSUAMBSK) in the handle, not the workspace as cuBLASLt does, so a per-(thread, device, stream) handle could not serve two streams at once. The handle is now reused only on the stream it ran on. Anyone running matmuls through ROCm's BLASLt path across multiple streams is the affected population.
[BugFix] load_state_dict(assign=True) keeps TensorDictParams parameters as parameters when the checkpoint is on another device pytorch/tensordict
A TensorDictParams built with device="meta" or "cuda" followed by load_state_dict(..., assign=True) against a checkpoint on another device loads the right values but converts every parameter into a buffer. It drops out of model.parameters(), so an optimizer constructed afterwards never trains those tensors, and nothing raises.
[BugFix] Apply prioritized-replay weights in CQL, IQL, REDQ, CrossQ and QMixer losses pytorch/rl
CQLLoss, DiscreteCQLLoss, IQLLoss, DiscreteIQLLoss, REDQLoss, CrossQLoss and QMixerLoss write td_error priorities for prioritized replay but never read the importance-sampling weights back, so PER training with any of them is biased. Each now exposes a priority_weight key in _AcceptedKeys, matching SACLoss.
[BugFix] Keep workers from reading half-written weights in SharedMem weight sync pytorch/rl
With the default SharedMemWeightSyncScheme on a shared device, trainer parameters, the shared buffer and each worker's parameters resolved to one storage, so a worker could read weights mid-write. Separately, PrioritizedSliceSampler could return slices crossing episode boundaries on a wrapped ndim=2 storage, and IncrementalTokenizer kept tokens.prompt out of sync with history.prompt for tool-us...
[rl] Terminal-Bench: grade the way harbor run does (#5237) pytorch/torchtitan
The Terminus-2 harness stops the agent's tmux server before tests/test.sh runs, killing anything the agent started with a plain &; in a Qwen3.5-9B-Base run on TMax that zeroed up to 5.6% of rollouts, 51 of 904. Grading now keeps the server alive until scoring, as harbor run does. Elsewhere in torchtitan, vLLM generators get decode-specialized FULL CUDA graphs that are DP>1-safe, and CrossEntrop...
Validate binaries in uv venvs instead of conda envs (#9331) pytorch/test-infra
On the infra side, binary validation moved off conda entirely to a new .github/scripts/uv_env.sh (install_uv, create_uv_venv, remove_uv_venv) with uv pinned at 0.13.7, the HUD gained a /target_determinator page, and the TD ratings job now weights past failures by age.
Action items
