$ the-wire · showcase
AsyncEnvPool shared-memory fixes, CUDA 12.6 nightly removal, and stale autotune Future collision
By RepoJournal · Filed · About PyTorch
10 people shipped this
PyTorch RL's AsyncBatchedCollector gets a compile-safe pause and a batch coordination rework, while the core repo stops publishing CUDA 12.6 nightlies.
Three fixes land in pytorch/rl's async collection stack. AsyncEnvPool queue results are now copied out of multiprocessing shared memory before returning, with lazy-stacked results copied while dense stacks skip the extra copy, fixing a slice-on-data race [1]. A new `AsyncBatchedCollector.pause()` context manager forces a quiescent point with a coordinator barrier, letting you compile between collects without deadlock; prefer compile-before-collect and use `pause()` only when lazy compilation is unavoidable [2]. AsyncBatchedCollector also now coordinates shared-memory environments from a single thread, batching ready requests and draining inference slots fairly, and returns an exact final frame batch even when `total_frames` is not divisible by `frames_per_batch` [3]. CPU affinity controls for worker and driver processes were added, applied before environment construction [4]. [1]
CUDA 12.6 nightly wheels disappear from pytorch/pytorch and pytorch/test-infra starting with the PyTorch 2.15 drops [5][6]. Only the builds are removed; code cleanup comes after. The referenced dev-discuss notice notes that 2.15 drops Maxwell, Pascal, and Volta support, so teams on those architectures should watch for the transition [5]. Separately, core fixes a stale autotune Future collision: `AsyncAutotuner.choice_hash_to_future` keyed benchmarks by shape and config only, so a second compilation in the same process silently reused the first compilation's Futures, causing errors if that first compilation aborted or cleared its cache dir [7]. The partitioner also no longer crashes when it encounters a `torch.device` node in metadata, which `compile_on_one_rank` can produce for a baked accelerator device operand [8].
In executorch, the NXP backend's op alias table has moved out of its test package. Three production modules read that table at import time, so the backend could not load at all without the test package; two dozen production files import those modules [9]. The Arm backend's rescale pass had the opposite problem: `InsertRescaleInt32Pass` assumed quantization metadata meant full integer conversion, but partially quantized binary operators preserve their Q/DQ boundaries and still execute in floating point, so the pass misidentified them [10]. Qualcomm's Windows ARM64 QNN backend now has CI validation with a new `build-qnn-windows-arm64` job that uses `venv` instead of `conda` [11]. In tensordict, compiled shallow copies now preserve independent nested TensorDict container structure while keeping the tensor leaves shared, matching eager semantics [12]. Test-infra's `r2_promote` got three fixes after the 2.14.0 promotion produced R2 throttling errors: it now caps concurrent multipart upload parts and total throughput [13].
"The dependency points the" [9] - from the executorch commit.
All other desks were quiet. Ship with confidence.
Action items
- → Update CI that pulls PyTorch nightly wheels for CUDA 12.6, and watch for the PyTorch 2.15 release that drops Maxwell/Pascal/Volta support pytorch/pytorch [immediate]
- → If you compile between collect steps in AsyncBatchedCollector, prefer compile-before-collect or use collector.pause() when lazy compilation is unavoidable pytorch/rl [plan]
- → Upgrade to a build that includes #196148 if you run multiple torch.compile jobs in one process that share shapes pytorch/pytorch [plan]
- → After NXP backend reload, verify your executorch wheel no longer depends on the test package for import pytorch/executorch [monitor]
References
- [1] [BugFix] Bound AsyncEnvPool queue shared mappings ↗ pytorch/rl
- [2] [Feature] Add compile-safe pause to AsyncBatchedCollector ↗ pytorch/rl
- [3] [Performance] Batch AsyncBatchedCollector shared-memory coordination ↗ pytorch/rl
- [4] [Feature] Add CPU affinity controls to AsyncEnvPool ↗ pytorch/rl
- [5] [CD] Remove CUDA 12.6 nightly binary builds (#196249) ↗ pytorch/pytorch
- [6] Remove cuda 12.6 support CD ↗ pytorch/test-infra
- [7] Fix stale autotune Future reuse across compilations (#196148) ↗ pytorch/pytorch
- [8] Fix partitioner crash on device-valued node metadata (#196146) ↗ pytorch/pytorch
- [9] Move the NXP op alias table out of the test package ↗ pytorch/executorch
- [10] Arm backend: Preserve Q/DQ in rescale pass ↗ pytorch/executorch
- [11] Qualcomm AI Engine Direct - Enable QNN Windows ARM64 build in CI ↗ pytorch/executorch
- [12] [Compile] Preserve nested structure in shallow TensorDict copies ↗ pytorch/tensordict
- [13] Fix R2 promotion: throttling, throughput, and R2_ONLY scope ↗ pytorch/test-infra
