$ the-wire · showcase
TorchRL gains native GRPO training; TensorRT fixes NCCL rebind crash
By RepoJournal · Filed · About PyTorch
15 people shipped this
coder-jayp
@coder-jayp
1 cited
quinnarnold
@quinnarnold
1 cited
vmoens
@vmoens
1 cited
Albert Malewski
@albmalamd
1 cited
cyy
@cyyever
1 cited
apbose
@apbose
1 cited
tp5uiuc
@tp5uiuc
1 cited
calebmkim
@calebmkim
MrlixiangWE
@MrlixiangWE
AmesingFlank
@AmesingFlank
Jean Schmidt
@jeanschmidt
chunhuanMeng
@chunhuanMeng
Bin Bao
@desertfire
shoumikhin
@shoumikhin
sibuachu
@sibuachu
TorchRL shipped a native GRPOTrainer with SOTA execution pipelines, and TensorRT fixed a silent output mismatch from a lost NCCL communicator.
TorchRL's new GRPOTrainer brings native RLHF support into torchrl/trainers/algorithms, reusing the standard Trainer's data collection and hook logic, while GRPOOptimizationStepper decouples autocast, gradient accumulation, and clipping from the main loop [1]. The same repo tightened SFT loss coverage with numerical tests that verify normalization, mean/sum reductions, and the KL-to-reference term, separating MinorSFT's reference-log-probability behavior from ordinary SFT [2]. A DreamerV3 block-GRU Triton backward optimization avoids expanding projected features and accumulates weight gradients directly from BF16/FP32 operands, preserving the public API and checkpoint layout [3]. TensorRT fixed a multi-device bug where disable_profiling() and set_resource_allocation_strategy() dropped the execution context without clearing nccl_initialized, leaving collectives with no communicator and causing roughly 98% output mismatch; the lazy re-bind now fires to restore the communicator [4]. TensorRT also made engines default to no runtime cache so an implicit cache owned by TorchTensorRTModule no longer gets freed out from under a module-less engine [5]. PyTorch core cleaned up a post-local SGD checkpoint reload race where rank 0 deleted the temporary file before other ranks opened it, now serializing the checkpoint in a memory buffer [6], and collapsed ivalue::from's tag dispatch to fix a misleading ambiguous-constructor error [7].
Action items
- → Review TorchRL's new GRPOTrainer if you use RLHF training workflows pytorch/rl [plan]
- → Update TensorRT if you use multi-device engines; rebinding prevents silent output mismatches pytorch/TensorRT [immediate]
- → Monitor TensorRT engine cache behavior for module-less engine builds pytorch/TensorRT [monitor]
References
- [1] [Trainer] Implement GRPOTrainer and SOTA execution pipelines ↗ pytorch/rl
- [2] [Quality] Add numerical SFT loss tests ↗ pytorch/rl
- [3] [Performance] Optimize DreamerV3 block-GRU Triton backward ↗ pytorch/rl
- [4] [fix] MD-TRT: rebind NCCL communicator after execution context invalidation ↗ pytorch/TensorRT
- [5] fix: engines default to no runtime cache; the module owns the implicit one ↗ pytorch/TensorRT
- [6] Fix post-local SGD checkpoint reload race (#194614) ↗ pytorch/pytorch
- [7] Collapse ivalue::from's tag dispatch into one function (#195105) ↗ pytorch/pytorch
