$ the-wire · showcase
ExecuTorch and TensorRT harden CUDA edge cases
By RepoJournal · Filed · About PyTorch
13 people shipped this
shoumikhin
@shoumikhin
3 cited
Natalia Gimelshein
@ngimel
2 cited
Nikita Shulga
@malfet
1 cited
albanD
@albanD
1 cited
Yuanyuan Chen
@cyyever
1 cited
SherlockNoMad
@SherlockNoMad
1 cited
abeakkas
@abeakkas
JakeStevens
@JakeStevens
digantdesai
@digantdesai
lanluo-nvidia
@lanluo-nvidia
jloftin-nv
@jloftin-nv
ivy-zhou
@ivy-zhou
acisseJZhong
@acisseJZhong
ExecuTorch and TensorRT shipped fixes for CUDA load failures, annotation leaks, and graph-lifting bugs, while torchdistributed and torchtitan clarified security and checkpoint handling.
A missing CUDA delegate weights blob used to fail only at first execute inside a generated kernel; ExecuTorch now fails at load with a message naming the blob and the expected constant count [1]. Three annotation-store findings from a large trainer led to a fix so a capture that fails after completing a mark_kernels scope no longer leaks annotations for the process lifetime, and clears the clear_kernel_annotations workflow [2]. Relatedly, make_graphed_callables no longer relies on the cyclic GC to free CUDA graphs, avoiding cudaGraphDestroy and friends mid-capture, which previously emitted "operation not permitted when stream is capturing" and invalidated the graph [3]. In TensorRT, exporting a partitioned graph that inlined multiple get_attr nodes could generate a forward with duplicate argument names causing a SyntaxError; constants are now lifted once per target [4]. A sidecar example now exercises a truly coalesced TensorRT plus CUDA program in the gate, because the previous x+1 test was handled entirely by TensorRT [5]. TorchTitan's GraphTrainer dropped its duplicated CUDA graph manager in favor of core TorchTitan's implementation, which also enables mark_kernels profiler annotations under cudagraph [6]. On the security side, PyTorch's SECURITY.md now documents that checkpointing and its format conversion utilities share the distributed-features trust model [7] and also clarifies that malformed deserialized objects and model files must be treated as untrusted [8]. ROCm removed version guards now that the minimum supported ROCm is 7.0 [9].
Action items
- → Upgrade ExecuTorch to include the CUDA delegate load fix if you rely on sidecar weight files pytorch/executorch [plan]
- → Review your TensorRT export paths for the constant-lifting fix if you use partitioned graphs pytorch/TensorRT [plan]
- → Review your security posture for checkpoints and model files given the clarified trust model pytorch/pytorch [monitor]
- → If you use the old GraphTrainer CUDA graph path in torchtitan, test with the new shared implementation pytorch/torchtitan [monitor]
References
- [1] Report NotFound when a CUDA delegate's weights blob is missing ↗ pytorch/executorch
- [2] Fix an annotation leak on failed capture, and retire the clear_kernel_annotations workflow (#195398) ↗ pytorch/pytorch
- [3] Free make_graphed_callables' CUDA graphs by refcount, not by the cyclic GC (#195397) ↗ pytorch/pytorch
- [4] Lift each constant once when exporting a partitioned graph ↗ pytorch/TensorRT
- [5] ci(executorch): run a coalesced TensorRT + CUDA program in the reference runner gate ↗ pytorch/TensorRT
- [6] Enable CudaGraph Annotations (#4359) ↗ pytorch/torchtitan
- [7] [SECURITY.md] Clarify that checkpoints share the distributed-features trust model (#195600) ↗ pytorch/pytorch
- [8] Update security guidelines in SECURITY.md (#195610) ↗ pytorch/pytorch
- [9] [ROCm][BE] Remove outdated version conditions (#193927) ↗ pytorch/pytorch
