$ the-wire · showcase
TensorRT drops duplicate plan copies, torchtitan compiles RoPE
By RepoJournal · Filed · About PyTorch · Composed from the cited sources · methodology
11 people shipped this
The day's work is about host memory and kernel boundaries: TensorRT removed three separate copies of the engine plan, torchtitan turned RoPE into a compiled region with measured prefill and train gains, and tensorclass closed the gap between its dataclass signature and its runtime.
perf(runtime): build C++ engines from a view of the plan instead of copies pytorch/TensorRT
TorchBind's conversion of the plan bytes to `std::vector<std::string>` copied large plans twice; the new `Engine.from_engine_tensor(info, tensor)` deserializes from a uint8 CPU tensor that views the caller's buffer. The PR reports setting up a 3 GiB engine peaked at 6 GiB of extra host memory and took 1.7 s, against 0.14 s for TensorRT deserializing the same bytes directly.
perf(conversion): hand the built engine's plan to the runtime without copying it pytorch/TensorRT
`interpret_module_to_result` now keeps TensorRT's `IHostMemory` and lets the runtime read the plan in place rather than copying it into `bytes` through a `BytesIO`; the engine-cache hit path got the same treatment. Callers that still need bytes convert explicitly, including `convert_exported_program_to_serialized_trt_engine`'s return value and the tuning sweeper's pickled trial results.
[rope][perf] Compile ComplexRoPE in real arithmetic (rope region) pytorch/torchtitan
Compiling RoPE in real arithmetic turns the `rope` region on by default for Llama 3, DeepSeek-V3 (Kimi K2.x inherits it), Muse Spark and Muse Glimmer. `rope` replaces the `complex_rope` and `cos_sin_rope` region names, and configs listing either old name now raise on the unknown name until switched.
[dsv3][perf] MLA RoPE: rotate q's rope dims within the row, as int32 pairs (#5529) pytorch/torchtitan
In the `mla_qk` region, q's rope dims are rotated within the contiguous q row and read and written as int32 pairs instead of being split off as a strided view. The author measures -30% to -32% prefill and -34% to -36% train for the DeepSeek-V3 and Kimi K2.5 q/k RoPE glue, at 89 to 99.6% of the bandwidth floor and bitwise with eager on GB300.
[Deprecation] Let int, float and bool tensorclass fields take a number with any batch size pytorch/tensordict
An `n: int = 3` field currently stores `tensor(3)` under an empty batch size and raises a batch-size mismatch under a non-empty one, so a call like `A(x=torch.zeros(4), batch_size=[4])` fails on its own default. From TensorDict 0.17 the plan on #1199 is to keep such values as `NonTensorData`, which makes today's tensor-valued fields a migration.
[Feature] Accept InitVar, field(repr=False) and kw_only=True in tensorclasses as in dataclasses pytorch/tensordict
Three dataclass habits failed on main: an `InitVar` raised `AttributeError: ... has no such field` when given, `field(repr=False)` was ignored and left noisy fields in the printout, and `class Config(TensorClass, kw_only=True)` raised `TypeError`. All three now behave as in `dataclasses`.
Build and ship the OpenVINO backend in the Windows wheels (#23997) pytorch/executorch
The OpenVINO delegate, which runs models on Intel CPUs, GPUs and NPUs, was previously only in the Linux wheels; the Windows CPU and CUDA wheels now build it too, so `pip install "executorch[openvino]"` on Windows x64 no longer requires a source build.
Action items
