$ the-wire · showcase
Tensordict's silent-wrong-answer bugs, and torchtitan opens multimodal SFT
By RepoJournal · Filed · About PyTorch · Composed from the cited sources · methodology
The day's most consequential changes are correctness fixes in tensordict, where three separate defects returned plausible but wrong results with nothing raised, alongside a torchtitan PR that finally makes VQA-style chat SFT trainable.
[BugFix] clone() of a tensordict that has a device keeps -0.0 when its leaves are all floating pytorch/tensordict
Cloning a TensorDict or tensorclass built with a device, or returned by to(device), turns every -0.0 of a floating leaf into 0.0 across float16, bfloat16, float32 and float64, on CPU and CUDA. `1 / td.clone()["x"]` is then inf where `1 / td["x"]` is -inf; the same tensordict without a device clones bit for bit.
[BugFix] TensorDictParams computes with the loaded weights after load_state_dict(assign=True) pytorch/tensordict
A model that holds parameters in a TensorDictParams and calls them through torch.vmap keeps computing with the old weights after load_state_dict(checkpoint, assign=True), if it ran one forward first. parameters() and params[key] show the loaded values and nothing raises; TorchRL's locked-parameter multi-agent networks hit it with a MultiAgentMLP(share_params=False).
[BugFix] flatten_keys(is_leaf=f) on a locked tensordict no longer returns another function's result pytorch/tensordict
On a locked tensordict, two calls to flatten_keys(is_leaf=...) with two functions made on the spot (lambdas, closures, bound methods, functools.partial) can return the first function's keys for the second call, with no error. This is a regression in 0.15.0, so pin or upgrade if your code flattens with per-call leaf predicates.
[data] Multimodal chat SFT: image in the user turn, loss on the answer only (#5885) pytorch/torchtitan
ChatProcessor was text-only and raised on any media, and the multimodal processors put loss on all text rather than on chat turns. This adds image-only multimodal chat SFT, so a VQA-style conversation with an image in the user turn trains with the loss on the answer only.
[lora] Keep selected modules fully trainable in LoRATransform pytorch/torchtitan
LoRATransform froze every non-target config with no way to keep a subtree trainable. Selected modules can now stay fully trainable, which lets a recipe fine-tune the vision encoder end to end while LoRA is applied to the decoder.
[Performance] update_ and copy_ from a lazy stack no longer stack its leaves into a temporary pytorch/tensordict
Also in tensordict: dense.update_(lazy) and dense.copy_(lazy) had stacked every leaf of a lazy stack into an unused temporary since 0.15.1, taking 126 ms instead of 20 ms and adding 257 MiB of peak memory with 3 leaves of 128 MiB. The data written was correct throughout; the fix removes the temporaries.
Action items