$ the-wire · showcase
JAX REMAT3 OVERHAUL CUTS MEMORY PRESSURE WITH SELECTIVE OFFLOADING
By RepoJournal · Filed · About Google · Composed from the cited sources · methodology
1 person shipped this
JAX is shipping a fundamental redesign of its rematerialization system that lets you offload specific activations to slower storage during training, reclaiming GPU memory without the checkpointing tax.
The remat3 refactor hits production with two major pieces: support for save_and_offload_only_these_names [1] and offload_dot_with_no_batch_dims [2], both defunctionalizing previous approaches into dataclass policies. The first lets you name exactly which residuals get device_put to offload storage on the forward pass and restored on the backward pass. The second does the same for matrix multiplications without batch dimensions, offloading their outputs while rematerializing any batched dots as before. This keeps the remat2 functional interface intact [1] so your existing code doesn't break. Meanwhile, custom_vjp3 now handles unhashable nondiff_argnums [3], closing a gap that was blocking some VJP patterns. The wins here are real: you trade slow storage I/O for GPU memory headroom on large training runs without recomputation overhead.
Action items
- → Review remat3 offloading docs and test against your largest models google/jax [plan]
- → Check custom_vjp3 patterns if you're using unhashable arguments google/jax [monitor]
References
- [1] [remat3] support offloading via save_and_offload_only_these_names google/jax ↗
- [2] [remat3] support offloading via offload_dot_with_no_batch_dims ↗ google/jax
- [3] [custom_vjp3] handle unhashable nondiff_argnums in custom_vjp3 ↗ google/jax
