The Wire · Showcase
JAX FIXES CACHE KEY BUGS BREAKING DISTRIBUTED TRAINING ACROSS ENVIRONMENTS
By RepoJournal · Filed · About Google
JAX's compilation cache is now shipping with environment-agnostic keys, fixing a critical issue where autotune paths were poisoning builds across machines.
The Mosaic kernel cache fix [1] strips debug info from TPU custom calls that were bloating cache keys and causing unnecessary recompilation. This lands alongside a collective operations fix [2] that corrects a regression in psum and pmean when handling uneven axis index groups with constants. The real impact: distributed training jobs that span multiple hardware configs will stop thrashing the autotune cache on every run. JAX also bumped minimum version floors [3], dropping support for jaxlib_extension versions below 452 and libtpu below 0.0.41. Breaking change in Pallas: `pl.program_id` no longer works inside `plgpu.kernel` [4], but the migration path is clean - swap to `lax.axis_index` and `lax.axis_size`. On the Python SDK side, google-cloud-python dropped Python 3.7-3.9 support [5] and fixed a pubsub generation issue [6] that was blocking client library builds. The GenAI SDK added reinforcement tuning support [7], moving one step closer to a complete fine-tuning API.
Action items
- → Pin JAX to the latest commit with cache key fix before scaling distributed jobs jax-ml/jax [plan]
- → Update google-cloud-python past version supporting Python 3.7-3.9 removal googleapis/google-cloud-python [plan]
- → If using Pallas multi-GPU kernels, migrate pl.program_id to lax.axis_index jax-ml/jax [monitor]
References
- [1] PR #35699: Strip Mosaic kernel debug info and exclude autotune cache dir from cache keys google/jax
- [2] Add tests for uneven axis index groups in collectives. ↗ google/jax
- [3] Remove stale version tests. google/jax
- [4] [pallas:mgpu] `plgpu.kernel` is now implemented in terms of `pl.kernel` ↗ google/jax
- [5] chore: remove references to Python 3.7, 3.8, and 3.9 (#17227) googleapis/google-cloud-python
- [6] fix: resolve issue with pubsub generation (#17219) googleapis/google-cloud-python
- [7] feat: Support Reinforcement Tuning in GenAI SDK ↗ googleapis/python-genai
FAQ
- What changed in Google on May 23, 2026?
- JAX's compilation cache is now shipping with environment-agnostic keys, fixing a critical issue where autotune paths were poisoning builds across machines.
- What should Google teams do about it?
- Pin JAX to the latest commit with cache key fix before scaling distributed jobs • Update google-cloud-python past version supporting Python 3.7-3.9 removal • If using Pallas multi-GPU kernels, migrate pl.program_id to lax.axis_index
- Which Google repositories shipped on May 23, 2026?
- google/jax, googleapis/google-cloud-python, googleapis/python-genai