$ the-wire · showcase
Gemma Scope 2 affine transcoders loaded wrongly, SAELens sparsity stats off by one
By RepoJournal · Filed · About Interpretability · Composed from the cited sources · methodology
3 people shipped this
Two correctness fixes land in SAELens: every Gemma Scope 2 `*_affine` transcoder was loading with its skip connection transposed and producing wrong output silently, and the sparsity-stat reset was firing a step early.
fix: load Gemma Scope 2 affine skip connections in the correct orientation jbloomAus/SAELens
Gemma Scope 2 stores `affine_skip_connection` as (d_in, d_out) and applies it as `x @ A`, while `SkipTranscoder` holds `W_skip` as (d_out, d_in), so the loader copied the matrix over untransposed and the skip connection ran backwards. Because the matrices are square nothing raised; every `*_affine` transcoder loaded cleanly and returned wrong numbers, so re-run any analysis you did on those SAEs.
fix: reset sparsity stats after a full feature_sampling_window of steps jbloomAus/SAELens
`SAETrainer.maybe_reset_sparsity()` checked `(n_training_steps + 1) % feature_sampling_window == 0`, so the first window only ever held `feature_sampling_window - 1` steps and with a default window of 2000 and 30000 total steps the running stats were wiped immediately before the final step. Sparsity figures logged from affected runs are computed over the wrong window.
Preserve disabled whitening when saving activation scalers jbloomAus/SAELens
Saving a scaler with whitening disabled into a directory that already held an older `activation_whitening.safetensors` would silently restore the stale whitening transform on the next load, so the reloaded scaler no longer matched what you saved. The scaler JSON now persists whether whitening is enabled and loading honors that flag; older JSON files without it keep the existing behavior.
fix: reset sparsity stats after a full feature_sampling_window of steps (#757) jbloomAus/SAELens
The sparsity-reset correction is merged as the commit behind PR 757, so it is in the tree rather than queued.
Action items
