$ the-wire · showcase
TransformerLens 4.3.0 lands with fixes for J-space projections, ablation, and bridge block discovery
By RepoJournal · Filed · About Interpretability · Composed from the cited sources · methodology
TransformerLens v4.3.0 shipped alongside fixes that correct two silently wrong analysis results and one silent empty-result path in the bridge, all of which affect numbers you may already have.
v4.3.0 TransformerLensOrg/TransformerLens
The v4.3.0 release heads the wire; the source gives no detail beyond the release notes link, so read those before pinning. Two of the fixes below are landing in this window.
fix: preserve selected J-space directions across atom scales TransformerLensOrg/TransformerLens
Sparse J-space projections used to report `[0, 1]` where the input reconstructed to `[2, 1]` with orthogonal atoms of norm 1 and 3000, because the default float32 `torch.linalg.pinv` rank cutoff scales with the ambient dimension and raw atom norms. Selected directions now survive scaling, so any J-space component numbers computed on unscaled atoms are worth recomputing.
fix(jacobian_lens): ablate token directions jointly TransformerLensOrg/TransformerLens
`JacobianLens.ablation_hooks` removed each token's lens vector one at a time, so with several related tokens the ablated activation kept a large projection along the earlier directions, a flaw the author notes it is the same defect as the duplicate entry 4 covering. It now projects onto the orthogonal complement of the span via `h - Q Q^T h`, so multi-token ablation runs measure what they claim...
fix(bridge): discover non-standard block lists in _enumerate_blocks TransformerLensOrg/TransformerLens
`_enumerate_blocks()` only walked a fixed tuple of names (`blocks`, `encoder_blocks`, `decoder_blocks`, `L_blocks`, `H_blocks`), so architectures like Raven/Huginn, which register `prelude`, `core_block` and `coda`, returned zero blocks with no error. On such a bridge `blocks_with()`, `composition_layer_indices()` and `attn_head_labels` now discover non-standard block lists instead of silently ...
Action items
- → Upgrade TransformerLens to v4.3.0 and re-run any sparse J-space or multi-token ablation analyses made before this fix TransformerLensOrg/TransformerLens [plan]
