$ the-wire · showcase
J-space direction loss and Jacobian lens joint ablation fixed in v4.3.0
By RepoJournal · Filed · About Interpretability · Composed from the cited sources · methodology
TransformerLens v4.3.0 fixes two measurement bugs that were quietly corrupting interpretability results for anyone projecting into sparse J-space or ablating multiple token directions at once.
fix: preserve selected J-space directions across atom scales TransformerLensOrg/TransformerLens
The default float32 rank cutoff in torch.linalg.pinv depends on ambient dimension and raw atom scale, so NNLS could select a genuinely independent direction and still report it as zero: with orthogonal atoms of norm 1 and 3000 in 4096 dimensions and target coordinates [2, 1], the reported J-space component was [0, 1]. Any sparse J-space decomposition with atoms on wildly different scales was si...
fix(jacobian_lens): ablate token directions jointly (#1941) TransformerLensOrg/TransformerLens
ablation_hooks removed each token's lens vector sequentially, h <- h - (h.v)v, which reintroduces components along earlier directions because lens vectors of related tokens are far from orthogonal. It now projects onto the orthogonal complement of their span, h <- h - Q Q^T h, so the ablated activation actually has zero projection onto all removed directions. If you used multi-token ablation to...
fix(bridge): discover non-standard block lists in _enumerate_blocks TransformerLensOrg/TransformerLens
_enumerate_blocks() only walked a fixed tuple of attribute names ("blocks", "encoder_blocks", "L_blocks" and friends), so architectures registering block lists elsewhere enumerated zero blocks with no error. On a Raven bridge, blocks_with(), composition_layer_indices() and attn_head_labels returned empty results; they now discover the lists properly.
Release v4.3.0 TransformerLensOrg/TransformerLens
v4.3.0 ships RelevanceLens.fit with provenance tracking, integration and docs, adds an svd_circuits demo notebook with a slow name-mover parity check, and fixes stateful generation fallback so sampling penalties are preserved. The two lens fixes above land in the same release.
Action items
- → Upgrade TransformerLens to 4.3.0 and rerun any J-space or Jacobian lens ablations done on earlier versions TransformerLensOrg/TransformerLens [plan]
