$ the-wire · showcase
TransformerLens 4.2.0 fixes Cohere bridge and optimizer parity
By RepoJournal · Filed · About Interpretability · Composed from the cited sources · methodology
TransformerLens 4.2.0 ships three fixes that change what operators get out of the box: correct Cohere RoPE alignment in the HF bridge, a synthetic bias that no longer lands in the optimizer, and tie-aware sparse probe metrics.
fix(bridge): rotate Cohere and Cohere2 with their own RoPE convention TransformerLensOrg/TransformerLens
CohereForCausalLM and Cohere2ForCausalLM previously did not match Hugging Face through the bridge because PositionEmbeddingsAttentionBridge applied the Llama RoPE formula to Cohere's cos/sin layout, which repeats every frequency twice; the bridge now rotates Q and K with Cohere's own convention. If you compare Cohere activations or logits against HF, that comparison should now line up without a...
fix: keep synthetic output biases out of the optimizer TransformerLensOrg/TransformerLens
Wrapping a model with no output bias used to add one anyway as a trainable parameter, handing the optimizer 50,257 extra values to update on DistilGPT2; the synthetic bias is now a buffer, so bridge.parameters() returns only real parameters and a layer wrapped with a genuine bias keeps it. Trainers that previously filtered this value out by hand can drop that workaround.
v4.2.0 TransformerLensOrg/TransformerLens
The 4.2.0 release adds tie-aware roc_auc and average_precision to the sparse probe metrics, fixes sparse probe control seeding, and makes hf_generate honor stop_at_eos=False. It also derives the projection_kernel default rank tolerance from the computed rank, so you no longer need to hand-tune that value.
Release v4.2.0 TransformerLensOrg/TransformerLens
The release is tagged non-breaking with new functionality, and it flags a documentation update, so read the notes before assuming existing hooks or bridge calls are untouched. Upgrade path is a normal version bump.
Prep for release v4.2.0 – Contribution follow ups TransformerLensOrg/TransformerLens
The rest of the release window was cleanup: extra test layers covering edge cases introduced by the new contributions, with no operator-facing behavior change.
Action items
- → Upgrade TransformerLens to 4.2.0 before your next Cohere interpretability run TransformerLensOrg/TransformerLens [plan]
