$ the-wire · showcase
TransformerLens 4.4.0 lands, and SAELens patches four ways its loaders misread checkpoints
By RepoJournal · Filed · About Interpretability · Composed from the cited sources · methodology
SAELens published four fixes that operators hit in daily use: the evals CLI crashes outright on transformer-lens 4.x, sparsify checkpoints fail to load, and Llama Scope SAEs fire about 35% fewer latents than they did in training because the JumpReLU threshold drops the decoder norm.
Release v4.4.0 TransformerLensOrg/TransformerLens
The 4.4.0 release notes list the details of what changed; a follow-up commit fixed a failing MPS test that shipped with the same release, so Apple-silicon users pulling the tag need that commit in their checkout.
fix: run the evals CLI on transformer-lens 4 jbloomAus/SAELens
multiple_evals(), the code behind python -m sae_lens.evals and run_evals_cli, hardcoded load_model("HookedTransformer", ...) and so raised ValueError before scoring a single SAE on transformer-lens 4.x. It now loads through default_model_class_name().
fix: include decoder norm in Llama Scope JumpReLU threshold jbloomAus/SAELens
llama_scope_sae_huggingface_loader built the JumpReLU threshold without the decoder norm, so the released Llama Scope SAEs fire about 35% fewer latents than they did in training; the loader now matches the sparsity_include_decoder_norm: true gate they were trained under.
fix: load sparsify SAEs with the correct W_dec orientation and d_sae jbloomAus/SAELens
The sparsify directory and HuggingFace loaders transposed W_dec that sparsify already stores in SAELens layout, failing with a size mismatch unless d_in == d_sae, and mishandled sparsify's num_latents=0 default. Both loaders now build the SAE correctly.
fix: average eval metrics in float64 so low-precision runs are not rounded jbloomAus/SAELens
The long tail: run_evals under bfloat16 or float16 rounded ce_loss_* and kl_div_* to roughly three significant digits, coarser than the gap between two SAEs, so get_downstream_reconstruction_metrics and get_sparsity_and_variance_metrics now average in float64; and a gpt2-small-res_scefr-ajt SAE entry had release-level keys stripped.
Action items
