$ the-wire · showcase
inspect_evals 0.25.0 rescores SWE-Lancer accuracy
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
inspect_evals v0.25.0 changes how SWE-Lancer (v2-B) scores manager decisions, so published accuracy for that eval will move against the same model outputs.
v0.25.0 UKGovernmentBEIS/inspect_evals
The release notes for v0.25.0 say SWE-Lancer's score_manager_variant now records a missing, malformed or unreadable manager_decisions.json as 0 with a distinguishing reason, while read timeouts and permission errors error the sample instead of scoring 0 -- and because errored samples leave the denominator, accuracy is no longer dragged down by them. Proposal-id matching also changed: a quoted o...
docs: tag changelog entries with the lint rule whose finding they cleared UKGovernmentBEIS/inspect_evals
Changelog entries that clear an inspect-evals-lint finding now end with the rule code in square brackets, and past entries were backfilled to match. For operators this makes it traceable which lint rules a given eval version has already cleared.
chore(deps): bump the actions group with 7 updates UKGovernmentBEIS/inspect_evals
The actions group bump covers seven workflow dependencies, including anthropics/claude-code-action 1.0.241 to 1.0.248, astral-sh/setup-uv 10.2.0 to 10.3.0, and patch releases for actions/upload-artifact and actions/download-artifact. CI maintenance only; nothing changes in how you run evals.
