$ the-wire · showcase
Inspect logs stop mutating mid-run, Math.Calc regrades
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
inspect_ai fixed a logging bug that silently rewrote the record of what every model saw, while inspect_evals changed how Math.Calc grades, so scores from the infinite_bench benchmark can move for reasons unrelated to your models.
fix: logged model inputs and outputs no longer change when an agent later edits a message in place UKGovernmentBEIS/inspect_ai
Previously, editing a message in place after a generation, for example a react on_continue that rewrites an earlier tool result's content, metadata or error, also changed the input recorded on every earlier ModelEvent, so your .eval logs no longer described what the model actually saw at that step. Logged inputs and outputs are now snapshots that survive later mutation.
fix(infinite_bench): Math.Calc offset correction skipped for proxied GPT models UKGovernmentBEIS/inspect_evals
The infinite_bench Math.Calc scorer no longer inspects the model name when deciding whether to apply the leading-offset correction, so proxied GPT models are graded on the same footing as direct ones. The move is one-directional: Math.Calc can only score higher, which means Math.Calc numbers from earlier runs are not directly comparable to new ones.
fix(monitor): parse_score_from_string drops the sign and returns the absolute value UKGovernmentBEIS/control-arena
parse_score_from_string used digit-only capture groups, so a monitor prompted on a signed scale like [-1, 1] that wrote "-1" inside a sentence was parsed as 1, flipping the sign of the suspicion score. Monitor scores on signed ranges from earlier control-arena runs are suspect; re-check any run where a monitor scored a negative in prose.
fix: pass usedforsecurity=False to hashlib.md5 so evals load on FIPS hosts UKGovernmentBEIS/inspect_evals
On FIPS-enforcing hosts, hashlib.md5 in create_stable_id raised ValueError and stopped every eval that builds sample ids during dataset loading, and the same failure hit the metadata hashes in DROP, RACE-H, SQuAD and VQA-RAD. The calls now pass usedforsecurity=False, so those evals load on FIPS hosts again.
perf: retrying an eval whose logs are on S3 no longer re-uploads the prior attempt's log before the retry starts UKGovernmentBEIS/inspect_ai
Retrying an eval whose logs live on S3 used to re-upload the entire seeded log before any work ran, about 11 s for a 119 MB prior and 18 s for a 195 MB prior on an 11 MB/s link; the first write now copies the prior log's member area server-side and uploads only the new bytes. The rest is small: a lock bump to inspect-sentinel 0.2.0, a pandas 3.1 fix in score_to_float(), and a lock refresh to in...
Action items
- → Upgrade inspect_evals past the Math.Calc fix and re-baseline infinite_bench scores before comparing against prior runs UKGovernmentBEIS/inspect_evals [plan]
