$ the-wire · showcase
Inspect fixes score silently reading zero, plus F1 and computer-use typing bugs
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
inspect_ai shipped four fixes that change numbers operators already trust: dict-valued scores that came back as 0.0, F1 scores that moved with whitespace, Google computer-use typing that appended instead of replacing, and huggingface-hub back off the dependency cooldown.
fix: missing dict score values no longer count as zero UKGovernmentBEIS/inspect_ai
With no epoch reducer configured, scoring applied mean_score() with the default value_to_float(), so an unrecognized string like "pass" became 0.0 before accuracy()'s own converter ran: three passes and one fail logged 0.0 instead of 0.75. Missing dict score values now no longer count as zero, which means score numbers from evals run on earlier versions may not match what you get today.
fix: f1 stop words not removed when separated by a newline or tab (#5940) UKGovernmentBEIS/inspect_ai
f1(stop_words=[...]) only stripped a stop word when a space or hyphen separated it from surrounding text, so newline, tab, and carriage-return separators, common in multi-line model output, left them in the token counts. The same answer now scores the same regardless of its whitespace, so F1 figures for multi-line outputs shift against prior runs.
fix(google): computer use doesn't clear field before typing text (#6145) UKGovernmentBEIS/inspect_ai
Google computer-use evals that typed into a field without clearing it first were appending to existing content rather than replacing it; the fix lands as commit 6088d4b. Re-run any Google computer-use task whose results depended on typed field contents.
chore(deps): remove huggingface-hub from the 6-hour cooldown (#6340) UKGovernmentBEIS/inspect_ai
huggingface-hub comes off the 6-hour cooldown that a prior change had limited to direct model provider SDKs and Inspect packages, restoring it to the standard dependency policy after its pin was kept deliberately as a follow-up. No action for users unless you pin inspect_ai's resolution yourself.
