$ the-wire · showcase
inspect_evals 0.27.0 lands with AgentDojo locale fix and PaperBench solver change
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
inspect_evals v0.27.0 reaches PyPI with eval-correctness fixes operators will want on their next run, while inspect_ai commits fix three scoring and tooling behaviors that silently changed results.
Release v0.27.0 UKGovernmentBEIS/inspect_evals
The release carries an AgentDojo fix that reads the bundled suite YAML as UTF-8 instead of the locale encoding, so travel user tasks 9 and 11 no longer score incorrectly and the workspace suites load under non-UTF-8 code pages, plus a BrowseComp change that swaps the optional browsing agent's browser tools for configurable web search and drops the default Docker requirement. Anyone who has been...
PaperBench: remove the deprecated web_browser tools from the default solver (#2766) UKGovernmentBEIS/inspect_evals
The default solver no longer carries the deprecated web_browser tools. They never worked in the sandbox PaperBench ships, since there is no inspect-tool-support service, so the first browser call raised PrerequisiteError and ended the sample in error; removing them without a replacement lets the solver construct and run.
fix: exact() and f1() no longer match a negative number to its positive (#5913) UKGovernmentBEIS/inspect_ai
The shared tokenizer split on every hyphen, so "-5" became "5" before the number check, and boundary punctuation stripping removed the sign from "(-40)"; a minus sign that opens a number now stays with it, while ranges like "3-5" and bracket forms like "f(-1)" still split as before. Any exact() or f1() score you computed on negative numeric answers was wrong.
fix: inspect score no longer imports a file a log names as a scorer, and prints what it loads from the log (#6085) UKGovernmentBEIS/inspect_ai
A scorer name recorded in a log used to go through scorer_from_spec(), so a name like path.py@name or an existing path made inspect score import that file; recorded names now resolve through scorer_from_log_name() against the registry first, and the command prints what it loads from the log. Scores from untrusted logs are safer to re-run, and you can see which scorer actually executed.
fix: exclude dataclass non-init fields from tool parameters (#5283) UKGovernmentBEIS/inspect_ai
Dataclass fields marked non-init are excluded from generated tool parameters, and supplying one is now rejected with a recorded error rather than silently ignored, so custom schemas built from dataclasses will see a narrower and stricter parameter set. The remaining long tail is housekeeping: sandbox-tools v37 keeps all exec_remote command output and now requires a version bump when any build i...
Action items
- → Upgrade inspect_evals to 0.27.0 and re-run AgentDojo travel tasks before citing prior scores UKGovernmentBEIS/inspect_evals [plan]
- → Re-score any exact() or f1() results involving negative numbers after upgrading inspect_ai UKGovernmentBEIS/inspect_ai [plan]
