$ the-wire · showcase
inspect_evals 0.26.0 drops GAIA's browser tools; background subagent failures now fail samples
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
inspect_evals v0.26.0 changes what the default GAIA agent can do, while inspect_ai fixes three failures that could quietly corrupt results: swallowed background subagent errors, a broken Windows file:// path, and host-timezone drift in analysis timestamps.
v0.26.0 UKGovernmentBEIS/inspect_evals
The default GAIA agent no longer has deprecated browser tools; it now uses web search, with `web_search_providers` controlling provider selection. Defaults to built-in model search, with Tavily as fallback requiring `TAVILY_API_KEY`, and Anthropic search needs Inspect 0.3.278 or later. If your GAIA runs relied on interactive browser actions, they are gone and results are not comparable across t...
fix: fail samples on unexpected background subagent errors UKGovernmentBEIS/inspect_ai
Unexpected exceptions raised by background subagents used to be absorbed, so a sample kept scoring after an infrastructure failure that would have killed a foreground subagent, for instance an agent working against an empty filesystem after its sandbox was replaced. Background dispatch now propagates those failures to the sample runner, so `fail_on_error` and `retry_on_error` apply to them as t...
fix: local_path no longer drops the Windows drive from file:// URIs UKGovernmentBEIS/inspect_ai
On Windows, `list_eval_logs()` and `resolve_logs()` returned `file://D:/logs/...` URIs whose drive letter landed in the URL netloc, so `local_path()` dropped it and resolved against the current drive. `read_eval_log`, `read_eval_log_headers`, `evals_df`, and `samples_df` raised on those logs as a result; they now carry the drive through.
fix: keep timezone-less dataframe timestamps on the UTC clock (#5767) UKGovernmentBEIS/inspect_ai
Timezone-less timestamps read through dataframes are now treated as UTC rather than being rescheduled onto the host clock, so analysis results no longer shift by the operator's local offset. The regression test runs on a +05:30 host, which is the case that used to pass silently on UTC CI.
ci: pin the release build and npm toolchains UKGovernmentBEIS/inspect_ai
The remaining work is supply-chain hygiene: the release build and npm publish steps pinned their toolchains, so an unpinned `build`, `setuptools`, or npm tool could no longer alter published artifacts. Nothing changes for operators, but it is what stands between you and a tampered release.
Action items
- → Upgrade inspect_evals to 0.26.0 and set TAVILY_API_KEY (or pin a search provider) before running GAIA, since the defa... UKGovernmentBEIS/inspect_evals [plan]
- → Upgrade inspect_ai to a build containing the background subagent failure fix, then re-check any runs that scored thro... UKGovernmentBEIS/inspect_ai [plan]
