$ the-wire · showcase
Inspect AI tightens log handling, BrowseComp swaps in web_search
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
Inspect AI shipped a breaking change so that reading, editing or recomputing a log no longer imports task files or builds models the log names, and BrowseComp replaced its deprecated web_browser tools with configurable web_search.
fix!: recomputing, editing or recovering a log no longer imports files or builds models it names (#6007) UKGovernmentBEIS/inspect_ai
Recomputing metrics, editing a score, recovering a crashed log and eval-set recovery used to import the task file recorded in the log, parse sample score keys as scorer file specs, and rebuild models and registry objects from metric options. Logs from untrusted sources are now treated as data only, so any workflow that relied on that implicit loading will need to set the objects up itself befor...
fix: tool approval seeing unvalidated or differently converted arguments (#5652) UKGovernmentBEIS/inspect_ai
Approval used to run before arguments were validated and converted, so a human approver or viewer could green-light a call that later failed validation, or one the tool read differently after conversion (the PR calls out bool("false") evaluating to True). Policy globs were also matched against an unescaped rendering whose argument order the model chooses; approval now runs on the validated call...
BrowseComp: replace deprecated web_browser with web_search UKGovernmentBEIS/inspect_evals
BrowseComp moves from the deprecated web_browser() tools to web_search() with a new web_search_providers option for provider selection, native search where supported and Tavily as the fallback. The default Docker requirement is gone, and the eval is bumped to a new version, so check existing run configs and any pinned Docker setup before re-scoring.
fix: score edits that clear a field to None no longer read back as "UNCHANGED" (#6063) UKGovernmentBEIS/inspect_ai
Score edits that clear a field to None were written with exclude_none, dropped, and read back as the "UNCHANGED" default, which affected clears of answer, explanation and reason plus the original-score entry in Score.history. If you edited scores through the log UI or API, re-check those edits; deserialization now distinguishes an intentional clear from an untouched field.
fix: sandbox events from other tasks no longer dropped while one task suppresses its own (#6044) UKGovernmentBEIS/inspect_ai
no_events() set a flag on the shared SandboxEnvironmentProxy, so while one task was inside the block (an exec_remote poll, human-agent or sandbox-tools setup) every other task on the same sandbox lost its exec, read_file and write_file events. Suppression is now tracked per proxy in a ContextVar, so parallel evals over a shared sandbox keep their full event streams.
Action items
- → Update Inspect AI callers that recompute, edit or recover logs to set up task files, models and registry objects expl... UKGovernmentBEIS/inspect_ai [plan]
- → Move BrowseComp configs off web_browser() to web_search() with web_search_providers, and drop any default Docker depe... UKGovernmentBEIS/inspect_evals [plan]
