$ the-wire · showcase
Inspect's Perplexity provider moves to the Agent API, and inspect_evals drops web_browser
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
Two forced migrations landed today: Perplexity's retired Sonar Chat Completions endpoint breaks every Inspect request until the provider is updated, and two eval suites have swapped or vendored the deprecated web_browser tools ahead of their removal.
fix!: Perplexity requests failing after Sonar moved to the Agent API (#5703) UKGovernmentBEIS/inspect_ai
Perplexity retired Sonar Chat Completions, so every request from the Inspect provider was failing with a 403 pointing at /v1/responses. The provider now posts to https://api.perplexity.ai/v1 and maps model names to Agent API ids, with a bare sonar resolving to perplexity/sonar; the fix is marked breaking, so check any hardcoded model strings in your task configs.
GAIA: replace deprecated web_browser with web_search UKGovernmentBEIS/inspect_evals
GAIA now uses web_search() with a web_search_providers selector in place of the deprecated web_browser(), requires Inspect 0.3.278 or later, and bumps the task version from 3-B to 4-C. Anthropic search stays on direct calls so code execution remains inside the GAIA sandbox, where task files are actually readable.
GDM Self-proliferation: vendor deprecated web_browser tools UKGovernmentBEIS/inspect_evals
GDM Self-proliferation vendors Inspect AI 0.3.263's browser tools locally so the task keeps running after web_browser() disappears, preserving the pinned container, tool descriptions and approval rules, and holding the task version at 6-B. The PR reports before/after runs on 7 October using Inspect 0.3.263.
feat: record bridged host tool calls as tool events in the transcript (#5741) UKGovernmentBEIS/inspect_ai
Host tools executed through sandbox_agent_bridge(bridged_tools=...) now leave a ToolEvent in the transcript instead of only the ModelEvent that proposed them, including denied calls that previously left nothing. Each event carries metadata["bridge"] naming the server, tool, model-facing name, proposing call and grant, which is what you need to audit what a sandboxed agent actually ran.
fix: warn when a sample or task sandbox config is not used (#5736) UKGovernmentBEIS/inspect_ai
Compose files declared for the docker sandbox type are now forwarded to any docker-compatible sandbox regardless of filename, so configs not named compose.yaml are no longer dropped silently; a warning is logged when the sandbox is not docker-compatible. Files declared for other sandbox types keep the filename check.
Action items
- → Upgrade inspect_ai to a release containing the Perplexity Agent API migration before your next Perplexity-backed eval... UKGovernmentBEIS/inspect_ai [immediate]
- → Upgrade inspect_ai to 0.3.278 or later before running GAIA version 4-C UKGovernmentBEIS/inspect_evals [plan]
