$ the-wire · showcase
inspect_ai recovers stalled sandbox polls, control-arena ships v20.0.0
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
Sandbox stalls no longer kill your samples, control-arena's v20 release changes monitor prompt handling, and a missing decorator in inspect_evals has been poisoning every gdm_strategic_rule_breaking run.
Add opt-in recovery for exec_remote polls that time out during a sandbox stall (#5600) UKGovernmentBEIS/inspect_ai
exec_remote polls that time out during a sandbox stall now recover instead of erroring the sample, opt-in. Previously ExecRemoteProcess._poll only retried RuntimeError, so a SandboxTimeoutError escaped the process iterator and the bridged agent's proxy monitor raised; the poll is idempotent, so a long-running eval on a flaky sandbox server no longer loses the sample.
v20.0.0 UKGovernmentBEIS/control-arena
The v20.0.0 release is a breaking one: the last_tool_calls monitor prompt is now sanitized, analysis inputs are validated with raise rather than assert, and primitives moved from asyncio to anyio. If you pin control-arena and rely on the current monitor prompt text or on assertion-based input checks, read the notes before upgrading.
fix: preserve tool output limits for scoring and replay UKGovernmentBEIS/control-arena
Under Inspect 0.3.257 final scorers run in the parent task and read an unset ContextVar, so they could see the agent's full retained tool output even when the agent only saw an excerpt; the resolved limit is now persisted per sample. Replay and rescoring helpers recover the config for older logs too, so your existing control-arena logs score the same way they did originally.
fix(gdm_stealth): add missing @tool decorator to chat_with_manager (5-B) UKGovernmentBEIS/inspect_evals
chat_with_manager in gdm_stealth's strategic_rule_breaking tools returned a Tool without the @tool decorator, so Inspect could not register it and every gdm_strategic_rule_breaking sample errored at the first model.generate() call with nothing scored. The decorator is back; rerun any of those samples you treated as a result.
Fix model request tracking entries leaking when a request fails or is cancelled UKGovernmentBEIS/inspect_ai
Providers only cleared HttpHooks._requests when they returned a result or handled an error, so requests that raised or were cancelled kept their entry for the life of the provider, and since the OpenAI-compatible client now survives credential refreshes those entries outlived the refresh. Requests through a failing or cancelled provider no longer accumulate tracking entries across a long evalua...
Action items
- → Rerun gdm_strategic_rule_breaking samples from inspect_evals after the chat_with_manager @tool fix UKGovernmentBEIS/inspect_evals [plan]
