$ the-wire · showcase
Long Anthropic requests stop retrying forever; recovered samples keep their store
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
Inspect fixed two eval-integrity bugs operators hit in practice, a runaway retry loop on long Anthropic inputs and recovery dropping sample store data, while the EC2 sandbox lost its last documented path and a stale CI tag left a security tightening unapplied.
fix(anthropic): long requests no longer retry forever on the 10 minute read timeout UKGovernmentBEIS/inspect_ai
Anthropic requests whose input takes longer than the SDK's 10 minute read deadline used to time out and, with the default unlimited max_retries, retry forever; whether you hit it depends on non-streaming calls or a proxy buffering the streaming keepalives. The fix makes that timeout non-transient so long-input evals fail instead of hanging.
fix: recovered samples keep their store UKGovernmentBEIS/inspect_ai
When an eval crashed after a sample finished but before it was flushed to the .eval file, recover_eval_log brought the sample back with store={} from both the buffer and filestore paths, even though the StoreEvents were sitting in the buffer. Recovered samples now carry the data your solver and scorer wrote.
fix: ACP approval and ask_user requests stuck on a client that switched sample or rejected the request UKGovernmentBEIS/inspect_ai
Under --acp-server, a human_approver prompt or ask_user form could hang when the client holding it loaded another sample on the same connection (Zed keeps several threads on one connection) or rejected the request. Those samples no longer sit waiting on a client that cannot answer.
fix: a second eval() in the same process ignores a changed display setting (#6267) UKGovernmentBEIS/inspect_ai
The display object is built once per process, so in a script or notebook the second eval() ignored a changed display argument: display="plain" then display="none" still printed output. Each eval now actually drives rendering, which matters if you batch runs in one process.
docs: remove the deprecated EC2 sandbox from the sandbox listings (#6240) UKGovernmentBEIS/inspect_ai
The EC2 sandbox is gone from the docs listings and sandbox="ec2" no longer points you at installing it, because it cannot be made secure: the agent can use the AWS credentials the VM needs for SSM. If you still select it, move to another sandbox.
Action items
- → Stop using the EC2 sandbox (sandbox="ec2") in Inspect and migrate those evals to another sandbox before the next run UKGovernmentBEIS/inspect_ai [plan]