RepoJournal
Safety Evals Safety Evals
52 wires and counting

$ follow Safety Evals

Keep up with Safety Evals in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.

or

fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.

Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.

$ status

wire 2026-11-12
stories 26

© 2026 RepoJournal Home Showcase How it works Privacy

$ the-wire · showcase

Long Anthropic requests stop retrying forever; recovered samples keep their store

By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology

Inspect fixed two eval-integrity bugs operators hit in practice, a runaway retry loop on long Anthropic inputs and recovery dropping sample store data, while the EC2 sandbox lost its last documented path and a stale CI tag left a security tightening unapplied.

fix(anthropic): long requests no longer retry forever on the 10 minute read timeout UKGovernmentBEIS/inspect_ai

by jjallaire

Anthropic requests whose input takes longer than the SDK's 10 minute read deadline used to time out and, with the default unlimited max_retries, retry forever; whether you hit it depends on non-streaming calls or a proxy buffering the streaming keepalives. The fix makes that timeout non-transient so long-input evals fail instead of hanging.

fix: recovered samples keep their store UKGovernmentBEIS/inspect_ai

by brandly

When an eval crashed after a sample finished but before it was flushed to the .eval file, recover_eval_log brought the sample back with store={} from both the buffer and filestore paths, even though the StoreEvents were sitting in the buffer. Recovered samples now carry the data your solver and scorer wrote.

fix: ACP approval and ask_user requests stuck on a client that switched sample or rejected the request UKGovernmentBEIS/inspect_ai

by brandly

Under --acp-server, a human_approver prompt or ask_user form could hang when the client holding it loaded another sample on the same connection (Zed keeps several threads on one connection) or rejected the request. Those samples no longer sit waiting on a client that cannot answer.

fix: a second eval() in the same process ignores a changed display setting (#6267) UKGovernmentBEIS/inspect_ai

by Matt Brandly

The display object is built once per process, so in a script or notebook the second eval() ignored a changed display argument: display="plain" then display="none" still printed output. Each eval now actually drives rendering, which matters if you batch runs in one process.

docs: remove the deprecated EC2 sandbox from the sandbox listings (#6240) UKGovernmentBEIS/inspect_ai

by Art O Cathain

The EC2 sandbox is gone from the docs listings and sandbox="ec2" no longer points you at installing it, because it cannot be made secure: the agent can use the AWS credentials the VM needs for SSM. If you still select it, move to another sandbox.

Quick answers

What shipped in Safety Evals on November 12, 2026?
Inspect fixed two eval-integrity bugs operators hit in practice, a runaway retry loop on long Anthropic inputs and recovery dropping sample store data, while the EC2 sandbox lost its last documented path and a stale CI tag left a security tightening unapplied. In total, 13 commits and 13 pull requests landed.
Who contributed to Safety Evals on November 12, 2026?
4 developers shipped this update, including jjallaire, Art O Cathain, brandly, and Eric Patey.
What were the notable Safety Evals updates?
fix(anthropic): long requests no longer retry forever on the 10 minute read timeout, fix: recovered samples keep their store, and fix: ACP approval and ask_user requests stuck on a client that switched sample or rejected the request.