RepoJournal
Safety Evals Safety Evals
52 wires and counting

$ follow Safety Evals

Keep up with Safety Evals in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.

or

fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.

Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.

$ status

wire 2026-11-11
stories 41

© 2026 RepoJournal Home Showcase How it works Privacy

$ the-wire · showcase

Inspect adds Claude Fable 5.2 support, with a Bedrock caveat

By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology

The day's operator-facing work is a corrected model entry for a new Claude release, a fixed eval_set that stops re-running your tasks, and a scorer fix that was inverting real grades.

feat(anthropic): support Claude Fable 5.2 (#6443) UKGovernmentBEIS/inspect_ai

by Charles Teague

Inspect now knows Fable 5.2's context window, output limit and default effort instead of inheriting them from Fable 5.1. The computer use tool is the practical catch: Fable 5.2 rejects computer_20251124 everywhere except Amazon Bedrock, so agents doing computer use on other providers will break if Inspect keeps sending it.

fix: eval_set no longer re-runs or double-returns a task logged by separate runs (#6428) UKGovernmentBEIS/inspect_ai

by Ransom Richardson

When a log directory held logs of one task from separate runs, eval_set treated each run as a different task and re-ran work that had already finished, returning it twice. Your eval_set results over accumulated logs should now stop double-returning and stop burning compute.

fix(browse_comp): grader verdict inverted by 'incorrect:' and lost to capitalisation UKGovernmentBEIS/inspect_evals

by WatchTree-19

browse_comp_scorer matched the grader verdict case-sensitively and without a word boundary, so 'incorrect:' read as correct and capitalised field labels read as no verdict at all. Scores from that scorer on past runs deserve a second look before you compare them to anything.

feat: each message keeps one id through the agent bridge and the log, and tool events record the message that proposed them (#6143) UKGovernmentBEIS/inspect_ai

by Charles Teague

Message ids now survive the agent bridge and the log, and tool events record which message proposed them, so post-hoc log readers stop matching on content. Logs you already have keep their old ids; anything that resolves messages by content needs updating.

chore(deps): require uv 0.13.4 or later (#6461) UKGovernmentBEIS/inspect_ai

by Eric Patey

Also: required-version for uv moved from >=0.11.8 to >=0.13.4, because older uv still trusts the compromised emSign root (GHSA-hq85-5cpj-mr62), and inspect_evals pinned its abstention_bench dataset downloads to upstream commits after SelfAware began failing checksum validation in the nightly smoke test.

Quick answers

What shipped in Safety Evals on November 11, 2026?
The day's operator-facing work is a corrected model entry for a new Claude release, a fixed eval_set that stops re-running your tasks, and a scorer fix that was inverting real grades. In total, 18 commits, 22 pull requests, and 1 releases landed.
Who contributed to Safety Evals on November 11, 2026?
6 developers shipped this update, including Charles Teague, Eric Patey, Ransom Richardson, MattFisher, github-actions, and WatchTree-19.
What were the notable Safety Evals updates?
feat(anthropic): support Claude Fable 5.2 (#6443), fix: eval_set no longer re-runs or double-returns a task logged by separate runs (#6428), and fix(browse_comp): grader verdict inverted by 'incorrect:' and lost to capitalisation.