$ the-wire · showcase
Inspect adds Claude Fable 5.2 support, with a Bedrock caveat
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
The day's operator-facing work is a corrected model entry for a new Claude release, a fixed eval_set that stops re-running your tasks, and a scorer fix that was inverting real grades.
feat(anthropic): support Claude Fable 5.2 (#6443) UKGovernmentBEIS/inspect_ai
Inspect now knows Fable 5.2's context window, output limit and default effort instead of inheriting them from Fable 5.1. The computer use tool is the practical catch: Fable 5.2 rejects computer_20251124 everywhere except Amazon Bedrock, so agents doing computer use on other providers will break if Inspect keeps sending it.
fix: eval_set no longer re-runs or double-returns a task logged by separate runs (#6428) UKGovernmentBEIS/inspect_ai
When a log directory held logs of one task from separate runs, eval_set treated each run as a different task and re-ran work that had already finished, returning it twice. Your eval_set results over accumulated logs should now stop double-returning and stop burning compute.
fix(browse_comp): grader verdict inverted by 'incorrect:' and lost to capitalisation UKGovernmentBEIS/inspect_evals
browse_comp_scorer matched the grader verdict case-sensitively and without a word boundary, so 'incorrect:' read as correct and capitalised field labels read as no verdict at all. Scores from that scorer on past runs deserve a second look before you compare them to anything.
feat: each message keeps one id through the agent bridge and the log, and tool events record the message that proposed them (#6143) UKGovernmentBEIS/inspect_ai
Message ids now survive the agent bridge and the log, and tool events record which message proposed them, so post-hoc log readers stop matching on content. Logs you already have keep their old ids; anything that resolves messages by content needs updating.
chore(deps): require uv 0.13.4 or later (#6461) UKGovernmentBEIS/inspect_ai
Also: required-version for uv moved from >=0.11.8 to >=0.13.4, because older uv still trusts the compromised emSign root (GHSA-hq85-5cpj-mr62), and inspect_evals pinned its abstention_bench dataset downloads to upstream commits after SelfAware began failing checksum validation in the nightly smoke test.
Action items
- → Upgrade uv to 0.13.4 or later in any environment that locks or syncs inspect_ai UKGovernmentBEIS/inspect_ai [immediate]
- → Re-check computer use evals with Fable 5.2 only on Amazon Bedrock until the tool path is updated UKGovernmentBEIS/inspect_ai [plan]