$ cat ai-safety-evals/week/2026-11-09.log
the week in review · Nov 9 – Nov 15, 2026
inspect_evals 0.29.0 and 0.28.0 fix grading and SSRF hole
By RepoJournal · composed from the cited sources · human-reviewed weekly · methodology
Two eval releases land in one week, while inspect_ai adds Claude Fable 5.2, stops infinite Anthropic retries, and corrects sample retry and recovery behavior.
fix(bfcl): restrict web fetches to public destinations UKGovernmentBEIS/inspect_evals
BFCL's two opt-in V4 web-search categories fetch model-selected URLs inside the evaluator process, so a scored run could reach local and private services, redirects included. Page fetches are now restricted to public destinations; if you run those categories, update before scoring anything else.
v0.29.0 UKGovernmentBEIS/inspect_evals
by github-actions[bot]
BrowseComp's grader was case-sensitive and had no word boundary, so "Correct: yes" scored as wrong and "incorrect: yes" in reasoning scored as correct. The verdict now comes from the last readable correct: line, and a reply with no readable verdict is no longer silently interpreted.
feat(anthropic): support Claude Fable 5.2 (#6443) UKGovernmentBEIS/inspect_ai
Claude Fable 5.2 (claude-fable-5-2, released 2026-11-09) has no model-info entry in Inspect yet, leaving its context window, output limit and default effort unknown even though most request handling already works. The Bedrock path was flagged as a caveat in this week's briefing, so verify it before routing Bedrock traffic through the new model.
fix(anthropic): long requests no longer retry forever on the 10 minute read timeout UKGovernmentBEIS/inspect_ai
Anthropic requests with very long inputs could retry forever against the SDK's 10 minute read deadline, since non-streaming responses send nothing until the whole message is ready. Your long-input evals will now fail rather than spin; budget for it in retry and cost planning.
fix: recovered samples keep their store UKGovernmentBEIS/inspect_ai
When an eval crashed after a sample completed but before it was flushed to the .eval file, recover_eval_log brought the sample back with an empty store. Recovered samples now keep their store, so resume workflows no longer hand you samples whose state silently vanished.
fix: task retries no longer reuse one dynamic sample's result for another sample without an id (#6314) UKGovernmentBEIS/inspect_ai
A dynamic sample added without an id, from next_samples(), a follow-up returned by sample_complete or sample_abandoned, or enqueue_sample(), could have its result reused for a different sample on retry. If you drive tasks with SampleSource, check that your retried samples are now scoring independently.
fix: the sdist no longer includes the sandbox tools binaries; an install built from it downloads them at first use (#6306) UKGovernmentBEIS/inspect_ai
Each inspect-ai release was adding 72.3 MB to the PyPI project, with the two glibc sandbox tools binaries (about 29 MB) shipped in both the wheel and the sdist. The source distribution no longer carries them and downloads them at first use, which matters for air-gapped or pre-baked image installs.
chore(deps): require uv 0.13.4 or later (#6461) UKGovernmentBEIS/inspect_ai
Running alongside the bigger fixes: the project now requires uv 0.13.4 or later for sync and lock, up from 0.11.8, so bump contributor and CI images before the next lock. The EC2 sandbox also came out of the sandbox listings, deprecated because it cannot be made secure.
$ ls ai-safety-evals/week/ # the briefings behind this review
Keep up with Safety Evals in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.
fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.
Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.