$ cat ai-safety-evals/week/2026-10-19.log
the week in review · Oct 19 – Oct 25, 2026
Inspect 0.3.282 and 0.3.283 both break configs
By RepoJournal · composed from the cited sources · human-reviewed weekly · methodology
Per-sample tool state and dataset `checkpoint` fields change meaning in the same week inspect_evals 0.26.0 retires GAIA's browser tools.
v0.3.283 UKGovernmentBEIS/inspect_ai
Dataset loaders now read a top-level `checkpoint` field as the sample's checkpoint configuration and raise when it isn't valid, so datasets that used that name for unrelated data fail at load. Rename or restructure the field before you upgrade past 0.3.283.
v0.3.282 UKGovernmentBEIS/inspect_ai
Tool state is now scoped per agent, so agents running concurrently inside one sample keep separate state instead of sharing it. If your samples relied on the old shared state, the agents will no longer see each other's tool calls.
v0.26.0 UKGovernmentBEIS/inspect_evals
GAIA v4-C swaps the default agent's deprecated browser tools for web search, controlled by the new `web_search_providers` setting. It defaults to built-in model search with Tavily as fallback, which requires `TAVILY_API_KEY`; GAIA runs without it now behave differently.
fix!: Perplexity requests failing after Sonar moved to the Agent API (#5703) UKGovernmentBEIS/inspect_ai
Perplexity retired Sonar Chat Completions and every request from the provider was failing with a 403 pointing at /v1/responses. The provider now sends Responses requests, so Perplexity-backed evals run again once you are on a build that includes the fix.
GDM Self-proliferation: vendor deprecated web_browser tools UKGovernmentBEIS/inspect_evals
GDM Self-proliferation vendors Inspect AI 0.3.263's browser tools locally so the task keeps working once `web_browser()` is removed from Inspect. Task construction, saved browser setup and the SP01/SP05 scorers import the local implementation.
AssistantBench: replace deprecated browser with Playwright MCP UKGovernmentBEIS/inspect_evals
AssistantBench moves off the deprecated `web_browser()` to a sandboxed Playwright MCP setup that keeps interactive browsing, with the existing agent, prompt, search, dataset and scorer preserved. Expect a different sandbox and image to pull on first run.
fix: logged model inputs and outputs no longer change when an agent later edits a message in place UKGovernmentBEIS/inspect_ai
`ModelEvent.input` and `.output` no longer hold live message objects, so logs stop changing when agent code edits an earlier message in place after a generation. Reruns of your own log analysis against those fields are safe now.
feat(sentinel): steps carry a person's earlier human() decisions in the conversation UKGovernmentBEIS/inspect_ai
Steps now carry a person's earlier `human()` decisions, since `step.escalations` only covered the current step and monitors kept re-flagging tool calls a reviewer had already cleared. Human-in-the-loop runs should see fewer repeat questions.
$ ls ai-safety-evals/week/ # the briefings behind this review
Keep up with Safety Evals in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.
fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.
Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.