$ cat ai-safety-evals/week/2026-10-05.log
the week in review · Oct 5 – Oct 11, 2026
Control-arena v20 and inspect_evals v0.24 break APIs
By RepoJournal · composed from the cited sources · human-reviewed weekly · methodology
Both releases ship breaking changes; inspect_ai adds Claude Haiku 5.5 and Claude Opus 5.5 support and provider request id logging.
v20.0.0 UKGovernmentBEIS/control-arena
The sanitize fix for the last_tool_calls monitor prompt is the breaking one: if you run that monitor, your prompt expectations change and the release is flagged breaking in the changelog. Analysis code now raises on invalid inputs instead of asserting, so callers that relied on asserts being stripped under optimized execution need to catch real exceptions.
v0.24.0 UKGovernmentBEIS/inspect_evals
Breaking for anyone whose pipelines call eval tasks directly: The Agent Company moves to a v3-B variant with Rocket.Chat infrastructure and five named Rocket.Chat tasks (admin_check_employees_budget_and_reply, finance_apply_tax_credit, finance_create_10k_income_report, hr_check_attendance_multiple_days_department and peers), plus an NPC solver for LLM-driven colleagues. Check which task identif...
Support for Claude Haiku 5.5 (#5752) UKGovernmentBEIS/inspect_ai
Claude Haiku 5.5 support means you can point runs at claude-haiku-5-5 today; the release notes describe 1M context, 128K output, adaptive thinking on by default, medium effort by default, and a June 2026 knowledge cutoff. Model-name detection already classified it as a known model, so existing configs that name it will resolve. Opus 5.5 support landed the same week, so both new Claude models ar...
Log provider request and response ids for every model call UKGovernmentBEIS/inspect_ai
Every model call now logs the provider request id, paired with ts-mono#734, which regenerates the viewer's TypeScript types and shows the IDs on the model event's Info tab. For operators chasing support tickets, safety or abuse reports, or billing questions, you can now point a provider at the exact request without reconstructing it from logs.
Accept an integer seed for `shuffle` in dataset loaders UKGovernmentBEIS/inspect_ai
shuffle=42 used to be silently wrong: the loaders only accepted a bool, so any nonzero value was truthy and the shuffle ran with seed=None, changing order every run. csv_dataset(), json_dataset(), file_dataset() and hf_dataset() now take an integer seed, so seeded shuffle actually reproduces across runs.
Fix trace log growing to many GB when generate caching is enabled (#5730) UKGovernmentBEIS/inspect_ai
With generate caching on, computing the cache key traced every component in full (whole message history, generate config, tool schemas) on each cached call, and trace logs grew quadratically with conversation length. The trace is dropped, so long cached conversations no longer produce multi-GB trace files on disk.
fix(swe_bench): default to the official SWE-bench DockerHub images (6-D) UKGovernmentBEIS/inspect_evals
swe_bench defaults back to the official SWE-bench DockerHub images. The epoch-built images adopted in #942 were smaller but built in October 2024 and miss an upstream fix, so official-image scores are the ones to trust; if your saved results came from the epoch images, expect them not to be comparable.
fix(tool): exclude *args and **kwargs from tool schema and parameter parsing UKGovernmentBEIS/inspect_ai
Tools with *args or **kwargs no longer advertise them to the model as required args and kwargs properties. The model can't fill a Python variadic by name anyway, so schemas are smaller and calls against variadic tools stop demanding arguments that were never fillable.
$ ls ai-safety-evals/week/ # the briefings behind this review
Keep up with Safety Evals in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.
fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.
Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.