$ the-wire · showcase
Petri needs explicit realism roles, Opus 5.5 support lands
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4)
10 people shipped this
Charles Teague
@dragonstyle
1 cited
Ransom Richardson
@ransomr
1 cited
Matt Brandly
@brandly
1 cited
github-actions[bot]
@github-actions[bot]
1 cited
Tania
@ItsTania
1 cited
kaifronsdal
@kaifronsdal
1 cited
ndalton12
@ndalton12
1 cited
Ben Dixon
@bndxn
Chen Yufeiyang
@feiiiiii5
JustinOlive
@JustinOlive
Petri's realism_filter now fails loudly instead of silently scoring with the auditor model, and both Petri and inspect_ai shipped the provider fixes that unblock Claude Opus 5.5 runs.
feat!: require the realism model role for realism_filter safety-research/petri
The realism approver used to fall back to the auditor model role when no realism role was set; it now fails at the first realism check with an error naming the role you must provide. If you run realism_filter, add --model-role realism=... to your config before your next run, since the fallback no longer exists.
feat: support Claude Opus 5.5 by requiring inspect-ai 0.3.276 and inspect-scout 0.5.3 safety-research/petri
Petri's judge and realism approver get structured answers through Inspect Scout's answer tool, which older Scout releases forced, so neither could run against Opus 5.5 at all. The minimums are now inspect-ai 0.3.276 and inspect-scout 0.5.3.
Fix forced tool choice being ignored on Claude when reasoning_effort is set (#5764) UKGovernmentBEIS/inspect_ai
With reasoning_effort set, the Anthropic provider omitted tool_choice on every thinking request, so forced tool choice was silently ignored. It now omits tool_choice only under extended thinking (budget_tokens), since adaptive thinking accepts a forced choice and just skips thinking on that turn.
Fix forced web_search failing with 400 on Claude 4.6+ and mid-stream provider errors reaching bridged agents as HTTP 200 (#5716) UKGovernmentBEIS/inspect_ai
Forced web_search 400ed on Claude 4.6+ because the _20260209 web tools defaulted allowed_callers to the code-execution caller only; they now allow direct calls and pass a caller-supplied allowed_callers through. The same fix keeps mid-stream provider errors from reaching bridged agents as HTTP 200.
Fix compaction triggering early on OpenAI reasoning models (#5750) UKGovernmentBEIS/inspect_ai
Compaction was adding stamped redacted_reasoning_tokens on top of usage.input_tokens, double counting replayed encrypted reasoning that both already include. OpenAI reasoning model runs should stop compacting early.
v0.24.0 UKGovernmentBEIS/inspect_evals
The release adds Rocket.Chat infrastructure and NPC solver tasks to The Agent Company, renames max_messages to max_turns to match upstream, and gives agent containers shm_size: "1gb" so Chromium has enough /dev/shm. The rename will break existing configs that still pass max_messages.
ci: stop recurring workflow runs on forks (#2644) UKGovernmentBEIS/inspect_evals
Every job in checks.yml, markdown-lint.yml and workflow-lint.yml is now guarded so fork pushes no longer fire CI runs that fail on missing upstream secrets. Fork owners stop getting failure email.
Action items
- → Add --model-role realism=... to any config that enables realism_filter; the auditor-model fallback is gone safety-research/petri [immediate]
- → Bump inspect-ai to 0.3.276 and inspect-scout to 0.5.3 to run Petri on Claude Opus 5.5 safety-research/petri [plan]
- → Rename max_messages to max_turns in The Agent Company configs to keep them working on inspect_evals v0.24.0 UKGovernmentBEIS/inspect_evals [plan]
