$ the-wire · showcase
Inspect evals get reproducible answer shuffling, control-arena stops running docs code
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
The day's operator-relevant work is a sweep of seeded answer-choice shuffling across inspect_evals tasks and two control-arena fixes, one of which stops docs publishing from silently running real evaluations.
fix(lab-bench): seed answer choice shuffling UKGovernmentBEIS/inspect_evals
All eight LAB-Bench tasks called retrieve_hf_dataset() without exposing shuffle_choices, and the helper's default of True meant construction was not reproducible. The helper now defaults to a fixed seed of 42 and shuffle_choices is exposed per task, so repeated runs of the same task present the same answer positions.
fix(sad): use fixed default seed UKGovernmentBEIS/inspect_evals
The five public SAD task functions defaulted seed to None, so with shuffle_choices=True two constructions of the same task could order answer choices differently; the stages tasks also drew their prompt randomization from that seed. A shared fixed default of 42 is now in place, with explicit seed values still honored.
fix(truthfulqa): seed answer choice shuffling UKGovernmentBEIS/inspect_evals
truthfulqa passed shuffle_choices=True to hf_dataset, leaving answer-choice order to an unseeded generator, so two constructions could present different answer positions for the same sample. No error surfaced; the task simply completed with a different layout. The PR separates row-level shuffle from answer-choice shuffling and seeds the latter.
feat: update default models and stop executing docs code UKGovernmentBEIS/control-arena
The docs examples were executable Quarto {python} cells configured with execute: enabled: true and freeze: true. CI rendered with --no-execute, but quarto publish gh-pages executed every cell without cached results, running real evaluations against whatever API keys and Docker were on the publishing machine, using the since-retired anthropic/claude-sonnet-4-20250514. The cells no longer execute...
fix(defer_to_resample): append the chosen resample to state.messages so it is executed (#901) UKGovernmentBEIS/control-arena
When the first untrusted proposal was flagged and a resample scored below defer_threshold, defer_to_resample set that resample as state.output but returned the original state; the resamples ran on copies through inspect_ai.agent.run(), so the chosen message never reached state.messages and the react loop, which reads tool calls from the last message in history, never ran it. The chosen resample...
