$ the-wire · showcase
Sentinel monitors gain model-call visibility, unhandled escalations now halt samples
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
inspect_ai's sentinel layer now watches the model call itself, not just tools, and refuses to let a risky call through when nothing is configured to handle the escalation.
fix(sentinel): terminate the sample when an escalation reaches the top with nothing to handle it UKGovernmentBEIS/inspect_ai
A root sentinel protocol that returns escalate with no handler used to let the tool call or result proceed as if the decision were continue, logging only a warning; it now terminates the sample. If you deployed a lone threshold with an escalate score, or a sequential whose last link escalates, as a policy gate, expect samples to stop instead of silently passing.
feat: sentinel monitors can watch every agent's model calls UKGovernmentBEIS/inspect_ai
Monitors and protocols were limited to BeforeToolCall and AfterToolCall, so nothing could inspect the request before it went to the model or the assistant message before the agent saw it. Generate stages (BeforeGenerate, AfterGenerate) close that gap and satisfy AISI control requirements R1 through R3.
feat: sentinel step.input is the exact model input even after events are evicted, or None when unavailable UKGovernmentBEIS/inspect_ai
step.input now holds exactly what was sent to the model call that produced step.call, even after ModelEvents are evicted from the log, and reads as None when it genuinely is not available. Monitors comparing a decision against the original model input get requirement R2 without depending on log retention.
fix(eval): distinguish custom datasets in cached evaluations (#906) UKGovernmentBEIS/control-arena
get_control_task hashed the setting, micro-protocol, mode and eval config into its automatic task name but not a custom dataset, so eval_set could match an old log and reuse cached results from different data; an empty custom dataset also fell through to the setting's full dataset. Custom datasets are now part of the identity, and empty ones no longer silently expand.
fix(simpleqa): give every sample a stable id UKGovernmentBEIS/inspect_evals
simpleqa_verified samples now carry the dataset's original_index as an id, and simpleqa falls back to a hash of the problem and answer, so sample ids survive re-runs and line-up across logs. Note that on main every simpleqa sample's metadata still reads as topic, answer_type and urls all None.
fix(swebench): diff against the image's setup commit, not base_commit UKGovernmentBEIS/control-arena
The long tail: on 9 of 18 swe_django_sidetask instances submission_diff carried a mode-only entry for roughly 6,400 tracked files (about 0.9 MB) before the agent touched anything, and the fix diffs against the image's setup commit rather than base_commit; simpleqa, HLE and Fortress also gained stable ids and blank-output scoring.
Action items
- → Re-run any control-arena evaluation that used a custom dataset with eval_set caching; cached logs may have been match... UKGovernmentBEIS/control-arena [plan]
- → Re-run inspect_ai sentinel configs that rely on an unhandled escalate to pass through; such samples now terminate UKGovernmentBEIS/inspect_ai [plan]
