$ the-wire · showcase
Inspect hardens graded-answer prompts against forged data markers
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
inspect_ai closed a prompt-injection hole in model-graded scoring, stopped reusing stale sandbox tool trees across upgrades, and made sandbox identity scopes cover custom tools, while inspect_evals tightened agentic_misalignment verdict parsing.
fix: harden model-graded scorer prompts against fake data markers in the graded answer (#6282) UKGovernmentBEIS/inspect_ai
A submission ending in a forged [END DATA], a fake grader note and a fresh [BEGIN DATA] could get the judge to grade a wrong answer correct most of the time; model_graded_qa() and model_graded_fact() now neutralise the forged marker rather than rewriting it only.
fix: sandbox tools installed by another Inspect release are no longer reused (#6302) UKGovernmentBEIS/inspect_ai
Inspect used to reuse whatever trustworthy sandbox tool tree sat at /var/tmp/.da7be258e003d428, regardless of version, so a tree left by an older release rejected RPC parameters added since the upgrade. Tools from another Inspect release are no longer reused, so a first eval after upgrading pays the reinstall.
feat: custom tools inside a sandbox identity scope run as the scope's user (#6316) UKGovernmentBEIS/inspect_ai
Inside a sandbox_identity(user) scope, a custom tool or scaffold calling sandbox().exec(), read_file() or write_file() without a user still ran as the sandbox default, root in most images. Those calls now run as the scope's user, which changes file ownership and permissions for any custom tool you wrote against the old behavior.
fix: keep cached responses separate when model arguments change (#6271) UKGovernmentBEIS/inspect_ai
Cached responses are now keyed on a fingerprint of the model arguments, taken on each attempt after the before-generate hooks, with dictionary order included and credentials excluded. Changing a model argument no longer makes a run reuse a response cached under different arguments.
fix(agentic_misalignment): require a single tagged grader verdict UKGovernmentBEIS/inspect_evals
The blackmail, leaking and murder graders in agentic_misalignment accepted yes or no as substrings inside other words, so <answer>yesterday's draft was leaked</answer> counted as yes. They now require one complete final <answer> element whose normalized content is exactly yes or no, and no longer extract a verdict from completions that omit the element.
Action items
- → Upgrade inspect_ai before relying on model_graded_qa() or model_graded_fact() verdicts, since forged data markers cou... UKGovernmentBEIS/inspect_ai [plan]
- → Re-run prior agentic_misalignment results scored under the old graders; substring matching could count leaked, blackm... UKGovernmentBEIS/inspect_evals [plan]
