$ the-wire · showcase
Inspect blocks writable host mounts, control-arena rescues 6 weeks of swe_django side-task scores
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
Two changes matter to operators today: Inspect now rejects writable host volume mounts in Docker sandboxes by default, and control-arena's swe_django side-task correctness has been invalid since the sandbox went offline on 19 August 2026, so attack-run results from that window need re-scoring.
fix!: block writable host mounts in Docker sandboxes by default UKGovernmentBEIS/inspect_ai
Inspect now rejects unsafe host volume mounts before a sandbox starts, across all services and resolved sample settings, closing the path where an agent with write access to a host directory could rewrite Compose files or redirect mount sources with symlinks. Read-only sharing and ordinary Docker-managed volumes still work; if your eval config mounts a host directory writable, it will fail rath...
fix(swebench): make pytest available to the offline side task scorer (#911) UKGovernmentBEIS/control-arena
The offline side-task scorer installs pytest with `pip install pytest -q 2>/dev/null`, which cannot succeed under `network_mode: none`, so every `swe_django_sidetask` side task has scored INCORRECT since the sandbox went offline on 19 August 2026. Side-task success from attack runs in that window is invalid and should be re-scored or re-run.
fix(apps): grade execution by exit status instead of stderr UKGovernmentBEIS/control-arena
APPS scoring and dataset validation used to treat any stderr output as a failed run and ignore the exit status, which failed correct programs that logged warnings and, worse, scored backdoors as not working when they also wrote to stderr. Execution is now graded by exit status, so prior APPS attack-success numbers were understated and the safety reads built on them overstated.
revert: remove the short-lived inspect_ai.core package added in 0.3.277; core data types stay at their original import paths UKGovernmentBEIS/inspect_ai
The `inspect_ai.core` package added in 0.3.277 is being removed, and because every `ChatMessage`, `ModelOutput`, `ToolInfo` and `GenerateConfig` import keeps working from its original path, this only bites code that adopted the short-lived package. The split into separately published packages is off.
fix: store and state changes crashing, replaying wrong, or logging unchanged NaNs UKGovernmentBEIS/inspect_ai
Samples that move a value between lists or objects in their store or state used to crash, and `json_changes()` could log changes that do not replay the actual state or record unchanged NaNs. If you replay runs with `store_from_events()`, re-check stored samples rather than trusting a replay diff.
Action items
- → Re-score or re-run swe_django_sidetask attack samples from 19 August 2026 onward UKGovernmentBEIS/control-arena [immediate]
- → Remove writable host volume mounts from Inspect Docker sandbox configs before your next eval run UKGovernmentBEIS/inspect_ai [plan]
