$ the-wire · showcase
inspect_ai closes two log-recovery path escapes
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
Two fixes to `inspect log recover` and `recover_eval_log()` close ways a crafted `.buffer` directory could write or read files outside the recovery sandbox, on top of sandbox lifecycle fixes for E2B, Modal, Daytona and Podman and a scoring-fidelity change in InfiniteBench's Code.Run.
fix: confine temporary attachment files during crashed-log recovery (#6626) UKGovernmentBEIS/inspect_ai
When `inspect log recover` (or `eval_retry` on a `started` log) rebuilt a crashed log, it used attachment IDs from the companion `.buffer` to construct temporary file paths, so whoever controlled the buffer could have Inspect create or overwrite a host file outside the temp directory with content they supplied, bounded only by the recovering user's write permissions. If you recover logs that ar...
fix: a malformed hash in a shared-buffer manifest can no longer make log recovery open a file outside the buffer (#6627) UKGovernmentBEIS/inspect_ai
Recovery that falls back to the `.buffer/` directory beside an eval log used a content hash from `manifest.json` to build filenames, and a malformed hash could make it open a file outside the buffer. Both this fix and the attachment-path fix land in the same recovery path, so treat them as one upgrade.
fix: missed sandbox cleanup and LookupError on E2B, Modal and Daytona with several sandbox configurations or an empty SampleSource seed (#6633) UKGovernmentBEIS/inspect_ai
E2B, Modal, Daytona and Podman store their running-sandbox list and run id in `ContextVar`s that `task_init` binds and `sample_init` and `task_cleanup` read, and each `task_init` ran in its own context: with several sandbox configurations or an empty `SampleSource` seed you got missed cleanup and a `LookupError`. Teams running multi-config sandbox evals should expect stray billable sandboxes to...
fix(cti_realm): propagate threat-alignment judge API errors instead of scoring 0.0 (#2238) UKGovernmentBEIS/inspect_evals
`_score_cti_threat_alignment` wrapped the judge `generate` call in a blanket `except Exception` returning `0.0`, so a judge outage zeroed the highest-weighted trajectory component (C0, weight 1.25) while the run still reported `status: success`. API errors now propagate instead of being scored as failures, which means a `cti_realm` run can fail loudly where it used to hand you a quietly wrong n...
fix(infinite_bench): align Code.Run scoring with official rule (#2429) UKGovernmentBEIS/inspect_evals
The prompt-matched template scorer stays the primary Code.Run metric but now extracts signed integers that carry terminal punctuation or quotes, while the pinned official InfiniteBench last-token scorer is reported as a separate metric; the template scorer also rejects decimal prefixes, identifiers, and an earlier answer followed by an empty final template. Historically recorded Code.Run scores...
Action items
- → Upgrade inspect_ai to a build containing the log-recovery path fixes before recovering logs from untrusted .buffer di... UKGovernmentBEIS/inspect_ai [plan]