RepoJournal
Safety Evals Safety Evals
52 wires and counting

$ follow Safety Evals

Keep up with Safety Evals in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.

or

fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.

Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.

$ status

wire 2026-11-19
stories 35

© 2026 RepoJournal Home Showcase How it works Privacy

$ the-wire · showcase

inspect_ai closes two log-recovery path escapes

By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology

Two fixes to `inspect log recover` and `recover_eval_log()` close ways a crafted `.buffer` directory could write or read files outside the recovery sandbox, on top of sandbox lifecycle fixes for E2B, Modal, Daytona and Podman and a scoring-fidelity change in InfiniteBench's Code.Run.

fix: confine temporary attachment files during crashed-log recovery (#6626) UKGovernmentBEIS/inspect_ai

by Craig Walton

When `inspect log recover` (or `eval_retry` on a `started` log) rebuilt a crashed log, it used attachment IDs from the companion `.buffer` to construct temporary file paths, so whoever controlled the buffer could have Inspect create or overwrite a host file outside the temp directory with content they supplied, bounded only by the recovering user's write permissions. If you recover logs that ar...

fix: a malformed hash in a shared-buffer manifest can no longer make log recovery open a file outside the buffer (#6627) UKGovernmentBEIS/inspect_ai

by Craig Walton

Recovery that falls back to the `.buffer/` directory beside an eval log used a content hash from `manifest.json` to build filenames, and a malformed hash could make it open a file outside the buffer. Both this fix and the attachment-path fix land in the same recovery path, so treat them as one upgrade.

fix: missed sandbox cleanup and LookupError on E2B, Modal and Daytona with several sandbox configurations or an empty SampleSource seed (#6633) UKGovernmentBEIS/inspect_ai

by Ransom Richardson

E2B, Modal, Daytona and Podman store their running-sandbox list and run id in `ContextVar`s that `task_init` binds and `sample_init` and `task_cleanup` read, and each `task_init` ran in its own context: with several sandbox configurations or an empty `SampleSource` seed you got missed cleanup and a `LookupError`. Teams running multi-config sandbox evals should expect stray billable sandboxes to...

fix(cti_realm): propagate threat-alignment judge API errors instead of scoring 0.0 (#2238) UKGovernmentBEIS/inspect_evals

by Chen Yufeiyang

`_score_cti_threat_alignment` wrapped the judge `generate` call in a blanket `except Exception` returning `0.0`, so a judge outage zeroed the highest-weighted trajectory component (C0, weight 1.25) while the run still reported `status: success`. API errors now propagate instead of being scored as failures, which means a `cti_realm` run can fail loudly where it used to hand you a quietly wrong n...

fix(infinite_bench): align Code.Run scoring with official rule (#2429) UKGovernmentBEIS/inspect_evals

by Sylvester Kaczmarek

The prompt-matched template scorer stays the primary Code.Run metric but now extracts signed integers that carry terminal punctuation or quotes, while the pinned official InfiniteBench last-token scorer is reported as a separate metric; the template scorer also rejects decimal prefixes, identifiers, and an earlier answer followed by an empty final template. Historically recorded Code.Run scores...

Quick answers

What shipped in Safety Evals on November 19, 2026?
Two fixes to `inspect log recover` and `recover_eval_log()` close ways a crafted `.buffer` directory could write or read files outside the recovery sandbox, on top of sandbox lifecycle fixes for E2B, Modal, Daytona and Podman and a scoring-fidelity change in InfiniteBench's Code.Run. In total, 17 commits and 18 pull requests landed.
Who contributed to Safety Evals on November 19, 2026?
4 developers shipped this update, including Ransom Richardson, Craig Walton, sylvesterkaczmarek, and feiiiiii5.
What were the notable Safety Evals updates?
fix: confine temporary attachment files during crashed-log recovery (#6626), fix: a malformed hash in a shared-buffer manifest can no longer make log recovery open a file outside the buffer (#6627), and fix: missed sandbox cleanup and LookupError on E2B, Modal and Daytona with several sandbox configurations or an empty SampleSource seed (#6633).