RepoJournal
Safety Evals Safety Evals
52 wires and counting

$ follow Safety Evals

Keep up with Safety Evals in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.

or

fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.

Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.

$ status

wire 2026-11-18
stories 57

© 2026 RepoJournal Home Showcase How it works Privacy

$ the-wire · showcase

Grader failures no longer become grades

By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology

inspect_evals shipped two scoring-accuracy fixes that change what your eval numbers mean: failed graders now leave samples unscored instead of silently recording a zero, and MakeMeSay's judge parser reads the verdict the judge actually stated.

fix(sciknoweval): route grader failures to unscored, not fabricated grades UKGovernmentBEIS/inspect_evals

by melcheikh

The three model-graded scorers in `sciknoweval/scorer.py` used to write 0 when the grader's reply could not be parsed. In `model_graded_score` that 0 sits outside the 1 to 5 scale every `type: score` prompt defines, so a grader miss was scored worse than the worst rating the instrument can express and stayed in the denominator; in `model_graded_mcq` the 0 collided with option E's own score, so ...

fix(makemesay): robust judge awareness verdict parsing (4-A) (#2396) UKGovernmentBEIS/inspect_evals

by Martín el Cheikh

`parse_judge_awareness_verdict` located the verdict with `re.search(r'[01]', text)`, so any judge that quoted the input or reasoned before answering was read off the first 0 or 1 in its reply; a response containing "Sentence 1" was logged as aware. The judge prompt now asks for a final `ANSWER: 1` or `ANSWER: 0` line and the parser reads the last line carrying that label, so expect self-awarene...

fix: harden checkpoint snapshot IDs against option injection (#6580) UKGovernmentBEIS/inspect_ai

by Craig Walton

Restic snapshot operands now carry an option terminator, so a snapshot ID that begins with a dash can no longer be interpreted as a restic option. The maintainers state there are no imminent exploitable vulnerabilities here, and existing egress and resume validation already covered current callers: this closes a residual path for a new caller rather than a live hole.

fix: replay pairs each tool call with its own approvals and sentinel checks when a call id repeats UKGovernmentBEIS/inspect_ai

by dragonstyle

A provider that reuses ids, a model output cache hit, or an agent bridge handing the same call out twice can repeat a tool call id, and replay paired approvals and sentinel checks with calls by order in the log while the viewer guessed from arguments. Each tool call is now matched with its own approvals and sentinel checks, which means replayed results from runs with repeated ids should be re-c...

Register submission: τ-Rec: A Verifiable Benchmark for Agentic Recommender Systems UKGovernmentBEIS/inspect_evals

by github-actions

The register submission engine added τ-Rec, a verifiable benchmark for agentic recommender systems, as `register/tau_rec/eval.yaml` from the submitter's arXiv paper. Nothing further changed across either repo today: the two remaining stories are a docs addition to SECURITY.md and an internal ACP session lifecycle cleanup.

Quick answers

What shipped in Safety Evals on November 18, 2026?
inspect_evals shipped two scoring-accuracy fixes that change what your eval numbers mean: failed graders now leave samples unscored instead of silently recording a zero, and MakeMeSay's judge parser reads the verdict the judge actually stated. In total, 25 commits, 31 pull requests, and 1 releases landed.
Who contributed to Safety Evals on November 18, 2026?
6 developers shipped this update, including Craig Walton, Charles Teague, Ransom Richardson, brandly, github-actions, and melcheikh.
What were the notable Safety Evals updates?
fix(sciknoweval): route grader failures to unscored, not fabricated grades, fix(makemesay): robust judge awareness verdict parsing (4-A) (#2396), and fix: harden checkpoint snapshot IDs against option injection (#6580).