$ the-wire · showcase
Grader failures no longer become grades
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
inspect_evals shipped two scoring-accuracy fixes that change what your eval numbers mean: failed graders now leave samples unscored instead of silently recording a zero, and MakeMeSay's judge parser reads the verdict the judge actually stated.
fix(sciknoweval): route grader failures to unscored, not fabricated grades UKGovernmentBEIS/inspect_evals
The three model-graded scorers in `sciknoweval/scorer.py` used to write 0 when the grader's reply could not be parsed. In `model_graded_score` that 0 sits outside the 1 to 5 scale every `type: score` prompt defines, so a grader miss was scored worse than the worst rating the instrument can express and stayed in the denominator; in `model_graded_mcq` the 0 collided with option E's own score, so ...
fix(makemesay): robust judge awareness verdict parsing (4-A) (#2396) UKGovernmentBEIS/inspect_evals
`parse_judge_awareness_verdict` located the verdict with `re.search(r'[01]', text)`, so any judge that quoted the input or reasoned before answering was read off the first 0 or 1 in its reply; a response containing "Sentence 1" was logged as aware. The judge prompt now asks for a final `ANSWER: 1` or `ANSWER: 0` line and the parser reads the last line carrying that label, so expect self-awarene...
fix: harden checkpoint snapshot IDs against option injection (#6580) UKGovernmentBEIS/inspect_ai
Restic snapshot operands now carry an option terminator, so a snapshot ID that begins with a dash can no longer be interpreted as a restic option. The maintainers state there are no imminent exploitable vulnerabilities here, and existing egress and resume validation already covered current callers: this closes a residual path for a new caller rather than a live hole.
fix: replay pairs each tool call with its own approvals and sentinel checks when a call id repeats UKGovernmentBEIS/inspect_ai
A provider that reuses ids, a model output cache hit, or an agent bridge handing the same call out twice can repeat a tool call id, and replay paired approvals and sentinel checks with calls by order in the log while the viewer guessed from arguments. Each tool call is now matched with its own approvals and sentinel checks, which means replayed results from runs with repeated ids should be re-c...
Register submission: τ-Rec: A Verifiable Benchmark for Agentic Recommender Systems UKGovernmentBEIS/inspect_evals
The register submission engine added τ-Rec, a verifiable benchmark for agentic recommender systems, as `register/tau_rec/eval.yaml` from the submitter's arXiv paper. Nothing further changed across either repo today: the two remaining stories are a docs addition to SECURITY.md and an internal ACP session lifecycle cleanup.
Action items
- → Re-run sciknoweval scores before comparing against earlier results, since grader failures now land as unscored instea... UKGovernmentBEIS/inspect_evals [plan]