RepoJournal
Safety Evals Safety Evals
52 wires and counting

$ follow Safety Evals

Keep up with Safety Evals in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.

or

fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.

Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.

$ status

wire 2026-11-14
stories 43

© 2026 RepoJournal Home Showcase How it works Privacy

$ the-wire · showcase

Timeouts stop scoring as wrong answers, shared messages survive messages_df()

By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology

Inspect 固定 three silent measurement biases today: scorer timeouts that landed in accuracy denominators, a dataframe that dropped rows, and a Moonshot provider that overwrote Kimi K3 reasoning effort with max.

fix: math() timeouts no longer look like wrong answers in eval logs (#6384) UKGovernmentBEIS/inspect_ai

by Bharath Vemula

The per-release scorer's two timeout branches now attach a reason to the returned Score instead of leaving it blank, so a slow host can no longer be counted as a wrong answer in the accuracy denominator or silently dropped from metrics. The release notes state both branches now set `rea...` where previously only the private `metadata["math_scorer_status"]` carried the trace.

fix: messages_df() keeps messages that several samples share (#6389) UKGovernmentBEIS/inspect_ai

by David Tivris

`messages_df()` used to drop a row whenever an earlier sample carried the same `message_id`, which hit every system message that `react()` reuses across samples. The serial read and the parallel merge now dedupe on `(sample_id, message_id)`, so each sample keeps its rows for shared messages and events; logs and scores themselves were never affected.

fix(moonshot): honor low and high reasoning effort for Kimi K3 (#5946) UKGovernmentBEIS/inspect_ai

by Elliott Olson

The Moonshot provider forced every Kimi K3 reasoning-effort value to `max`, including the supported `low` and `high` settings, so an eval configured for low effort was not actually run cheap. K3 now receives `low`, `high`, and `max` unchanged, with `none` and `minimal` mapped to `low`, `medium` to `high`, and `xhigh` to `max`. Re-run any K3 eval whose results you read as low-effort runs.

fix(computer): type action doesn't use Return for newlines (#6490) UKGovernmentBEIS/inspect_ai

by Anthony Duong

The computer-use `type` action no longer presses Return to produce newlines, a behavior change operators of the computer tool will notice in multi-line text entry. Check any task that relies on newline-separated input being submitted a line at a time.

build: drop the uv cooldown for in-house packages (#213) safety-research/petri

by Eric Patey

Petri's uv config now sets `inspect-ai` and `inspect-scout` to `false` under `[tool.uv.exclude-newer-package]`, so in-house packages skip the cooldown while the global `exclude-newer = "7 days"` stays in place; `anthropic`, `google-genai`, and `openai` remain at `"6 hours"`. The dependency-cooldown test was updated to enforce the split.

Quick answers

What shipped in Safety Evals on November 14, 2026?
Inspect 固定 three silent measurement biases today: scorer timeouts that landed in accuracy denominators, a dataframe that dropped rows, and a Moonshot provider that overwrote Kimi K3 reasoning effort with max. In total, 21 commits and 22 pull requests landed.
Who contributed to Safety Evals on November 14, 2026?
5 developers shipped this update, including David Tivris, Elliott Olson, Bharath Vemula, Eric Patey, and Anthony Duong.
What were the notable Safety Evals updates?
fix: math() timeouts no longer look like wrong answers in eval logs (#6384), fix: messages_df() keeps messages that several samples share (#6389), and fix(moonshot): honor low and high reasoning effort for Kimi K3 (#5946).