$ the-wire · showcase
Timeouts stop scoring as wrong answers, shared messages survive messages_df()
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
Inspect 固定 three silent measurement biases today: scorer timeouts that landed in accuracy denominators, a dataframe that dropped rows, and a Moonshot provider that overwrote Kimi K3 reasoning effort with max.
fix: math() timeouts no longer look like wrong answers in eval logs (#6384) UKGovernmentBEIS/inspect_ai
The per-release scorer's two timeout branches now attach a reason to the returned Score instead of leaving it blank, so a slow host can no longer be counted as a wrong answer in the accuracy denominator or silently dropped from metrics. The release notes state both branches now set `rea...` where previously only the private `metadata["math_scorer_status"]` carried the trace.
fix: messages_df() keeps messages that several samples share (#6389) UKGovernmentBEIS/inspect_ai
`messages_df()` used to drop a row whenever an earlier sample carried the same `message_id`, which hit every system message that `react()` reuses across samples. The serial read and the parallel merge now dedupe on `(sample_id, message_id)`, so each sample keeps its rows for shared messages and events; logs and scores themselves were never affected.
fix(moonshot): honor low and high reasoning effort for Kimi K3 (#5946) UKGovernmentBEIS/inspect_ai
The Moonshot provider forced every Kimi K3 reasoning-effort value to `max`, including the supported `low` and `high` settings, so an eval configured for low effort was not actually run cheap. K3 now receives `low`, `high`, and `max` unchanged, with `none` and `minimal` mapped to `low`, `medium` to `high`, and `xhigh` to `max`. Re-run any K3 eval whose results you read as low-effort runs.
fix(computer): type action doesn't use Return for newlines (#6490) UKGovernmentBEIS/inspect_ai
The computer-use `type` action no longer presses Return to produce newlines, a behavior change operators of the computer tool will notice in multi-line text entry. Check any task that relies on newline-separated input being submitted a line at a time.
build: drop the uv cooldown for in-house packages (#213) safety-research/petri
Petri's uv config now sets `inspect-ai` and `inspect-scout` to `false` under `[tool.uv.exclude-newer-package]`, so in-house packages skip the cooldown while the global `exclude-newer = "7 days"` stays in place; `anthropic`, `google-genai`, and `openai` remain at `"6 hours"`. The dependency-cooldown test was updated to enforce the split.
Action items
- → Upgrade inspect_ai to a build containing the math() timeout fix if you rely on accuracy figures from timed-out evals UKGovernmentBEIS/inspect_ai [plan]
- → Re-run Kimi K3 evals that specified low or high reasoning effort, which the Moonshot provider was silently rewriting ... UKGovernmentBEIS/inspect_ai [plan]