RepoJournal
Safety Evals Safety Evals
52 wires and counting

$ follow Safety Evals

Keep up with Safety Evals in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.

or

fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.

Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.

$ status

wire 2026-11-16
stories 14

© 2026 RepoJournal Home Showcase How it works Privacy

$ the-wire · showcase

OpenAI compaction stops crashing on image and file messages

By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology

inspect_ai fixed a pydantic ValidationError in OpenAI native compaction that any eval passing an image or file into a user message would hit, while inspect_evals queued four new benchmark registrations and a dependency bump.

fix: OpenAI native compaction crashes when a user message contains an image or file UKGovernmentBEIS/inspect_ai

by tim-hua-01

With the OpenAI Responses API, Model.compact() raised a pydantic ValidationError ("Input should be a valid string") because the compaction path only accepted text back from the API; it now handles image and file content. If you run OpenAI evals with multimodal user messages and native compaction enabled, the crash surfaces mid-run, so pull the fix before the next sweep.

Register submission: FinQA: Numerical Reasoning over Financial Data UKGovernmentBEIS/inspect_evals

by github-actions

The register submission engine opened register/finqa/eval.yaml for FinQA, numerical reasoning over financial data, drawn from arXiv 2109.00122v3. The derived title and description in that YAML are editable before merge, so anyone who knows the eval should check the framing before it lands.

Register submission: BaxBench: Backend Application Generation Benchmark UKGovernmentBEIS/inspect_evals

by github-actions

BaxBench, the backend application generation benchmark from arXiv 2502.11844v3, arrived the same way via register/baxbench/eval.yaml. Two of the day's four registrations target how models do applied work (financial QA, backend code) rather than static multiple-choice.

Register submission: MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models UKGovernmentBEIS/inspect_evals

by github-actions

MonitorBench, a chain-of-thought monitorability benchmark from arXiv 2603.28590v2, was registered at register/monitor_bench/eval.yaml. For operators building monitor-style evals, this is the one to watch land; the generated title and description are still open to edit.

chore(deps): bump the actions group across 1 directory with 8 updates UKGovernmentBEIS/inspect_evals

by dependabot

Dependabot moved the actions group across 8 updates, including anthropics/claude-code-action from 1.0.248 to 1.0.269, astral-sh/setup-uv from 10.3.0 to 10.3.3, and actions/setup-python from 7.0.0 to 7.1.0. The CTI-MCQ cyber threat intelligence registration (register/cti_mcq/eval.yaml, arXiv 2406.07599v3) joined the queue alongside it.

Quick answers

What shipped in Safety Evals on November 16, 2026?
inspect_ai fixed a pydantic ValidationError in OpenAI native compaction that any eval passing an image or file into a user message would hit, while inspect_evals queued four new benchmark registrations and a dependency bump. In total, 4 commits and 10 pull requests landed.
Who contributed to Safety Evals on November 16, 2026?
3 developers shipped this update, including tim-hua-01, github-actions, and dependabot.
What were the notable Safety Evals updates?
fix: OpenAI native compaction crashes when a user message contains an image or file, Register submission: FinQA: Numerical Reasoning over Financial Data, and Register submission: BaxBench: Backend Application Generation Benchmark.