$ the-wire · showcase
OpenAI compaction stops crashing on image and file messages
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
inspect_ai fixed a pydantic ValidationError in OpenAI native compaction that any eval passing an image or file into a user message would hit, while inspect_evals queued four new benchmark registrations and a dependency bump.
fix: OpenAI native compaction crashes when a user message contains an image or file UKGovernmentBEIS/inspect_ai
With the OpenAI Responses API, Model.compact() raised a pydantic ValidationError ("Input should be a valid string") because the compaction path only accepted text back from the API; it now handles image and file content. If you run OpenAI evals with multimodal user messages and native compaction enabled, the crash surfaces mid-run, so pull the fix before the next sweep.
Register submission: FinQA: Numerical Reasoning over Financial Data UKGovernmentBEIS/inspect_evals
The register submission engine opened register/finqa/eval.yaml for FinQA, numerical reasoning over financial data, drawn from arXiv 2109.00122v3. The derived title and description in that YAML are editable before merge, so anyone who knows the eval should check the framing before it lands.
Register submission: BaxBench: Backend Application Generation Benchmark UKGovernmentBEIS/inspect_evals
BaxBench, the backend application generation benchmark from arXiv 2502.11844v3, arrived the same way via register/baxbench/eval.yaml. Two of the day's four registrations target how models do applied work (financial QA, backend code) rather than static multiple-choice.
Register submission: MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models UKGovernmentBEIS/inspect_evals
MonitorBench, a chain-of-thought monitorability benchmark from arXiv 2603.28590v2, was registered at register/monitor_bench/eval.yaml. For operators building monitor-style evals, this is the one to watch land; the generated title and description are still open to edit.
chore(deps): bump the actions group across 1 directory with 8 updates UKGovernmentBEIS/inspect_evals
Dependabot moved the actions group across 8 updates, including anthropics/claude-code-action from 1.0.248 to 1.0.269, astral-sh/setup-uv from 10.3.0 to 10.3.3, and actions/setup-python from 7.0.0 to 7.1.0. The CTI-MCQ cyber threat intelligence registration (register/cti_mcq/eval.yaml, arXiv 2406.07599v3) joined the queue alongside it.
Action items
- → Upgrade inspect_ai to a version containing the OpenAI native compaction fix if your evals send images or files in use... UKGovernmentBEIS/inspect_ai [plan]