RepoJournal
Safety Evals Safety Evals
52 wires and counting

$ follow Safety Evals

Keep up with Safety Evals in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.

or

fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.

Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.

$ status

wire 2026-11-15
stories 10

© 2026 RepoJournal Home Showcase How it works Privacy

$ the-wire · showcase

Enum and date tool defaults fail every call; Anthropic compaction usage undercounted

By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology

Two defects that silently corrupt eval runs shipped fixes today: tools with Enum, date or Path defaults aborted on their first model call, and compacting Anthropic requests reported fewer billed tokens than were actually charged.

fix: tools with an `Enum`, `date` or `Path` parameter default failing every model call (#6458) UKGovernmentBEIS/inspect_ai

by Shaan GS

A tool whose parameter defaulted to a plain Enum member, a date or a Path raised TypeError: Object of type Unit is not JSON serializable on the first model call, on every provider including mockllm, so any sample using such a tool failed immediately. The schema now sends the default's JSON value ("celsius") and the tool runs; drop your workaround of defaulting to None.

fix: Anthropic usage undercounted when a request compacts its context (#6390) UKGovernmentBEIS/inspect_ai

by David Tivris

For an Anthropic request that compacts its context, ModelUsage omitted billed tokens: streamed responses lost usage.iterations and neither transport summed cache tokens across them. Usage now sums input, output and cache tokens over usage.iterations on both transports, so cost figures from compacting runs are no longer low.

fix: samples_df no longer uses pandas' deprecated drop_duplicates(inplace=True) (#6383) UKGovernmentBEIS/inspect_ai

by Sanskar Chandawar

With pandas 3.1, samples_df(), messages_df() and events_df() emitted Pandas4Warning from four drop_duplicates(inplace=True) calls; the keyword goes away in pandas 4. The calls no longer use it, so analysis pipelines that treat warnings as errors stay quiet.

Quick answers

What shipped in Safety Evals on November 15, 2026?
Two defects that silently corrupt eval runs shipped fixes today: tools with Enum, date or Path defaults aborted on their first model call, and compacting Anthropic requests reported fewer billed tokens than were actually charged. In total, 4 commits, 5 pull requests, and 1 releases landed.
Who contributed to Safety Evals on November 15, 2026?
3 developers shipped this update, including ShaanGS, tivris, and Sanskar Chandawar.
What were the notable Safety Evals updates?
fix: tools with an `Enum`, `date` or `Path` parameter default failing every model call (#6458), fix: Anthropic usage undercounted when a request compacts its context (#6390), and fix: samples_df no longer uses pandas' deprecated drop_duplicates(inplace=True) (#6383).