$ the-wire · showcase
Enum and date tool defaults fail every call; Anthropic compaction usage undercounted
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
Two defects that silently corrupt eval runs shipped fixes today: tools with Enum, date or Path defaults aborted on their first model call, and compacting Anthropic requests reported fewer billed tokens than were actually charged.
fix: tools with an `Enum`, `date` or `Path` parameter default failing every model call (#6458) UKGovernmentBEIS/inspect_ai
A tool whose parameter defaulted to a plain Enum member, a date or a Path raised TypeError: Object of type Unit is not JSON serializable on the first model call, on every provider including mockllm, so any sample using such a tool failed immediately. The schema now sends the default's JSON value ("celsius") and the tool runs; drop your workaround of defaulting to None.
fix: Anthropic usage undercounted when a request compacts its context (#6390) UKGovernmentBEIS/inspect_ai
For an Anthropic request that compacts its context, ModelUsage omitted billed tokens: streamed responses lost usage.iterations and neither transport summed cache tokens across them. Usage now sums input, output and cache tokens over usage.iterations on both transports, so cost figures from compacting runs are no longer low.
fix: samples_df no longer uses pandas' deprecated drop_duplicates(inplace=True) (#6383) UKGovernmentBEIS/inspect_ai
With pandas 3.1, samples_df(), messages_df() and events_df() emitted Pandas4Warning from four drop_duplicates(inplace=True) calls; the keyword goes away in pandas 4. The calls no longer use it, so analysis pipelines that treat warnings as errors stay quiet.
Action items
- → Upgrade inspect_ai to a build containing the Enum/date/Path tool-default fix before running any eval that uses such t... UKGovernmentBEIS/inspect_ai [immediate]
- → Upgrade inspect_ai before relying on token or cost totals from Anthropic runs that compact context UKGovernmentBEIS/inspect_ai [plan]