$ the-wire · showcase
Inspect adds tool state scopes, fixes usage undercounting
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
2 people shipped this
inspect_ai shipped a breaking change giving concurrently running agents their own tool state, and fixed token and cost accounting that undercounted whenever a provider sent several billed requests inside one generate() call.
feat!: tool state scopes so agents running concurrently in one sample keep separate tool state (#5861) UKGovernmentBEIS/inspect_ai
Built-in tools that keep state in the sample store (memory, bash_session, web_browser, skill) used to share that state across every agent in a sample that left instance at its default; a new tool_state_scope() lets concurrent agents such as inspect_swarm members keep separate state. Marked breaking, so check your tool configuration against the new default before upgrading.
fix: usage undercounted when a provider call sends several billed requests (#5643) UKGovernmentBEIS/inspect_ai
Three providers sent more than one billed request inside a single generate() call but reported the usage of only one, so token and cost accounting and limit enforcement undercounted. Anthropic pause_turn continuations now sum usage over the head request and every continuation, with the chain bounded; rerun any cost or token figures produced under the old accounting.
fix: gemini-2.5-pro repeating tool calls after a tool result (#5753) UKGovernmentBEIS/inspect_ai
gemini-2.5-pro repeated tool calls after a tool result far more often through Inspect than through the google-genai SDK, sometimes until a limit stopped the sample. Same-batch trials of check_tools_force traced it to the system hint, which drove extra tool calls in 4 to 8% of samples; the fix sends tool results in the documented shape and drops the hint.
ci: prewarm pytest's assertion-rewrite cache so CI test workers start sooner (#5897) UKGovernmentBEIS/inspect_ai
Prewarming pytest's assertion-rewrite pycs and suspending GC during collection cuts the time before the first CI test runs by roughly 40%, from about 54s per test leg. Contributor tooling only, nothing to change in how you run evals.
Action items
- → Review inspect_ai agents that share a sample with default instance before taking the tool_state_scope() breaking change UKGovernmentBEIS/inspect_ai [plan]
