$ the-wire · showcase
Per-agent sandbox identities, and a math scorer that stops lying
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
inspect_ai lets several agents share one sandbox without sharing a Linux user, while inspect_evals fixes a scorer that was recording genuine antlr4 breakage as a model failure.
fix(math): surface an unusable antlr4 runtime instead of scoring every sample wrong UKGovernmentBEIS/inspect_evals
The math scorer's is_equiv_sympy swallowed every exception and returned False, so a missing or mismatched antlr4-python3-runtime (4.13.2 under sympy 1.14.0, for example) scored every sample wrong, is_equiv_sympy("2", "2") included, with the cause visible only at debug log level. ImportError from parse_latex now propagates instead, so an unusable runtime surfaces as an error rather than as near-...
feat: run each agent's sandbox tools as its own user (#6190) UKGovernmentBEIS/inspect_ai
sandbox_identity(user, cwd=, sandbox=) is a ContextVar scope that makes the built-in exec inside it run as the given Linux user. Several agents can now share one sandbox with per-agent isolation, since built and library-built tools are shared objects whose user has to be set at call time rather than construction.
fix(agent-bridge): Gemini tool calls through the chat completions bridge keep their thought signatures UKGovernmentBEIS/inspect_ai
On the chat completions bridge, messages_from_openai restored only the first reasoning block and dropped its internal field, so for Gemini 3 the thought signature stayed in the assistant text and tool calls went out unsigned. Those calls used to come back as a 400 from Gemini; they now keep their signature across the round trip.
fix(run_evals): skip SKIPPABLE_TASKS up front and make clean() failures non-fatal UKGovernmentBEIS/inspect_evals
The smoke runner no longer dies when clean() raises, such as an OSError: Directory not empty on arvo while an orphaned cybergym download is still writing, and SKIPPABLE_TASKS is now applied up front so skip lists actually keep a task from running. If you run the eval sweep, a stuck cleanup now costs you a chunk rather than the whole run.
feat: experimental transcript events for agent communication and agent swarm evidence UKGovernmentBEIS/inspect_ai
Step one of the swarm event-type design lands experimental transcript events for agent communication and swarm evidence, replacing the InfoEvents that made hiding a swarm's messages also hide every other info event in the viewer. Also merged: the dev extra moves ruff from 0.9.6 to 0.17.2 and the python-packages group takes 36 dependency bumps, plus a fix that stops core_bench from constructing ...
