$ the-wire · showcase
Inspect fixes silently dropped screenshots and misread timeouts
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
Two evaluation-correctness bugs landed in inspect_ai today: agents mixing computer() with other tools lost their screenshots, and Docker exec misreported a command's own exit 124 as a timeout, both of which change what your runs actually measure.
fix: computer() keeps its latest screenshots when other tools have also run (#6180) UKGovernmentBEIS/inspect_ai
Any agent running computer() alongside another tool stopped receiving screenshots after that tool's first result, with every computer image, newest included, overwritten by "Screenshot removed to reduce size of input...". The fix restores the latest screenshots; it affected every provider except OpenAI, so only prior runs on OpenAI were trustworthy if you mixed tools.
fix: Docker exec with a timeout returns a command's own exit 124 instead of raising TimeoutError UKGovernmentBEIS/inspect_ai
With the Docker sandbox, sandbox().exec(cmd, timeout=N) raised TimeoutError whenever the command exited 124, so a command like timeout 10 pytest was reported as an exec timeout even though it finished early. A bash(timeout=180) call told the model it ran past 180 seconds after about 10, which means scorer results from sandboxed test suites need a rerun.
fix: tool_with() keeps a tool's current name and descriptions unless they are passed, so renaming an MCP tool works (#6181) UKGovernmentBEIS/inspect_ai
tool_with() used to write None for every name or description argument you didn't pass, wiping what the tool already had; tools built with ToolDef(...).as_tool() lost their description and parameter schema when renamed, falling back to the function's own. Unpassed arguments now leave the existing name and descriptions intact, so renaming an MCP tool no longer strips its metadata.
fix(math): time-limit sympy comparisons without SIGALRM on Windows UKGovernmentBEIS/inspect_evals
expression_exact_match_sympy, one of the MATH task's three default scorers, could not work on Windows at all: is_equiv_sympy timed comparisons with signal.SIGALRM, which Windows lacks, and every comparison returned "not equal". The equivalent Windows commits to inspect_evals, so any Windows MATH scores you collected are invalid.
fix(scicode): download test_data.h5 from a pinned HuggingFace mirror instead of the quota-limited Google Drive file UKGovernmentBEIS/inspect_evals
The SciCode task could not be constructed because downloading test_data.h5 from Google Drive was rejected by quota; it now pulls from a pinned HuggingFace mirror, which unblocks the nightly smoke test and any local SciCode run that hit the same download failure.
chore(ruff): enable PLW1514 and read files in src/ as UTF-8 UKGovernmentBEIS/inspect_evals
The long tail: ruff rule PLW1514 is now enabled and 79 unspecified-encoding sites in src/ were given explicit UTF-8, following the same fix in tests/ and tools/, which ends non-ASCII read failures on Windows (cp1252) and ASCII CI locales. The sample buffer's nine shutdown booleans became two state machines and CI now installs dependencies through shared Meridian composite actions, both internal...
Action items
- → Re-run any inspect_ai evals where Docker sandbox exec calls reported timeouts, and any tool-mixing runs on non-OpenAI... UKGovernmentBEIS/inspect_ai [plan]
- → Discard MATH task scores produced on Windows before the sympy timeout fix landed UKGovernmentBEIS/inspect_evals [plan]
