$ the-wire · showcase
inspect_ai 0.3.288 drops google-genai 2.x
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
inspect_ai 0.3.288 makes google-genai 3.0.0 a hard requirement, changes what the generated Docker sandbox provides by default, and verifies every sandbox-tools binary against SHA256SUMS before injection.
v0.3.288 UKGovernmentBEIS/inspect_ai
The release notes lead with breaking changes: environments pinned to `google-genai` 2.x now fail the `validate_google_client` version check and need 3.0.0 or later, evaluations using the automatically generated Docker sandbox must supply an explicit image if they rely on the `nonroot` account, `pipx`, or Playwright browser dependencies, and calls to `web_browser()` or its `web_browser_tools()` ...
fix!: require google-genai>=3.0.0, accept Gemini /v1/ on sandbox proxy, default Developer API to v1beta (#6352) UKGovernmentBEIS/inspect_ai
The sandbox agent bridge proxy previously accepted only `/v1beta/models/*` and `/models/*`, so a client on `google-genai` 3.0.0 with its new `/v1/models/...` default got a 404; the proxy now accepts `/v1/` while the Developer API defaults to v1beta.
fix: verify every non-dev sandbox-tools binary before inject (#6342) UKGovernmentBEIS/inspect_ai
A local or cached sandbox-tools binary could reach injection without a digest check, including a PyPI install that fell back to downloading from S3 after its bundled glibc binary was missing. `_open_executable` now reads and verifies each published-name binary against `SHA256SUMS` first.
Take the runtime helpers from inspect-evals-utils (#2833) UKGovernmentBEIS/inspect_evals
The runtime helpers under `inspect_evals.utils` (`deps_utils`, `huggingface`, `load_dataset`, `code`, `messages`, `prompts`, `shuffle`) now re-export `inspect_evals_utils` with no deprecation warning, and `constants.INSPECT_EVALS_CACHE_PATH` comes from its cache module, so existing imports keep working; standalone eval repos on inspect-evals-template v2.0 use the same package instead of copying...
docs(design): code execution from a malicious eval log, as built (#6189) UKGovernmentBEIS/inspect_ai
The malicious-log code-execution design is now written up as built, covering `trust_log`, the metric-option allow-list, resolve-first edit and recovery, eval-set's sandbox, `model_info()` direct lookup, and `inspect score`'s scorer names and records. Separately, the working-time-under-concurrency design doc was rewritten to match its merged implementation.
Action items
- → Install google-genai>=3.0.0 before upgrading inspect_ai to 0.3.288 UKGovernmentBEIS/inspect_ai [immediate]
- → Replace web_browser() and web_browser_tools() calls with the new web_search equivalents UKGovernmentBEIS/inspect_ai [plan]
- → Set an explicit Docker image in generated-sandbox evals that need nonroot, pipx, or Playwright UKGovernmentBEIS/inspect_ai [plan]
