$ the-wire · showcase
inspect_ai 0.3.286 enforces sandbox user permissions, and inspect_evals closes a host-takeover hole
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
inspect_ai 0.3.286 breaks Docker file-read behavior that callers relied on while inspect_evals moves model-written candidate code out of the host process entirely, and both changes land on operators who run these tools daily.
v0.3.286 UKGovernmentBEIS/inspect_ai
Docker read_file() now runs as the container's default execution user rather than through Docker's privileged copy API, so any eval harness or solver that counted on reading root-owned paths without saying so will start failing. Ask for user="root" explicitly for privileged reads, and note the new runtime requirements: tar with --no-recursion, /bin/sh, and /usr/bin/timeout. This is a breaking c...
fix(gdm_self_reasoning): isolate max-tokens candidate execution UKGovernmentBEIS/inspect_evals
On the max-tokens task, a submission could control verifier termination and cross the evaluation isolation boundary, which the PR describes as allowing evaluation-host takeover. Candidate execution now runs in a dedicated, network-isolated verifier service, with the supervisor spawning candidate code as a lower-privilege child under no_new_privs. Re-run any results you still trust from that tas...
fix(theagentcompany): stop publishing ownCloud on host ports UKGovernmentBEIS/inspect_evals
TheAgentCompany published ownCloud on a random host port bound to every interface, carrying the admin credential that the task prompts hand to the agent, so other machines reaching the host or agents in concurrent samples could read sample files. It affected 42 task configurations, 38 of the 39 run by default. The ownCloud and Collabora host port mappings are gone and both stay on the internal ...
perf: reduce memory spikes and stalls when finalizing long samples (#4879) UKGovernmentBEIS/inspect_ai
Finalizing a sample under realtime logging held several copies at once; on a 2,000-event x 20KB benchmark that peaked at 2.2x the serialized sample size and stalled the event loop for roughly 120 ms. Long realtime-logged samples should now finalize without those spikes, which matters if you run large sample counts with logging on.
fix: restic checkpoint resume breaking on large repos in the k8s sandbox (#6040) UKGovernmentBEIS/inspect_ai
Checkpoint resume in the k8s sandbox sent the entire host-side restic repo as the input of a single sandbox exec, which broke on large repos. Resume now copies in chunks through a copy_in helper, so resuming long k8s runs no longer depends on the repo fitting in one exec payload.
Action items
- → Upgrade inspect_ai to 0.3.286 and audit any read_file() calls that need privileged or alternate-user reads UKGovernmentBEIS/inspect_ai [immediate]
- → Update inspect_evals before rerunning theagentcompany or gdm_self_reasoning tasks, and re-run prior results from thos... UKGovernmentBEIS/inspect_evals [immediate]
