$ the-wire · showcase
Slimmer inspect-ai sdist, correct dynamic sample retries
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
Inspect's two operator-facing fixes land today: source installs no longer ship the sandbox binaries, and task retries stop borrowing one dynamic sample's result for another.
fix: task retries no longer reuse one dynamic sample's result for another sample without an id (#6314) UKGovernmentBEIS/inspect_ai
A SampleSource sample admitted without an id used to take the next free integer in arrival order, and since retries and checkpoint resume key on (id, epoch) alone, a retried task could reuse a different dynamic sample's result. If you drive evals from next_samples(), sample_complete, sample_abandoned, or enqueue_sample, your retries and resumed logs now resolve to the sample that actually produ...
fix: the sdist no longer includes the sandbox tools binaries; an install built from it downloads them at first use (#6306) UKGovernmentBEIS/inspect_ai
The sdist no longer carries the two glibc sandbox tools binaries (about 29 MB of a 72.3 MB per-release footprint), so an install built from source fetches them on first use instead. If you build your eval images from source behind an air-gapped proxy, the download now happens at runtime rather than install time.
test: move Together tests and docs off retired Together models (#6431) UKGovernmentBEIS/inspect_ai
Together retired four models from serverless inference, which had left 12 live tests failing with 400s such as "Unable to access non-serverless model MiniMaxAI/MiniMax-M2.7." If your eval configs still name those Together model IDs, they will fail the same way; the tests and docs on main now point elsewhere.
MMIU: avoid caching incomplete asset directories UKGovernmentBEIS/inspect_evals
MMIU's cache check was extracted_dir.is_dir(), which a failed download or extraction satisfies anyway, so the next run skipped preparation and died on missing images. Asset preparation now extracts into a temporary sibling and renames it into place, and caches left incomplete by earlier runs or killed downloads are recovered, so existing MMIU installs repair themselves on the next invocation.
docs: design the eval-sharding merge command (inspect log merge-shards) (#6345) UKGovernmentBEIS/inspect_ai
Also landed: a design for inspect log merge-shards LOG --task TASK, which will build the task from inspect eval's own options (-T, --model, --model-role, --epochs) rather than rebuilding it from the log; plus the long tail of contributor docs and CI pinning, including the ruff matrix finally running on its declared Python versions.