$ the-wire · showcase
Inspect 0.3.283 breaks datasets that use a `checkpoint` field
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4)
9 people shipped this
Inspect 0.3.283 starts reading a top-level `checkpoint` field into every dataset sample and rejecting values it can't validate, which will hit any dataset that already uses that name for something else.
v0.3.283 UKGovernmentBEIS/inspect_ai
Dataset loaders now read a top-level `checkpoint` field as the sample's checkpoint configuration and raise when it isn't valid, so datasets that happen to use `checkpoint` for unrelated data will fail to load. The escape hatches are explicit: pass `FieldSpec(checkpoint=None)` to skip the field, or `FieldSpec(checkpoint="<column>")` to read checkpoint settings from another column, and note that ...
fix!: preserve sample checkpoint settings when loading datasets (#5411) UKGovernmentBEIS/inspect_ai
The migration half of the same change: loading a dataset used to silently drop `Sample.checkpoint`, losing per-sample overrides such as a failure tolerance of `0` or empty capture paths on write-and-reload round trips. Those settings now survive, including nested snapshot configuration and zero or empty overrides.
docs(design): code execution from a malicious eval log (#5728) UKGovernmentBEIS/inspect_ai
A design doc for a trust model on eval logs commits to removing code-import paths from read operations: `recompute_metrics`, `edit_score`, log recovery and eval-set resume can all currently import files or construct models named by a log, and eval-set resume lets a log replace your task sandbox. No behavior has shipped yet, but operators who score or resume logs from untrusted sources should re...
AssistantBench: replace deprecated browser with Playwright MCP UKGovernmentBEIS/inspect_evals
AssistantBench swaps the deprecated `web_browser()` for a sandboxed Playwright MCP, keeping the same agent, prompt, search, dataset and scorer; it needs Inspect 0.3.264+ via the `assistant_bench` extra and bumps comparability version to 6-B, so scores are not directly comparable with earlier runs.
fix(inspect 0.3.278 compatibility) UKGovernmentBEIS/inspect_evals
inspect_evals catches up to Inspect 0.3.278: `openai` moves from 3.1.0 to 3.4.0 and HLE tests now expect `PrerequisiteError` where Inspect used to raise `LookupError`. Only matters if you are pinning either package alongside inspect_evals.
fix: preserve monitor views for repeated tool calls UKGovernmentBEIS/control-arena
Two Control Arena scorer fixes change what the monitor sees: repeated calls to the same attack tool no longer collide under the tool's function name, and `has_backdoor` now compares complete results rather than the first line of a multiline repr. Scores produced under the old comparison are not directly comparable.
build: take the docs group out of dev so uv sync skips quarto-cli (#185) safety-research/petri
Petri's `doc` group is out of `dev`, so a plain `uv sync` or `uv run` no longer pulls `quarto-cli`, which had no wheel and was built from sdist on every job. `types-PyYAML` moves to `dev` because Pyright needs it.
Action items
- → Audit datasets that use a `checkpoint` column before moving to inspect_ai 0.3.283; set `FieldSpec(checkpoint=None)` w... UKGovernmentBEIS/inspect_ai [plan]
