$ the-wire · showcase
Remote MCP credentials no longer land in eval logs
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
inspect_ai closed a plaintext secret leak in eval logs, and the surrounding projects shipped operator-facing fixes that change what your runs actually measure.
fix: remote MCP server credentials are no longer written to eval logs (#6492) UKGovernmentBEIS/inspect_ai
If you run an HTTP MCP server with execution="remote" and pass authorization= or headers=, those credentials were written in plaintext to the eval log three times per sample: in ModelEvent.tools[].options.headers, in StateEvent changes to TaskState.tools, and in the recorded provider request. They are no longer logged. Any log you already produced from such a run still contains them.
Add MBPP run configurations and an agentic default solver UKGovernmentBEIS/inspect_evals
MBPP moves to a run-config layout: default.yaml runs an agentic coding evaluation with Python and submit tools on the sanitized dataset and five independent agent runs per problem, with temperature, reasoning effort and generation token limits left unset so model and provider config apply. single_turn.yaml preserves the previous three-shot generation settings, and task-level pass@k reducers are...
fix: apply tag filters when selecting seeds and dimensions by tag safety-research/petri
Selecting seeds or judge dimensions with a tags: selector silently ignored the separate tags= filter, so evaluation inputs or dimensions outside your requested subset could be pulled in. Both arguments are now honored together; no migration is required.
fix: keep removed seed tools removed after restart_conversation(keep_tools=True) (#209) safety-research/petri
restart_conversation(keep_tools=True) is documented to carry over all tools currently available to the target, but a seed tool the auditor had removed came back: the restart began with every seed tool and only re-applied the snapshot on top. Removed seed tools now stay removed across the restart.
fix(analysis): reject out-of-range audit budgets in upfront audit metrics UKGovernmentBEIS/control-arena
compute_safety_upfront_audit() and compute_usefulness_upfront_audit() document audit_budget as a percentage in [0, 100] and the safety docstring promises a ValueError out of range, but nothing checked it. An out-of-range budget returned a negative or above-100% audited count instead of failing, so check any saved metric that came from a budget you weren't sure about.
Action items
- → Upgrade inspect_ai to a release containing the MCP credential fix and audit existing eval logs from remote MCP runs f... UKGovernmentBEIS/inspect_ai [plan]
- → Point MBPP runs at default.yaml or single_turn.yaml explicitly, since the task default is now read from default.yaml UKGovernmentBEIS/inspect_evals [plan]