RepoJournal
Safety Evals Safety Evals
52 wires and counting

$ follow Safety Evals

Keep up with Safety Evals in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.

or

fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.

Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.

$ status

wire 2026-11-17
stories 50

© 2026 RepoJournal Home Showcase How it works Privacy

$ the-wire · showcase

Remote MCP credentials no longer land in eval logs

By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology

inspect_ai closed a plaintext secret leak in eval logs, and the surrounding projects shipped operator-facing fixes that change what your runs actually measure.

fix: remote MCP server credentials are no longer written to eval logs (#6492) UKGovernmentBEIS/inspect_ai

by Charles Teague

If you run an HTTP MCP server with execution="remote" and pass authorization= or headers=, those credentials were written in plaintext to the eval log three times per sample: in ModelEvent.tools[].options.headers, in StateEvent changes to TaskState.tools, and in the recorded provider request. They are no longer logged. Any log you already produced from such a run still contains them.

Add MBPP run configurations and an agentic default solver UKGovernmentBEIS/inspect_evals

by GuidoBergman

MBPP moves to a run-config layout: default.yaml runs an agentic coding evaluation with Python and submit tools on the sanitized dataset and five independent agent runs per problem, with temperature, reasoning effort and generation token limits left unset so model and provider config apply. single_turn.yaml preserves the previous three-shot generation settings, and task-level pass@k reducers are...

fix: apply tag filters when selecting seeds and dimensions by tag safety-research/petri

by vayllon301

Selecting seeds or judge dimensions with a tags: selector silently ignored the separate tags= filter, so evaluation inputs or dimensions outside your requested subset could be pulled in. Both arguments are now honored together; no migration is required.

fix: keep removed seed tools removed after restart_conversation(keep_tools=True) (#209) safety-research/petri

by Albert Lie 이영덕

restart_conversation(keep_tools=True) is documented to carry over all tools currently available to the target, but a seed tool the auditor had removed came back: the restart began with every seed tool and only re-applied the snapshot on top. Removed seed tools now stay removed across the restart.

fix(analysis): reject out-of-range audit budgets in upfront audit metrics UKGovernmentBEIS/control-arena

by albertlieyingadrian

compute_safety_upfront_audit() and compute_usefulness_upfront_audit() document audit_budget as a percentage in [0, 100] and the safety docstring promises a ValueError out of range, but nothing checked it. An out-of-range budget returned a negative or above-100% audited count instead of failing, so check any saved metric that came from a budget you weren't sure about.

Quick answers

What shipped in Safety Evals on November 17, 2026?
inspect_ai closed a plaintext secret leak in eval logs, and the surrounding projects shipped operator-facing fixes that change what your runs actually measure. In total, 25 commits and 25 pull requests landed.
Who contributed to Safety Evals on November 17, 2026?
7 developers shipped this update, including Charles Teague, Ransom Richardson, GuidoBergman, MattFisher, albertlieyingadrian, vayllon301, and Matt Brandly.
What were the notable Safety Evals updates?
fix: remote MCP server credentials are no longer written to eval logs (#6492), Add MBPP run configurations and an agentic default solver, and fix: apply tag filters when selecting seeds and dimensions by tag.