$ the-wire · showcase
PyRIT resumable scenario runs, correct jailbreak counts
By RepoJournal · Filed · About AI Red Teaming · Composed from the cited sources · methodology
PyRIT's day belongs to operators who pause and resume red-team campaigns: resuming now rebuilds the seed groups the run planned, so dataset edits no longer break it, and FuzzerGenerator stops over-counting jailbreaks from multi-score scorers.
[BREAKING] FEAT: Resume scenario runs from their stored planned seed groups (seed 1) microsoft/PyRIT
Resuming a scenario run now rebuilds its seed groups from a copy stored when the run was created. Until now, editing, refreshing or deleting a dataset under a paused run failed resume with "missing N planned seed group(s)" or "no seeds found in memory", and the new REST edit and delete routes made that an everyday action. This is a breaking change to how resume reads its state.
FIX: Count each jailbroken response once in FuzzerGenerator microsoft/PyRIT
FuzzerGenerator counted a jailbreak once per passing score, so a scorer returning several scores per response, such as AzureContentFilterScorer with one per harm category, inflated the count, repeated conversations in jailbreak_conversation_ids, and could halt at target_jailbreak_goal_count before that many responses were actually jailbroken. Each response now counts once when any of its scores...
Reset the deployed configuration from a per-branch seed microsoft/PyRIT
GUI configuration edits get written back to the folder holding pyritConfigFileUri, so they persist through later deployments; the new optional pyritConfigSeedUri points at a credential-free Blob folder with one subfolder per deployed branch, and deploy_app.sh replaces the deployed configuration from it before the database migration. Each pipeline deploy now starts from the configuration you see...
FEAT: Scanner dataset selection: required, linked and first datasets (-5.5) microsoft/PyRIT
Part of the ASSERT stack work: scanner dataset selection now distinguishes required, linked and first datasets. The design notes are linked from the PR if you build scanners against ASSERT.
fix(eval): preserve custom filters in JSON prompts promptfoo/promptfoo
Valid JSON prompts now render through the same configured Nunjucks environment as text prompts, and JSON.stringify escapes the resulting values, so a filter that produces a quoted label inside a nested JSON message no longer throws SyntaxError when the prompt is parsed. Sweeping up the rest: promptfoo moved oxc-parser from ^0.157.0 to ^0.158.0 and bumped @opentelemetry/context-async-hooks from ...
Action items