$ the-wire · showcase
PyRIT makes scenario dataset sizing explicit and starts deduping three loaders
By RepoJournal · Filed · About AI Red Teaming · Composed from the cited sources · methodology
10 people shipped this
Two PyRIT changes land in the same window: a breaking scenario dataset API that makes sources and limits explicit, and loader deduplication that cuts BeaverTails from 166,382 seeds to 14,402, both of which change what your existing attack datasets contain.
[BREAKING] FEAT: Make scenario dataset sources and limits explicit (#2956) microsoft/PyRIT
PyRIT's scenario dataset sources and limits are now explicit, and the change is flagged breaking, so scenario configs that relied on implicit dataset sourcing or default limits need to be updated before an upgrade. Operators running scenario-based campaigns should read this as the migration to plan around.
FIX Dedupe prompts in the BeaverTails, PKU-SafeRLHF and Aegis loaders microsoft/PyRIT
The BeaverTails, PKU-SafeRLHF and Aegis loaders now keep one seed per prompt and merge harm categories from every row carrying them, cutting BeaverTails from 166,382 seeds to 14,402, PKU-SafeRLHF from 73,907 to 38,641, and Aegis from 19,093 to 13,246. Your next refresh of those datasets will be roughly a tenth the size for BeaverTails, though all three stay in the huge size bucket.
feat(assertions): add rouge-l and rouge-s assertion types promptfoo/promptfoo
promptfoo adds rouge-l and rouge-s assertion types, including their not- forms, wired through assertion dispatch, configuration and schema, CSV parsing and the eval creator. If you score summarization or generation output against references, you can now express ROUGE-Lsum and skip-bigram overlap directly in your test config instead of scoring externally; blank text scores zero and ROUGE-N keeps...
feat(assertions): optional Unicode normalization for equals and contains promptfoo/promptfoo
String equals and contains assertions now accept normalizeUnicode: true, so composed and decomposed accents compare equal under NFC. Compatibility forms like NFKC must be named explicitly because they can make x² and x2 compare equal, and omitted or false settings preserve today's comparisons exactly; note it normalizes both strings before comparison only, and does not touch nested object value...
FIX: preserve child-result links when execution fails in SequentialAttack microsoft/PyRIT
When a child of SequentialAttack raised, the parent's error result kept the failed child's result but dropped metadata["child_attack_result_ids"], losing references to children that had finished before the failure. The generic strategy error path now records the persisted error result's attack_result_id, so partial-run results stay traceable. Also worth noting: PyRIT restored CodeQL security sc...
Action items
