$ the-wire · showcase
PyRIT scores ASSERT conversations standalone, garak splits product-key intents per product
By RepoJournal · Filed · About AI Red Teaming · Composed from the cited sources · methodology
14 people shipped this
Fahmid-Arman
@Fahmid-Arman
1 cited
richlundeen
@richlundeen
1 cited
patriciapampanelli
@patriciapampanelli
1 cited
leondz
@leondz
1 cited
sandeeptv
@sandeeptv
1 cited
mldangelo
@mldangelo
mldangelo-oai
@mldangelo-oai
renovate
@renovate
vayllon301
@vayllon301
pcbeingused333
@pcbeingused333
WaqarAnwar
@WaqarAnwar
Kaap10
@Kaap10
immu4989
@immu4989
Jeffrey Martin
@jmartin-tech
PyRIT ships standalone ASSERT policy-violation and overrefusal scoring, and garak stops applying the Windows key-format detector to PlayStation, Xbox and Photoshop probes, so existing garak results for those leaves will change.
FEAT: Add standalone ASSERT conversation scoring microsoft/PyRIT
PyRIT can now score retained conversations for ASSERT policy violations and overrefusal without rerunning the conversation, with optional category validation and policy-flag correction; native scores and evidence are preserved and unreadable media is rejected before judging. Executed usage examples ship alongside it.
Split product-key intent into per-product intents NVIDIA/garak
The single S003productkeys leaf is replaced by per-product leaves: S003productkeyswin10 and S003productkeyswin11 map to productkey.Win5x5, while the playstation, xbox and photoshop leaves have no detector yet and only run under run.serve_detectorless_intents. The Playstation/Xbox/Photoshop leaves previously had the Windows key-format detector applied to them, which the PR describes as producing...
feat(cli): add intent typology listing NVIDIA/garak
`garak --list_intents` lists the intent typology grouped by C, T, M and S, marks which intents have detector or stub content, and supports verbose output plus `--spec` inclusion and exclusion filters. Invalid intent codes now point you at the flag instead of failing opaquely.
fix(assertions): apply weight 0 to throwing python and ruby file assertions promptfoo/promptfoo
A Python or Ruby file assertion that raises an exception now honors `weight: 0`: the error and a score of zero are still recorded, and the row can pass, where previously the exception failed the row regardless. With the default or a positive weight, the same exception still fails it.
probe: add sibling probe `goodside.ThreatenIntent` NVIDIA/garak
garak adds `goodside.ThreatenIntent`, a sibling of `goodside.ThreatenJSON` implementing the strongarm tactic that tries to force a stub through; the JSON and no-padding parts of the original technique are not mapped into intents. The rest is routine: promptfoo repaired its Bedrock Agents Classic session and specialist-routing docs, kept trailing Google Live and Vertex Live transcripts, and took...
