$ the-wire · showcase
garak 0.17.1 ships hit@k, PyRIT widens SSRF loopback detection
By RepoJournal · Filed · About AI Red Teaming · Composed from the cited sources · methodology
garak 0.17.1 makes hit@k attack success rate part of the report you already read, while PyRIT closes two scan gaps that let real findings slip past it.
v0.17.1 NVIDIA/garak
hit@k lands alongside Python 3.14 support: the metric reports, per prompt, whether at least one of k attempts breached the target, instead of the pooled fraction of all generations, which records a jailbreak landing once in five as 20%. It is currently exposed as terminal output and per probe/detector pair in the final digest entry of report.jsonl.
FIX: Detect 127.0.0.0/8 loopback subnet and abbreviated IPv4 in SSRFOutputScorer (#3233) microsoft/PyRIT
The "Loopback URL Target" pattern in SSRFOutputScorer hardcoded 127.0.0.1, so targets anywhere else in the 127.0.0.0/8 block, and abbreviated IPv4 notations that inet_aton accepts, passed unflagged. Scan results you already trusted understated SSRF exposure.
feat(evaluators): add hit@k attack success rate metric NVIDIA/garak
The estimator follows pass@k from Chen et al. (2021) and is named hit@k because garak's framing is attack success rather than passing. If you tune threshold policies off pooled ASR, the numbers you compare against going forward will be higher by construction.
FIX: Remove the whole refused turn when Crescendo backtracks after a tool call (#3253) microsoft/PyRIT
When Crescendo backtracks after a refused tool call, the whole refused turn is now removed rather than left in the conversation state. Multi-turn runs that mix tool calls with backtracking were carrying turns the attack should have discarded.
report: update report probe tooltip text to be accurate (higher=better) NVIDIA/garak
The report previously told readers both high and low probe scores were better; high is correct, since the HTML report gives resilience rather than ASR and z-scores are higher-is-better. Tooltips now say so, which changes how you read existing reports.
FIX: preserve characters the ASCII art font cannot render microsoft/PyRIT
AsciiArtConverter passed prompts straight to art.text2art, which silently drops characters it has no glyph for: café converted to 3 glyphs and a fully non-ASCII prompt converted to the empty string.
Action items
- → Upgrade PyRIT before your next SSRF scan so 127.0.0.0/8 and abbreviated IPv4 targets are flagged microsoft/PyRIT [immediate]
- → Upgrade garak to 0.17.1 to get hit@k in report output and the Python 3.14 support NVIDIA/garak [plan]
