$ the-wire · showcase
PyRIT alpha fixed for trial count, TAP errors no longer read as refusals
By RepoJournal · Filed · About AI Red Teaming · Composed from the cited sources · methodology
PyRIT shipped fixes that change what your existing scores and attack outcomes actually mean, and moved its Python floor to 3.11, while promptfoo wired in Gemini 4 and the Gemini Decisions API.
FIX: Add a trial-invariant gold-vs-model Krippendorff alpha microsoft/PyRIT
krippendorff_alpha_combined concatenates human and model scores so each scorer trial counts as an extra rater, and the PR documents a scorer that ignores the response climbing from 0.086 at one trial to 0.828 at ten while MAE held at 0.381. The new krippendorff_alpha_gold_model measures a two-row alpha between the gold median human label and the per-response median model score, so the number st...
FIX: Report ERROR when every TAP branch crashed before a score microsoft/PyRIT
When every TAP or PAIR branch threw before anything was scored, the run returned FAILURE at a 0.00 best score, indistinguishable from a target that refused every prompt. It now returns ERROR with the branch error count and the last error as the reason and sets error_message on the result; any-branch-scored runs keep their previous behavior.
[BREAKING] FIX: Keep a private copy's endpoint and credentials the same as its source's (part 17) (#3984) microsoft/PyRIT
A private copy of a target or converter created through the API now inherits its source's endpoint, sign-in, and credentials, with the API refusing copy settings for those fields and the converter settings dialog no longer displaying them. Scripts that set endpoint or credentials on a copy will no longer take effect.
v1.2.0 microsoft/PyRIT
PyRIT v1.2.0 makes the CoPyRIT GUI the headline: reusable scenario configurations, grouped launches, persistent engagements and findings, and richer result analytics, plus new attack techniques, targets, converters, datasets, and scoring options. Python support is now >=3.11, <3.15, so anything still running 3.10 must move before the upgrade.
feat(providers): support Gemini 4 Argon and Hydrogen promptfoo/promptfoo
The provider adds gemini-4-hydrogen, generally available on the Gemini API (free and paid tiers) and Vertex AI, and the same branch carries the earlier Gemini 4 Argon support that Hydrogen's request handling depends on. The quiet remainder: a tsdown bump to 0.23.5, Bedrock Converse cost lookups for every us./global. GPT-6 system profile rather than a fixed pair, and google:decisions:gemini-deci...
Action items
- → Upgrade PyRIT to v1.2.0 on Python 3.11 or newer (3.10 and below are no longer supported) microsoft/PyRIT [plan]
- → Re-baseline any reported Krippendorff alpha against krippendorff_alpha_gold_model; the combined value rises with tria... microsoft/PyRIT [plan]
- → Re-run attack suites whose FAILURE results look like blanket refusals: those may now surface as ERROR microsoft/PyRIT [plan]