$ the-wire · showcase
PyRIT fences judge transcripts against role forgery
By RepoJournal · Filed · About AI Red Teaming · Composed from the cited sources · methodology
PyRIT closed a scorer bypass where a target reply could forge its own conversation turns, and its scenario constructors change shape in a breaking release; garak fixed two crashes that killed runs mid-flight.
FIX: Fence conversation transcript turns against role-line forgery microsoft/PyRIT
ConversationScorer._render_conversation emitted flat "{role}: {text}" lines, so a target reply containing literal User: or Assistant: lines forged extra turns inside its own turn and the wrapped judge scored a rewritten conversation. The turns are now fenced, which matters directly to anyone running ConversationScorer with SelfAskTrueFalseScorer for TASK_ACHIEVED-style grading: your existing an...
[BREAKING] MAINT: Simplify scenario construction and initializer parameters microsoft/PyRIT
PR 3404 simplifies how scenarios are constructed and which initializer parameters they accept. This is flagged BREAKING, so any run config or code that builds scenarios by hand needs rework before you pick up a release containing it.
probes: don't scale the iteration bound by an unset prompt cap NVIDIA/garak
IterativeProbe.probe() multiplied the attempt queue length by the soft prompt cap without checking one was set, so a run with follow_prompt_cap on and no cap configured died with "unsupported operand type(s) for *: 'int' and 'NoneType'" before probing anything. The bound now falls back to infinity instead of raising.
fix(harnesses): materialize a generator-returning probe's attempt results NVIDIA/garak
Harness.run accepts a generator-returning probe, then _run_detector calls len() on the unmaterialized result and raises "object of type 'generator' has no len()" at the first detector step. The attempts are now materialized, so probes written against the Iterable contract survive the harness instead of aborting the scan.
fix(deps): update protobufjs to 8.8.1 promptfoo/promptfoo
Prompfoo's protobufjs floor moves to 8.8.1 for the length-prefix fix: crafted 2^32 lengths could wrap to zero and be accepted by OTLP decoding, and the new regression confirms the receiver returns HTTP 400 without storing traces or spans for an overflowing message. The rest of the day is dependency alignment: json-schema-ref-parser 16.0.5, Zod 4.6.6, Storybook 10.6.2, aiohttp 3.14.4 and Pillow ...
Action items
- → Re-check PyRIT scenario construction against PR 3404 before adopting the release that contains the breaking initializ... microsoft/PyRIT [plan]
- → Raise protobufjs to 8.8.1 in promptfoo so OTLP decoding rejects wrapped length prefixes promptfoo/promptfoo [plan]
