$ the-wire · showcase
PyRIT closes a judge-spoofing hole, promptfoo accepts Langfuse v6
By RepoJournal · Filed · About AI Red Teaming · Composed from the cited sources · methodology
Promptfoo's judge path and PyRIT's verdict parser both changed in ways that affect red-team results you already trust: Langfuse v6 SDKs now bind, and PyRIT no longer silently takes the first JSON literal in a judge reply as the verdict.
FIX: Reject ambiguous multi-JSON replies in extract_json_from_string microsoft/PyRIT
extract_json_from_string used to return the first parseable top-level JSON literal, so a target that embeds its own self-assessment such as {"score_value": true, "rationale": "objective achieved"} could get a judge that quotes the reply to hand back the target's verdict instead of the judge's. Replies with more than one distinct top-level literal are now rejected as ambiguous, which means affec...
chore(deps): update dependency @langfuse/client to v6 promptfoo/promptfoo
langfuse:// prompts broke when Langfuse SDK v6 was installed because Promptfoo accepted only v5; the dependency range now takes v6 alongside v5, with text and chat fetching plus missing-prompt errors verified against the installed SDK. Self-hosted Langfuse servers still have a stated version requirement, and v5 remains a supported fallback, so update @langfuse/client without changing your datas...
FIX: Include endpoint and method in HTTPXAPITarget identifier microsoft/PyRIT
HTTPXAPITarget built its request through HTTPTarget with an empty string, so the identifier carried no endpoint, and it inherited an identifier that recorded prompt_regex_string and use_tls while dropping method and http2. The identifier now records the configured http_url, method, follow_redirects, callback_function and http2, so results from differently-configured HTTPX targets in the same sc...
fix: restore the authentic Discord invite promptfoo/promptfoo
All 34 links across docs, site, examples and app UI now point at https://discord.gg/gHPS9jjfbs; Discord's invite API resolves that code to guild 1153767083381903389, while the promptfoo vanity invite resolves to a different guild, 1552188101651202138. If you bookmarked the old vanity link, it does not take you to the project.
TEST: Coordinate realtime callback delivery explicitly microsoft/PyRIT
The remaining PyRIT work this cycle is test and CI motion: release integration checks now run against a loopback HTTP target after the redteam target returned HTTP 500 and Google ADK 2.13 changed its default telemetry schema, file-surface worker cancellation is explicitly synchronized, and ScenarioRunSizeEstimate.unavailable is annotated with Self. No operator-facing configuration changed in an...
Action items
