RepoJournal
AI Red Teaming AI Red Teaming
52 wires and counting

$ follow AI Red Teaming

Keep up with AI Red Teaming in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.

or

fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.

Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.

$ status

wire 2026-11-15
stories 55

© 2026 RepoJournal Home Showcase How it works Privacy

$ the-wire · showcase

Empty completions survive, tool history stays intact

By RepoJournal · Filed · About AI Red Teaming · Composed from the cited sources · methodology

promptfoo patched three provider paths that silently turned acceptable responses into errors, and PyRIT shipped API and reporting changes that alter how scores and generated media behave for anyone running eval suites against them.

fix(providers): preserve empty Vertex Claude and Llama completions promptfoo/promptfoo

by 67-luck

Vertex Claude and Llama completions that came back empty used to be rejected with "No output found in … API response", so assertions never ran against them; the falsy output check is now a shape check that only flags a response missing its content array. This matches what the native Anthropic provider already did, and follows the same fix landed for AI21 and Hugging Face, so a refusal-style emp...

fix(providers): preserve tool history for Gemini and Anthropic promptfoo/promptfoo

by NK-Works

Gemini and Anthropic targets now carry OpenAI-style completed tool turns through their coercion paths: calls keep their result associations, parallel results stay grouped, and native content, system/developer instructions and Gemini signatures survive the round trip. Malformed arguments and missing identifiers are rejected before a request is sent, so broken tool-calling test cases fail early i...

FEAT: Report single-trial scorer metrics alongside the majority vote microsoft/PyRIT

by WatchTree-19

ScorerEvaluator runs each response through the scorer num_scorer_trials times (3 for the published metrics) but then reports only the majority vote for objective accuracy, precision, recall and F1, and the median for harm MAE. The PR's own framing is that the published figures "describe a steadier scorer than the one that runs" in a real scenario; the change reports the single-trial metrics alo...

FIX: Support current Copilot SDK session API microsoft/PyRIT

by shawnrruss

GitHubCopilotTarget moves off remote_session to the Copilot SDK's opt-in cloud configuration, keeping conversations local and preserving native multi-turn history. Tools, skills, instruction/config discovery, memory, telemetry and background integrations are explicitly disabled, so copilot targets in your suites get an isolation guarantee the tests now assert.

fix(providers): normalize trailing Google AI Studio base URL slashes promptfoo/promptfoo

by 67-luck

getGoogleApiBaseUrl returned config.apiBaseUrl and GOOGLE_API_BASE_URL verbatim, and because both callers append /${apiVersion}/models/..., a trailing slash produced URLs like https://proxy.example/google//v1beta/models/... that exact-path proxies reject. Trailing slashes are now normalized, the same fix already applied to the Ollama path. The remaining long tail is CI and dependency work: Linu...

Quick answers

What shipped in AI Red Teaming on November 15, 2026?
promptfoo patched three provider paths that silently turned acceptable responses into errors, and PyRIT shipped API and reporting changes that alter how scores and generated media behave for anyone running eval suites against them. In total, 25 commits and 30 pull requests landed.
Who contributed to AI Red Teaming on November 15, 2026?
9 developers shipped this update, including 67-luck, NK-Works, renovate, mldangelo, aspire488, WatchTree-19, vshalpnjabi, and shawnrruss, and 1 more.
What were the notable AI Red Teaming updates?
fix(providers): preserve empty Vertex Claude and Llama completions, fix(providers): preserve tool history for Gemini and Anthropic, and FEAT: Report single-trial scorer metrics alongside the majority vote.