$ the-wire · showcase
Empty completions survive, tool history stays intact
By RepoJournal · Filed · About AI Red Teaming · Composed from the cited sources · methodology
promptfoo patched three provider paths that silently turned acceptable responses into errors, and PyRIT shipped API and reporting changes that alter how scores and generated media behave for anyone running eval suites against them.
fix(providers): preserve empty Vertex Claude and Llama completions promptfoo/promptfoo
Vertex Claude and Llama completions that came back empty used to be rejected with "No output found in … API response", so assertions never ran against them; the falsy output check is now a shape check that only flags a response missing its content array. This matches what the native Anthropic provider already did, and follows the same fix landed for AI21 and Hugging Face, so a refusal-style emp...
fix(providers): preserve tool history for Gemini and Anthropic promptfoo/promptfoo
Gemini and Anthropic targets now carry OpenAI-style completed tool turns through their coercion paths: calls keep their result associations, parallel results stay grouped, and native content, system/developer instructions and Gemini signatures survive the round trip. Malformed arguments and missing identifiers are rejected before a request is sent, so broken tool-calling test cases fail early i...
FEAT: Report single-trial scorer metrics alongside the majority vote microsoft/PyRIT
ScorerEvaluator runs each response through the scorer num_scorer_trials times (3 for the published metrics) but then reports only the majority vote for objective accuracy, precision, recall and F1, and the median for harm MAE. The PR's own framing is that the published figures "describe a steadier scorer than the one that runs" in a real scenario; the change reports the single-trial metrics alo...
FIX: Support current Copilot SDK session API microsoft/PyRIT
GitHubCopilotTarget moves off remote_session to the Copilot SDK's opt-in cloud configuration, keeping conversations local and preserving native multi-turn history. Tools, skills, instruction/config discovery, memory, telemetry and background integrations are explicitly disabled, so copilot targets in your suites get an isolation guarantee the tests now assert.
fix(providers): normalize trailing Google AI Studio base URL slashes promptfoo/promptfoo
getGoogleApiBaseUrl returned config.apiBaseUrl and GOOGLE_API_BASE_URL verbatim, and because both callers append /${apiVersion}/models/..., a trailing slash produced URLs like https://proxy.example/google//v1beta/models/... that exact-path proxies reject. Trailing slashes are now normalized, the same fix already applied to the Ollama path. The remaining long tail is CI and dependency work: Linu...