$ the-wire · showcase
promptfoo restores relative prompt labels, PyRIT adds message-side tool call scoring
By RepoJournal · Filed · About AI Red Teaming · Composed from the cited sources · methodology
Two operator-facing regressions from promptfoo's 0.123.1 line get fixed (relative labels for file prompts, numeric env values in config), PyRIT ships a scorer that works without a trace pipeline, and garak fixes a WebSocket loop bug that silently returned empty messages.
fix(config): restore relative labels for file prompts promptfoo/promptfoo
Relative prompt labels are back: prompt files are still read via absolute paths, but labels and prompt IDs are again relative to the working directory, so provider/test prompt references and --filter-prompts match what 0.123.1 used. Parent-directory globs and executable prompts keep their current path-resolution fixes, and only processor-generated filename components are normalized, so authored...
fix(providers): show the global install command for opt-in SDKs promptfoo/promptfoo
Since 0.123.1 the Slack, WatsonX, OpenAI Agents, Transformers, Langfuse and Codex Security SDKs are opt-in installs resolved from promptfoo's own install location, and the missing-package error now prints the command that actually works for a global install, e.g. npm install promptfoo @slack/web-api@^8.1.1. If you run promptfoo globally and previously followed the old per-project npm install hi...
FEAT: add MessageToolCallScorer for tool calls recorded in stored messages microsoft/PyRIT
MessageToolCallScorer is the message-side counterpart to OtelToolCallScorer: it scores ToolsCalled from the function-call pieces PyRIT already stores, so you can grade tool-call behavior without standing up a trace pipeline. A tool counts when a model-authored function_call piece is paired by call ID with a later function_call_output in the scored conversation up to that message, the same scope...
fix: reuse the WebSocket event loop across calls NVIDIA/garak
In garak's WebSocket generator, _call_model opened and closed a fresh event loop on every prompt while keeping self.websocket, so later sends ran against a closed loop and came back as an empty message. The generator now holds one loop across calls and drops it when pickling, which matters if any of your runs showed empty model output after the first prompt.
fix(config): accept numeric and boolean env values promptfoo/promptfoo
On the config side, the env: block again accepts numbers and booleans and coerces them to strings. A config with PROMPTFOO_EVAL_TIMEOUT_MS: 10000 used to make promptfoo validate config exit 1 with "Invalid input: expected string, received number", and entries like it plus an unrelated ApiKey scoring fix in garak round out a day of small operational repairs rather than new capability.
