Promptfoo 0.124.0 and 0.125.0 make four provider SDKs opt-in

By RepoJournal · composed from the cited sources · human-reviewed weekly · methodology

♥

Across two majors in October, prompts using WatsonX, Slack, Claude Agent, Codex or Jev now need those SDKs installed separately; PyRIT spent the month on attack coverage.

777 commits 904 PRs merged 7 releases 30 briefings covered

all ai-red-teaming reviews →

ai-red-teaming wire →

perf(deps)!: make Slack SDK opt-in promptfoo/promptfoo

by mldangelo-oai

Slack left the default install first, and selecting the provider without the SDK now fails with install guidance rather than a module error. If your CI image builds from a plain npm install, add @slack/web-api@^8.1.1 explicitly or those prompt runs break.

0.124.0 promptfoo/promptfoo

by promptfoobot[bot]

Beyond the WatsonX opt-in, 0.124.0 also drops the hosted ChatKit provider outright, so any config pointing at it needs a replacement before you upgrade. The release notes list it under breaking changes with no shim.

0.125.0 promptfoo/promptfoo

by promptfoobot[bot]

Claude Agent and Codex are no longer installed for you, and jks-js became an optional peer. Teams running Codex against Azure-backed auth need to declare both before 0.125.0 will load the provider.

perf(providers)!: make WatsonX SDKs opt-in promptfoo/promptfoo

by mldangelo-oai

WatsonX and ibm-cloud-sdk-core are now optional peers rather than default dependencies, which trims a default install but requires an explicit install for anyone actually scoring WatsonX models. Missing or unsupported versions report installation guidance at provider selection.

feat(providers): add TypeSafe Jev provider promptfoo/promptfoo

by athul0rameshan9

A new typesafe: provider routes llm-rubric and classifier assertions to TypeSafe's Jev model, which answers typed questions (Noul, Score, Choice) with probabilities instead of generated text. Worth wiring in if you want rubric grading that returns a distribution rather than a string to parse.

v0.17.1 NVIDIA/garak

by jmartin-tech

Garak 0.17.1 adds a hit@k attack success rate metric, exposed in terminal output and per probe/detector pair in the final digest entry of report.json. It also lands fixes to WebSocket and safe-token scanning, so existing scan results will shift slightly on rerun.

fix: preserve log context and validate media paths promptfoo/promptfoo

by mldangelo-oai

Legacy local-media operations now validate path components against the canonical storage root and use no-follow reads where supported; structured Winston logs keep their sanitized context fields. If you run a configured storage backend, upgrade before pointing scans at untrusted paths.

FEAT: Grid composite converter microsoft/PyRIT

by jsong468

Alongside: a GridCompositeConverter that hides prompt text in one cell of an image grid, PyRIT's attack analytics SDK, ProPILE leak probes, char-split and Cyrillic homoglyph attack sources, and a move of garak's probe intents into reusable stubs rounded out the month.

$ ls ai-red-teaming/month/ # the briefings behind this review

Fri Oct 2 Promptfoo drops Slack from default installs Sat Oct 3 WatsonX SDKs go opt-in, Jev arrives as a typed provider Sun Oct 4 Codex provider works on Windows, PyRIT pulls mcp 2.2.0 Mon Oct 5 promptfoo restores relative prompt labels, PyRIT adds message-side tool call scoring Tue Oct 6 promptfoo 0.124.0 makes WatsonX, Langfuse and Slack SDKs opt-in Wed Oct 7 PyRIT grows an attack registry, promptfoo tightens JS assertion grading Thu Oct 8 Promptfoo patches Sharp and Codex SDK, garak fixes WebSocket and safe-token scanning Fri Oct 9 IBM SDK held at 5.6.2 over leaked secret suffix, JSON extraction fixed Sat Oct 10 PyRIT makes scenario dataset sizing explicit and starts deduping three loaders Sun Oct 11 PyRIT ships ProPILE leak probes and a runtime techniques registry Mon Oct 12 Promptfoo drops stale caches, PyRIT ships an attack analytics SDK Tue Oct 13 PyRIT gets readiness gating and single-technique runs, promptfoo fixes repeat stability Wed Oct 14 PyRIT marks unscored objectives UNDETERMINED, and a breaking dataset-controls change lands Thu Oct 15 PyRIT locks down target creation and uploads Fri Oct 16 PyRIT readies attack audit views, garak patches its dependency surface Sat Oct 17 garak 0.17.1 ships hit@k, PyRIT widens SSRF loopback detection Sun Oct 18 PyRIT dedupes converter identifiers, masks combining-mark prompts Mon Oct 19 PyRIT fixes converter identity collisions that broke registry registration Tue Oct 20 PyRIT locks down credential-backed targets, deprecates label queries Wed Oct 21 Garak fixes BEAST beam search direction, Promptfoo locks down OpenAI provider config Thu Oct 22 PyRIT fences judge transcripts against role forgery Fri Oct 23 PyRIT calms Rust false positives and fixes attacker-view reporting Sat Oct 24 promptfoo 0.125.0 goes opt-in on Claude and Codex SDKs, PyRIT drops Gandalf Sun Oct 25 PyRIT closes a judge-spoofing hole, promptfoo accepts Langfuse v6 Mon Oct 26 Promptfoo moves off vulnerable MSAL, PyRIT adds a char-split attack and cloud token detection Tue Oct 27 PyRIT validates penalty parameters, corrects GCG gradient aggregation Wed Oct 28 PyRIT adds tool role-play attacks, garak moves probe intents into reusable stubs Thu Oct 29 Anthropic SDK tracing collides with promptfoo spans, and PyRIT adds a new attack technique Fri Oct 30 PyRIT tidies its warning UI and pins down converter state across prompt restores Sat Oct 31 promptfoo enforces body deadlines, PyRIT adds a Cyrillic homoglyph source

all ai-red-teaming reviews →

ai-red-teaming wire →