$ the-wire · showcase
Promptfoo patches Sharp and Codex SDK, garak fixes WebSocket and safe-token scanning
By RepoJournal · Filed · About AI Red Teaming · Composed from the cited sources · methodology
11 people shipped this
Two promptfoo dependency advisories close on upgrade and two garak scanner bugs that silently produced empty or masked scores get fixed, with a new OpenAI Decisions provider landing in the same day.
fix(deps): patch Sharp and Codex SDK security vulnerabilities promptfoo/promptfoo
Promptfoo now pins optional Sharp to 0.35.5 and requires Codex Security 0.2.x, whose dependency pins carry the patched smol-toml 1.9.0. The TOML fix applies even when promptfoo is installed inside another project, rather than depending on root-only npm overrides.
fix: reuse the WebSocket event loop across calls NVIDIA/garak
WebSocketGenerator created and closed a new event loop per prompt while reusing one socket, so from the second prompt on send/recv raised a loop-mismatch RuntimeError that a catch-all converted into Message(text=""), letting a scan look valid while scoring empty strings. Each generator now keeps a single event loop across calls.
feat(providers): add OpenAI Decisions support promptfoo/promptfoo
Operators can now target openai:decisions:<model>, covering predicate, choice, and ordered score questions over text and inline images plus text-output llm-rubric and classifier grading. Request and response validation, usage/cache accounting, refusal handling, quota metadata, and auth overrides ship with it.
[FIX]: Bind GUI Compose publication to localhost microsoft/PyRIT
The GUI Docker Compose mapping moves from all host interfaces to 127.0.0.1:8000:8000, matching the existing Jupyter default and leaving local access, the internal container listener, and authentication behavior unchanged. Existing installations must recreate the GUI container to pick up the new mapping.
fix(detectors): scope safe-token suppression to its own match NVIDIA/garak
ApiKey.detect held one detector-wide key_found flag, so a safe placeholder anywhere in the output reset it and masked every other credential match; an AWS sample key alongside a MongoDB safe example scored 0.0 where each alone scores correctly. Suppression is now scoped to the span that contains the safe token. Elsewhere, garak buffs stop dropping intent and reverse_translation_outputs on deriv...
Action items
- → Upgrade promptfoo to pick up Sharp 0.35.5 and Codex Security 0.2.x (smol-toml 1.9.0) promptfoo/promptfoo [plan]
- → Recreate the PyRIT GUI container to apply the loopback-only port binding microsoft/PyRIT [plan]
