$ the-wire · showcase
Promptfoo drops stale caches, PyRIT ships an attack analytics SDK
By RepoJournal · Filed · About AI Red Teaming · Composed from the cited sources · methodology
Promptfoo removed the config and MCP evaluation caches that hid fresh edits and new runs, while PyRIT added an attack analytics SDK and an Anthropic model-written evals loader operators can pull through SeedDatasetProvider.
fix: remove stale MCP evaluation summary cache promptfoo/promptfoo
MCP list_evaluations now reads summary rows on every request, so a long-lived server surfaces evaluations that were just created, updated, or deleted. The trade is explicit in the PR: repeated listings cost extra database reads, and the cacheStats fields come back zeroed.
fix(config): remove the default configuration cache promptfoo/promptfoo
Default-config discovery no longer retains parsed config objects or cached missing-file results, so declarative edits, new files, deletions, and parse failures all show up on the next load. CommonJS .js configs still work: successful VM fallback exports are cached for discovery only, and watch reruns invalidate them.
FEAT: Attack analytics SDK microsoft/PyRIT
PyRIT merged an attack analytics SDK, giving operators programmatic analysis over attack results instead of ad hoc post-processing.
FEAT: Add Anthropic model-written evals dataset microsoft/PyRIT
The Anthropic model-written evals land as a remote seed dataset loader in PyRIT, fetched through SeedDatasetProvider from the full anthropics/evals release rather than Hugging Face, which publishes only a subset. Supported collections are persona, sycophancy, advanced-ai-risk (human-written and LM-written), and winogenerated examples.
refactor(webui): consolidate media URL resolution promptfoo/promptfoo
The eval table now resolves storage and blob audio URLs through the shared media helpers, dropping the duplicate mediaStorage module, its loading state, and its effect; the audio player keys off the resolved URL so a source change reloads the right clip. The unreachable performance-history chart and its dead flag are also gone, with the three active charts unchanged.
Action items
