$ the-wire · showcase
promptfoo locks down metric formulas and retires a model path
By RepoJournal · Filed · About AI Red Teaming · Composed from the cited sources · methodology
11 people shipped this
mldangelo-oai
@mldangelo-oai
1 cited
alandelong-oai
@alandelong-oai
1 cited
varunj-msft
@varunj-msft
1 cited
Kaap10
@Kaap10
1 cited
ASHMEET555
@ASHMEET555
1 cited
renovate
@renovate
romanlutz
@romanlutz
dependabot
@dependabot
Darshika140
@Darshika140
Punisheroot
@Punisheroot
Jeffrey Martin
@jmartin-tech
The day's real operational changes land in promptfoo, where derived metric formulas now accept only a narrow arithmetic grammar and OpenAI provider routing shifts with model and endpoint changes, while PyRIT gains a keyless auth mode and a validation catch for round-robin runs.
fix(evaluator)!: restrict derived metric formulas promptfoo/promptfoo
The PR is flagged breaking: derived metric formulas are now limited to numeric arithmetic over scores, so function calls, property access, assignments, and formulas over 1,024 characters are rejected. Validation names the offending metric and stops before any test case runs, which means a config that used to fail mid-run now fails at load; weighted averages, chained scores, and `__count` expres...
fix(providers): update OpenAI models and API compatibility promptfoo/promptfoo
GPT-6.1 Terra and trusted-access model recognition, embedding/moderation shorthand, and cache/service-tier pricing are refreshed, Terra Bedrock Standard prices are applied, and Mantle requests default to `us-east-1` unless you set a region. Native Discovery now requires an explicit `chat` or `responses` endpoint instead of guessing, so configs that relied on the implicit path need to name it.
FIX: Reject mismatched normalizer templates in RoundRobinTarget microsoft/PyRIT
`RoundRobinTarget` used to pass validation when its inner targets carried different normalizer templates via `normalizer_overrides`; configuration validation now rejects the mismatch, because `TargetConfiguration.as_identifier_params()` serialized only the normalizer class qualname and two distinct `schema_instructions_template` values looked identical. Runs that silently mixed templates will n...
FEAT: Add a "no key" auth mode for OpenAI-compatible targets microsoft/PyRIT
OpenAI-compatible targets can now be configured with a "no key" auth mode, so endpoints behind an internal gateway or a local server no longer require a dummy API key to scan. Anyone running PyRIT against self-hosted model servers gets the most out of this.
harness: keep detectors intact across IntentProbe runs NVIDIA/garak
One IntentProbe used to overwrite the run-wide `detectors` list with per-intent detector name strings, crashing every probe queued after it in the same `Harness.run`. The loop variable is now `detectors_for_intent`; a multi-probe run that used to die partway through now finishes with its detectors intact. The rest of the garak queue is maintenance: IntentProbe prompt deduplication, a shared pay...
Action items
