$ the-wire · showcase
PyRIT adds an obfuscated-secret scorer and stops converters leaking system prompts into memory
By RepoJournal · Filed · About AI Red Teaming · Composed from the cited sources · methodology
The day's most useful operator-facing change is a new scorer that grades how much of a known secret a response gives away, alongside three fixes that make stored datasets, saved conversations and reports reflect what actually ran.
FEAT: Add SecretDisclosureScorer for partial and obfuscated disclosure of a known secret microsoft/PyRIT
SecretDisclosureScorer reports, on a float scale, how much of a known secret a response reveals, measured in bits. It searches the response as written, with separators stripped, with digit words and NATO letters mapped to characters, reversed, and with base64 and hex tokens decoded, and each of those views can be switched off.
FIX: send converter instructions in the user turn so converters write nothing to memory microsoft/PyRIT
LLM-backed converters used to write only their system prompt into memory and then call the target, so every conversion left a stray system-only conversation behind. They now send those instructions ahead of the prompt in the user turn and write nothing to memory, which keeps your conversation store clean after converter-heavy runs.
FIX: reload a sending conversation after another one has loaded microsoft/PyRIT
If you navigate away mid-send and return to that conversation by URL, the transcript of the conversation you left now gets reloaded instead of staying on screen with no loading state. The in-flight skip applies only while the hook still holds that conversation.
[FIX] Assign local YAML filename fallback to seeds so named queries find them microsoft/PyRIT
Local YAML datasets with no dataset_name used to show up in the provider list while get_seeds_async(dataset_name="<stem>") returned 0 rows and get_seed_dataset_names_async() omitted the name, because the filename fallback was applied at the provider layer but not to the seeds. The seeds now carry that fallback, so named queries find the dataset.
FIX Export the converted data type in JSON conversations and label HTML report pieces by it microsoft/PyRIT
JSON exports now carry the converted value's own type alongside the original, so HTML reports no longer switch on the original data_type for both values; the side-by-side "Original:" and "Converted:" display labels each correctly.