$ the-wire · showcase
Inspect logs provider request ids, dataset shuffle takes an int seed
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
Inspect now records the provider request and response ids behind every model call, and two config-level fixes land: seeded shuffles that actually seed, and run configs that replay without choking on token_limit_type.
Log provider request and response ids for every model call UKGovernmentBEIS/inspect_ai
Every generate request now records the provider's request ids in provider response headers (x-request-id, request-id, x-amzn-requestid, mistral-correlation-id) alongside the header name and HTTP status, including calls the provider SDK retried, and the provider's response id (resp_..., chatcmpl-..., msg_...) is captured too. When a safety or abuse report or a billing question points at one spec...
Accept an integer seed for `shuffle` in dataset loaders UKGovernmentBEIS/inspect_ai
shuffle on csv_dataset(), json_dataset(), file_dataset() and hf_dataset() accepted only a bool, so shuffle=42 was treated as truthy and the loader reseeded with seed=None, giving a different order every run, while shuffle=0 silently disabled shuffling. It now takes bool | int and any int, including 0, is the seed, matching how shuffle_choices and sample_shuffle already behave. Any eval whose re...
Keep token_limit_type when replaying a run config UKGovernmentBEIS/inspect_ai
Replaying an exported run config with --run-config failed at startup with Unknown GenerateConfig field(s): token_limit_type for any run that used an output-only token limit, and handwritten configs with the same keys failed identically because RunConfigInput.to_params() flattened eval_config into eval() arguments that accept token_limit but not token_limit_type. The type is now preserved, so ex...
fix(google): prefer explicit API keys over ambient ADC UKGovernmentBEIS/inspect_ai
On the Gemini Developer API, GOOGLE_USE_ADC=true in the environment routed every Google model through Application Default Credentials and discarded an explicitly passed API key in favor of an OAuth credential. When no auth mode is selected on the model, an explicit API key now wins over the ambient GOOGLE_USE_ADC default.
Chore(Docs Update + Agent Resources) UKGovernmentBEIS/inspect_evals
The threecb registry entry advertised 13 samples when the loader actually yields 35, one per elicitation across 11 challenge TOMLs (most declaring 4, three declaring 1); the docs and agent-resource cleanup also renames claude.md to agents.md and drops the repo context weekly trigger.
