$ the-wire · showcase
inspect_evals 0.28.0 closes an SSRF hole in BFCL web search
By RepoJournal · Filed · About Safety Evals · Composed from the cited sources · methodology
inspect_evals v0.28.0 ships the BFCL fetch restriction alongside a MATH scoring fix that operators running the v4-A math eval need to know about.
v0.28.0 UKGovernmentBEIS/inspect_evals
The release behind the BFCL hardening and the MATH v4-A changes is out; the MATH task now surfaces an unusable antlr4 runtime at task construction instead of scoring it as a wrong answer, errors samples whose target sympy cannot parse, and only scores the model wrong on LaTeX, sympify or type errors, while the answer normaliser keeps \infty and \left intact instead of mangling them into \iny an...
fix(bfcl): restrict web fetches to public destinations UKGovernmentBEIS/inspect_evals
The two opt-in V4 web-search categories had evaluators fetching model-chosen URLs in-process, so a model could steer readable server-side requests at loopback, private networks, and redirect targets; page fetches are now limited to HTTP(S) destinations whose resolved addresses are all public, dialed by validated IP with the original host header and TLS checks preserved, and every redirect is re...
chore(deps): bump the python-packages group with 4 updates UKGovernmentBEIS/inspect_ai
Routine group bump in inspect_ai: platformdirs 4.14.0 to 4.14.2, pytest, pytest-textual-snapshot and types-boto3. Nothing operator-facing.
docs: raise provider SDK floors instead of shimming both versions (#6379) UKGovernmentBEIS/inspect_ai
inspect_ai's contributor docs now codify the practice that already governed the last several provider SDK breaks: raise the minimum supported version instead of carrying compatibility shims, updating the dev extra floor in pyproject.toml, the provider's MIN_VERSION and uv.lock together. Worth reading if you pin inspect_ai against an older provider SDK.
Action items
- → Upgrade inspect_evals to 0.28.0 if you run BFCL's V4 web-search categories UKGovernmentBEIS/inspect_evals [immediate]
- → Re-baseline MATH v4-A results after 0.28.0, since samples that previously scored as wrong answers now error UKGovernmentBEIS/inspect_evals [plan]
