$ the-wire · showcase
MILES stops discarding whole prompt groups over one bad grade
By RepoJournal · Filed · About Alignment & Post-training · Composed from the cited sources · methodology
2 people shipped this
allenai/open-instruct reworked how MILES handles ungraded responses, made R1 format rewards accept final-answer-only answers, and surfaced symbolic math timeouts that were previously invisible in diagnostics and W&B.
Exclude ungraded responses from MILES prompt groups instead of omitting the group allenai/open-instruct
A response whose verifier or judge failed for infrastructure reasons used to cost you every graded sibling in its prompt group; the adapter now keeps it with a placeholder zero and metadata.reward_omission, and only the failed response is excluded from group reward baselines. core.exclude_ungraded defaults to true, and setting it to false restores whole-group omission.
Fix R1 format rewards with optional final-answer verifier dispatch allenai/open-instruct
R1 format rewards previously rejected valid answers in final-answer-only scoring because R1FormatVerifier checked for a closing-think marker that the pipeline had already stripped. The adapter now calls async_call_final_answer when it exists, so answer-tag checks work without the removed marker.
Expose symbolic math timeouts in MILES diagnostics and W&B metrics allenai/open-instruct
Symbolic math comparisons hit an internal five-second timeout that returned False and was logged only at DEBUG, so affected results read as completed verifications and the existing timeout metric counted only the 45-second worker deadline. The timeouts now show up in MILES diagnostics, warnings and W&B, giving you a way to measure how often they fire.
Fix legacy judge boolean verdicts collapsing to zero allenai/open-instruct
The legacy LMJudgeVerifier asks judges for a True/False SCORE string, but the shared extractor called float("True"), failed, and fell back to zero, so a compliant correct verdict earned no credit. This fixes the extraction so boolean verdicts score as intended.
Exclude ungraded responses from MILES prompt groups instead of omitting the group (#2209) allenai/open-instruct
The commit behind the ungraded-response change ships it with the same defaults: core.exclude_ungraded on, placeholder zero plus reward_omission metadata on failed grades.
