$ the-wire · showcase
MILES commits checkpoints before eval; verl fixes B300 MFU
By RepoJournal · Filed · About Alignment & Post-training · Composed from the cited sources · methodology
open-instruct's MILES starters now leave recoverable state before a long run can be preempted, and verl stops reporting zero MFU on B300 and GB300.
MILES: use bounded recovery saves and preserved milestones in learning starters allenai/open-instruct
The MILES starting configs now give you bounded recovery saves and retained milestone checkpoints, with guidance to size the save interval from actual runtime. Tiny examples previously saved every 25 updates with no retained milestones, and the Small four- and eight-node examples disabled saving and automatic resume entirely, so those starters now leave recoverable model and optimizer state bef...
MILES: commit recovery checkpoints before evaluation and extend Small heartbeats allenai/open-instruct
Rollout-boundary snapshots move to the driver, after any native save due at the same boundary, so the evaluation export is no longer on the critical path to committing recoverable state. The failure that prompted it: one job reached optimizer update 100 and lost its allocation during the native recovery save. Evaluations due between optimizer updates within a rollout still capture the exact req...
[hardware] fix: add B300 and GB300 peak FLOPS (#8772) verl-project/verl
B300 and GB300 were absent from `_DEVICE_FLOPS`, so `get_device_flops()` returned `inf` and `perf/mfu/actor` and `perf/mfu/actor_infer` were always 0. Both now have peak FLOPS entries, which means the MFU numbers in your training logs on those parts become readable.
[vllm, rollout, perf] refactor: read sampled-token log-probs from flat lists verl-project/verl
When `rollout.calculate_log_probs=True` runs without a top-k head, verl now reads sampled-token log-probs from vLLM's flat lists instead of a `Logprob` dict per generated token; outputs are unchanged. A standalone AsyncLLM benchmark on one B300 put rollout wall time at 6.29 s versus 6.99 s before, about 10% faster.