$ cat alignment/week/2026-11-09.log
the week in review · Nov 9 – Nov 15, 2026
MILES adopts anchored Math-Verify grading and fixes reward bugs
By RepoJournal · composed from the cited sources · human-reviewed weekly · methodology
verl shipped a batch-invariant actor forward, an MXFP4 MoE recipe and a TileLang fused cross-entropy head over the same week.
Use anchored Math-Verify grading by default in MILES allenai/open-instruct
MILES math grading now routes through pinned Hugging Face Math-Verify 0.9.0 with anchored prediction extraction, replacing a scalar-oriented verifier that could reward an incomplete root list and reject equivalent structured answers. Scores you collected under the old verifier are not comparable, so re-check any math run you are using as a baseline.
Put ToolRL rewards on the unit scale and gate truncation at the verifier floor allenai/open-instruct
MILES' truncation gate assumed zero was the floor score, which fails once ToolRL's negative rewards enter the mix: training could credit a response for hitting the length limit rather than completing a wrong answer. ToolRL rewards now sit on the unit scale, so mixed-verifier reward totals shift if you tuned thresholds against the old scale.
Guard unsafe Math-Verify equations with cached legacy comparison fallback allenai/open-instruct
A reference like f(1)=1 was accepting f(1)=0 or 3+4=12, because Math-Verify 0.9.0's equation-solving fallback treats two empty solution lists as equivalent. The new guard detects unsafe equation references and falls back to a cached legacy comparison, which changes scores on equation-heavy math sets.
Strip selected terminal special tokens before MILES grading allenai/open-instruct
The reward adapter passed rollout text to graders with generation stop tokens still attached, so a correct answer ending in <|endoftext|> failed an instruction to end with a given phrase. Quotation, last-word and JSON checks were all affected; if your graders test final-string formatting, re-run them after this lands.
[fsdp, trainer, cfg, doc, tests] feat: batch-invariant actor forward and sampler log-prob formula verl-project/verl
A new opt-in FSDP engine option, batch_invariant (default off), installs vLLM's matmul, softmax, mean and RMSNorm kernel overrides plus the cuBLAS and TF32 settings, and computes log-probs with the sampler formula. Turn it on when training and rollout numerics must match; leave it off to keep the default throughput.
[perf, fsdp, megatron, veomni] feat: add TileLang fused linear cross entropy verl-project/verl
A TileLang fused linear cross-entropy output head now works across FSDP, Megatron and VeOmni, and supports multi-label and top-k log probabilities with repeated labels so distillation and score-centering losses consume precomputed values. It is the head-level memory saving rather than a new training mode, so existing recipes pick it up by config.
[megatron, doc] feat: MXFP4 MoE recipe and Qwen3-30B-A3B example verl-project/verl
The MXFP4 counterpart to the MXFP8 recipe targets the routed experts of MoE models, shipping with a Qwen3-30B-A3B example: GRPO on DAPO-Math-17k across one 4-GPU Blackwell node. Operators on Blackwell gain the low-precision option without changing the rest of the recipe.
[hardware, recipe] feat: unify dense Qwen3 (0.6B to 32B) TPU GRPO recipes into run_qwen3_torchtitan.sh verl-project/verl
The dense Qwen3 TPU GRPO scripts in examples/tpu/grpo/ collapse into a single run_qwen3_torchtitan.sh, selected by MODEL_SIZE=0.6B, 1.7B, 4B, 8B, 14B or 32B, replacing the per-size launchers you currently invoke. Alongside: verl also landed score centering on the Megatron actor, a Qwen3-30B-A3B TPU GRPO recipe, and a Minerva fix that stops bold Answer: labels from staining the extracted answer.
$ ls alignment/week/ # the briefings behind this review
Keep up with Alignment & Post-training in about 3 minutes: what actually shipped — the commits, pull requests, releases, and security advisories that matter.
fair warning: these emails are deeply technical. diffs, version numbers, CVEs, benchmark deltas. if that's not your idea of a good read, this isn't your newsletter.
Folds into your digest — weekly by default, monthly if you prefer. Unsubscribe in one click.