verl lands score centering for bypass-mode REINFORCE

By RepoJournal · composed from the cited sources · human-reviewed weekly · methodology

♥

Operators gain TPU SFT and GRPO recipes, a VeOmni training backend, and a breaking removal of SGLANG_USE_CPU_ENGINE.

160 commits 155 PRs merged 25 briefings covered

all alignment reviews →

alignment wire →

[rollout, vllm, algo, fsdp, trainer, doc] feat: score centering for bypass-mode REINFORCE verl-project/verl

by cr-gao

The new loss implements RFC #8009 on top of Marek & Ryabinin's paper, targeting the drift term E_q[R] * E_q[grad log p] that pulls the trainer toward the wrong policy when rollout and training distributions diverge. If you run bypass-mode REINFORCE, your loss now includes a centering term you did not have last month; expect changed gradient magnitudes, not identical runs.

[rollout] remove leftover todo ENV, remove SGLANG_USE_CPU_ENGINE environment (#8069) verl-project/verl

by kahlun

The unconditional SGLANG_USE_CPU_ENGINE=1 set and del in _load_sglang() is gone from the rollout replica loader, so any environment or launch script that exported it must be cleaned up. SGLang rollouts now take the server's own CPU-engine decision.

[doc, hardware, worker, trainer] feat: add SFT support for TPU using TorchTitan engine (#8098) verl-project/verl

by askhat-g

SFT on TPUs now runs through TorchTitan with no TPU-specific checks in verl core: the ReduceOp.AVG, all_gather_object and FSDP2 workarounds are pushed down into the engine layer. Read this as the path to start SFT on TPU hardware instead of keeping it on GPU for post-training.

[hardware] feat: enable TPU GRPO with TorchTitan and vLLM via verl-hardware-plugin verl-project/verl

by lixali

examples/tpu/grpo/run_qwen3_4b_torchtitan.sh runs Qwen3-4B-Base GRPO on GSM8K with TorchTitan training, vLLM rollout, and verl-hardware-plugin. TPU GRPO is now a shipped recipe rather than a branch you maintain yourself, though the launch surface moved out to the plugin.

[ckpt] feat: add veomni model merger verl-project/verl

by wuxibin89

The new merger turns a DCP distributed checkpoint into a single model export, replacing the scatter of scripts most teams keep for shipping a trained policy out of verl. Pair it with the TP and ETP support added to Bridge checkpoint conversion.

[vllm, hardware] fix: vLLM-Ascend 0.23 sched_yield on ARM hosts and FIA causal mask for genuine decode (#8144) verl-project/verl

by fangweii1

vLLM-Ascend 0.23 needed two NPU fixes, a sched_yield call on ARM hosts and an FIA causal mask that only applies to genuine decode, and both are forward-ported and adapted to main's refactored apply_npu_vllm_patches(). If you pin vLLM-Ascend 0.23 on ARM nodes, this determines whether generation is correct at all.

[megatron] fix: backward through chunked vocab-parallel entropy verl-project/verl

by aoshen02

With the Megatron engine, entropy_from_logits_with_chunking=True failed in backward with a RuntimeError about a variable modified by an inplace operation whenever entropy stayed in the graph. The fix unblocks chunked-vocab entropy training that had been silently unavailable on Megatron.

[megatron,worker] fix: offload actor parameters after disaggregated weight sync (#7631) verl-project/verl

by Mao Shizhuo

In disaggregated mode with a non-naive checkpoint backend, Megatron actor parameters stayed on GPU after update_weights(). They now move back to CPU, which is where the memory you budgeted for afterward actually goes.

$ ls alignment/month/ # the briefings behind this review

Fri Oct 2 Score centering lands for bypass-mode REINFORCE in verl Thu Oct 8 verl docs rename the Ascend 950 product line Fri Oct 9 verl opens the profiler and attention paths to non-CUDA backends Sat Oct 10 TPU SFT via TorchTitan, Megatron TP/ETP checkpoint conversion Sun Oct 11 veRL merges veomni checkpoints, automates Ascend doc translation Mon Oct 12 Chunked-entropy backward fix for Megatron, plus a Quay description for Ascend images Tue Oct 13 verl adds a TPU GRPO recipe, drops SGLANG_USE_CPU_ENGINE Wed Oct 14 DeepSeek MTP support fixed for vLLM and Megatron backends Thu Oct 15 Fused cross-entropy lands on Ascend NPUs and Megatron, checkpoint offload stops assuming CUDA Fri Oct 16 FSDP2 CPU offload was silently corrupting parameters, and MILES gets FlashAttention Sat Oct 17 verl rejects conflicting vLLM engine_kwargs, open-instruct adds MILES OPD behind a Megatron learner Sun Oct 18 verl auto-aligns score centering with verified sampler support Mon Oct 19 VeOmni becomes a first-class uv training backend in verl Tue Oct 20 verl restructures rollout quantization; TPU paths gain runbooks in both repos Wed Oct 21 TPU GRPO image fixed, Megatron optimizer offload leaves HDO state alone Thu Oct 22 verl standardizes low-precision rollout kernels on DeepGEMM Fri Oct 23 Open-Instruct moves to Miles v0.1.2; verl fixes teacher top-k packing under CP Sat Oct 24 MILES ships a 25-benchmark eval panel and simpler config presets Sun Oct 25 GSM8K scoring parses answers as exact decimals Mon Oct 26 verl fixes three environment-level breakages in Megatron and vLLM Tue Oct 27 MILES resumes stop re-running startup eval, DeepEP lands in verl's uv images Wed Oct 28 Qwen3.5 GRPO lands on TPU v7x, and verl fixes a rollout stall that made weight syncs wait Thu Oct 29 MILES evaluations move to native SGLang; compiled attention base drops install layer to 2.77 MB Fri Oct 30 XPU rollout images, a tensordict 0.11 data fix, and a gated-judge guard Sat Oct 31 MILES gains fixed source mixtures, and 500s stop scoring as zero

all alignment reviews →

alignment wire →