$ the-wire · showcase
verl drops DataProto auto padding, adds single-rank shard dispatch
By RepoJournal · Filed · About Alignment & Post-training · Composed from the cited sources · methodology
verl shipped two breaking changes to how batches and shards move between ranks, plus a ROCm dataloader workaround that stops PPO steps from hanging.
[BREAKING][single_controller, data, doc] refactor: remove DataProto auto padding verl-project/verl
verl removed DataProto auto padding: `DP_COMPUTE_PROTO` now splits via `_split_args_kwargs_data_proto` and requires `len(batch) % world_size == 0`. The PR's own reasoning is that the old undo step is valid only when every worker returns the same rows in the same order at the same length, so any job relying on automatic divisibility padding must now pad explicitly with `pad_dataproto_to_divisor`.
[BREAKING][single_controller, worker, trainer] feat: dispatch shards only to the model-parallel source rank (#8708) verl-project/verl
N-D data-parallel dispatch gains an optional path that sends each shard to exactly one model-parallel source rank and leaves redistribution to a worker-side preprocess hook, instead of copying every shard to every rank in the model-parallel group. It is breaking and the worker does not create a model-parallel process group, so anything that assumed the old per-rank copy needs the hook.
[ci, hardware] fix: set KMP_INIT_AT_FORK=TRUE on ROCm to stop dataloader fork deadlock (#8713) verl-project/verl
The ROCm e2e workflow and `docker/rocm/Dockerfile.rocm` now set `KMP_INIT_AT_FORK=TRUE`, which stops PPO training steps from hanging forever when dataloader workers start. Setting it in the Dockerfile means ROCm images you build from it carry the fix; the cause was `scikit-learn` and `joblib` calling `os.environ.setdefault("KMP_INIT_AT_FORK", "FALSE")` when `transformers` imports them.
Action items