Train with OpenRLHF

Self-built image, DeepSpeed + vLLM colocate, GRPO vs REINFORCE++-baseline

sf new my-ppo --method openrlhf/ppo
sf submit my-ppo --profile h200:8

OpenRLHF 0.11.0 as a first-class framework. Methods: openrlhf/sft, dpo, rm, grpo, ppo, reinforce-baseline. Configuration is CLI flags rather than Hydra.

Upstream’s Dockerfiles install the dependency stack but not the openrlhf package, and publish no ready training image. An administrator must set FORGE_IMAGE_OPENRLHF, or register the openrlhf-0.11.0 runtime id, before submission works.

The Ray version is pinned by that image — 0.11.0 upstream uses ray==2.55.0. Do not copy the TRL or verl pin across; a mismatch presents as “the cluster came up but workers never registered”.

Create and submit

sf new my-orlh --method openrlhf/grpo
sf submit my-orlh --profile h200:8 \
  --model Qwen/Qwen3.5-9B \
  --train-data data/prompts.jsonl

Bindings (recipe):

Field	Flag
model	actor.model_name_or_path
train data	data.prompt_dataset
validation	eval.dataset

GRPO sets algo.advantage.estimator=group_norm, train.colocate_all=true, vLLM/DeepSpeed sleep flags. All roles share GPUs (colocate hybrid engine). Topology expands to each role’s num_nodes / num_gpus_per_node.

no_std_norm in the recipe is the Dr.GRPO setting.

openrlhf/reinforce-baseline is REINFORCE++-baseline (global batch norm + group mean). The other frameworks do not offer that estimator.

Image smoke

Operators: ./deploy/docker/build-runtimes.sh openrlhf and scripts/image_smoke.py. Users only need a deployment that already published the image.