> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Train with OpenRLHF

> Self-built image, DeepSpeed + vLLM colocate, GRPO vs REINFORCE++-baseline

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf new my-ppo --method openrlhf/ppo
sf submit my-ppo --profile h200:8
```

OpenRLHF **0.11.0** as a first-class framework. Methods: `openrlhf/sft`, `dpo`, `rm`, `grpo`, `ppo`,
`reinforce-baseline`. Configuration is CLI flags rather than Hydra.

<Warning>
  Upstream's Dockerfiles install the dependency stack but **not** the `openrlhf` package, and publish
  no ready training image. An administrator must set `FORGE_IMAGE_OPENRLHF`, or register the
  `openrlhf-0.11.0` runtime id, before submission works.

  The Ray version is pinned by that image — 0.11.0 upstream uses `ray==2.55.0`. Do not copy the TRL
  or verl pin across; a mismatch presents as "the cluster came up but workers never registered".
</Warning>

## Create and submit

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf new my-orlh --method openrlhf/grpo
sf submit my-orlh --profile h200:8 \
  --model Qwen/Qwen3.5-9B \
  --train-data data/prompts.jsonl
```

Bindings (recipe):

| Field      | Flag                       |
| ---------- | -------------------------- |
| model      | `actor.model_name_or_path` |
| train data | `data.prompt_dataset`      |
| validation | `eval.dataset`             |

GRPO sets `algo.advantage.estimator=group_norm`, `train.colocate_all=true`, vLLM/DeepSpeed sleep flags. All roles share GPUs (colocate hybrid engine). Topology expands to each role's `num_nodes` / `num_gpus_per_node`.

`no_std_norm` in the recipe is the Dr.GRPO setting.

`openrlhf/reinforce-baseline` is REINFORCE++-baseline (global batch norm + group mean). The other frameworks do not offer that estimator.

## Image smoke

Operators: `./deploy/docker/build-runtimes.sh openrlhf` and `scripts/image_smoke.py`. Users only need a deployment that already published the image.