> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Train with verl

> Required --model / --train-data, Hydra bindings, methods and estimators, optional experiment main.py

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf new my-verl --method verl/grpo --framework-version 0.9.0
sf submit my-verl --profile h200:8 \
  --model Qwen/Qwen2.5-7B --train-data data/train.parquet
```

ByteDance verl as a first-class runtime, catalog version **0.9.0**. Methods: `verl/grpo`, `dapo`,
`ppo`, `rloo`, `reinforce-baseline`, `remax`, `distillation`, `sft`.

Unlike NeMo-RL, verl needs `--model` and `--train-data` at submission. Observability is `platform`.

<Info>
  verl images are deployment artifacts (`FORGE_IMAGE_VERL` or `FORGE_RUNTIME_REGISTRY_FILE`). An
  administrator publishes one before anybody can submit; without it, submission fails naming the
  missing runtime rather than the method.
</Info>

## Create with an explicit version

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf new my-verl --method verl/grpo --framework-version 0.9.0
```

0.9 uses the unified `verl.trainer.main_ppo` (sync V1); upstream removed 0.8's `main_ppo_sync`, so the catalog no longer carries the older runtime.

If `experiments/my-verl/main.py` exists, that file is the entrypoint (verl-recipe style: subclass `TaskRunner` / `RayPPOTrainer`). Reward-only changes can stay on `custom_reward_function.path` in config.

## Model and data are required at submit

Unlike NeMo-RL, verl bindings expect explicit paths. The recipe maps:

| CLI / JobSpec       | Hydra key                      |
| ------------------- | ------------------------------ |
| `--model`           | `actor_rollout_ref.model.path` |
| `--train-data`      | `data.train_files`             |
| `--validation-data` | `data.val_files`               |

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf submit my-verl --profile h200:8 \
  --model Qwen/Qwen3.5-9B \
  --train-data data/train.parquet \
  --validation-data data/val.parquet
```

Platform dataset:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf submit my-verl --profile h200:8 \
  --model Qwen/Qwen3.5-9B \
  --train-dataset alice/gsm8k-zh@v2 \
  --train-data train.parquet
```

Prefer `data.train.dataset` in config; CLI is an override. [Datasets](/en/guides/datasets).

`sf validate` fails early if the required model/data declarations are missing.

## Picking the RL method

Every RL method here runs the same `verl.trainer.main_ppo`; the recipe pins one thing,
`algorithm.adv_estimator`, and that is what the method *is*.

| Method                    | Estimator                      | Baseline comes from                 | Notes                                                            |
| ------------------------- | ------------------------------ | ----------------------------------- | ---------------------------------------------------------------- |
| `verl/grpo`               | `grpo`                         | in-group mean ÷ in-group std        | needs `rollout.n ≥ 2`                                            |
| `verl/rloo`               | `rloo`                         | mean of the *other* samples         | unbiased; no std division. `rollout.n ≥ 2`                       |
| `verl/reinforce-baseline` | `reinforce_plus_plus_baseline` | group mean + global-batch whitening | same estimator as `openrlhf/reinforce-baseline`. `rollout.n ≥ 2` |
| `verl/remax`              | `remax`                        | the model's own greedy answer       | `rollout.n = 1`, plus one extra greedy pass per step             |
| `verl/ppo`                | `gae`                          | a trained critic                    | second full-size model; watch `train/value_loss`                 |
| `verl/dapo`               | `grpo`                         | in-group mean, on filtered groups   | the four DAPO techniques, on by default                          |

Writing `algorithm.adv_estimator` into `config.yaml` is rejected at submission: it is the method's
identity, not a knob. To change estimator, change method.

Loss variants (GSPO, CISPO, GMPO, clip-cov / kl-cov, GPG) *are* knobs — set `policy_loss_mode` on
any of these recipes.

## DAPO

`verl/dapo` is configuration, not a vendored trainer: 0.9 carries all four published techniques in the
mainline `main_ppo`, so the recipe pins `algorithm.filter_groups.enable` and the `dapo` reward manager
and ships the paper's values as parameter defaults (clip 0.2/0.28, `token-mean`, 4096-token overlong
buffer at factor 1.0, 16 samples per prompt, no KL on either path).

Two things to know before submitting one:

* Dynamic sampling judges a group on `filter_metric` **during rollout**, so the reward has to be
  available then: a rule-based reward function, or a reward model in its own pool. A colocated reward
  model scores after sampling and upstream asserts on that.
* Step time is variable by construction — the trainer keeps regenerating until enough groups qualify.
  For the "DAPO w/o dynamic sampling" configuration, use `verl/grpo` with `clip_ratio_high=0.28` and
  `loss_agg_mode=token-mean`.

## Rollout correction (TIS / IcePop)

The rollout engine (vLLM, bf16) and the trainer (FSDP, fp32) are not the same policy even with identical
weights, and stale or refilled rollouts widen the gap. Every verl RL recipe exposes the correction:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf submit my-verl --profile h200:8 \
  --set rollout_calculate_log_probs=true \
  --set rollout_is=sequence \
  --set rollout_is_threshold=2.0     # or 0.5_5.0 for IcePop (zeroes, not clamps, outside the band)
```

`rollout_calculate_log_probs` is not optional: without the rollout engine's own log-probs there is no
behaviour policy to compare against, and the platform rejects the combination at compile time rather
than letting it silently do nothing. Leaving `rollout_is` unset reports the mismatch metrics without
correcting anything, which is a reasonable first step on a new setup.

## On-policy distillation needs two pools

`verl/distillation` runs the teacher inference servers in their own Ray placement group, allocated on
top of the trainer's. So the job declares two resource pools and maps the `teacher` role to the
second one:

```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# spec.resources
pools:
  - {name: train, series: h100, nodes: 1, gpus_per_node: 8}
  - {name: teach, series: h100, nodes: 1, gpus_per_node: 8}
roles: {actor: train, rollout: train, teacher: teach}
```

The platform fills `trainer.nnodes` from the student pool and `distillation.nnodes` from the teacher
pool. A single-pool submission is rejected at compile time rather than hanging on placement.
Multi-pool jobs run on the kuberay and slurm backends only.

## Topology

Adapter writes `trainer.nnodes` / `trainer.n_gpus_per_node` from `FORGE_CLUSTER_*` (from the primary
pool when the job has a teacher pool).

## Export / eval

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf export my-verl --checkpoint <path> --checkpoint-format verl-fsdp
# also: verl-megatron
```

SFT eval can attach samples back to the training run (`sf eval` with `--run-id`). See [Pipelines](/en/guides/pipelines).