Train with verl

Required —model / —train-data, Hydra bindings, methods and estimators, optional experiment main.py

sf new my-verl --method verl/grpo --framework-version 0.9.0
sf submit my-verl --profile h200:8 \
  --model Qwen/Qwen2.5-7B --train-data data/train.parquet

ByteDance verl as a first-class runtime, catalog version 0.9.0. Methods: verl/grpo, dapo, ppo, rloo, reinforce-baseline, remax, distillation, sft.

Unlike NeMo-RL, verl needs --model and --train-data at submission. Observability is platform.

verl images are deployment artifacts (FORGE_IMAGE_VERL or FORGE_RUNTIME_REGISTRY_FILE). An administrator publishes one before anybody can submit; without it, submission fails naming the missing runtime rather than the method.

Create with an explicit version

sf new my-verl --method verl/grpo --framework-version 0.9.0

0.9 uses the unified verl.trainer.main_ppo (sync V1); upstream removed 0.8’s main_ppo_sync, so the catalog no longer carries the older runtime.

If experiments/my-verl/main.py exists, that file is the entrypoint (verl-recipe style: subclass TaskRunner / RayPPOTrainer). Reward-only changes can stay on custom_reward_function.path in config.

Model and data are required at submit

Unlike NeMo-RL, verl bindings expect explicit paths. The recipe maps:

CLI / JobSpec	Hydra key
--model	actor_rollout_ref.model.path
--train-data	data.train_files
--validation-data	data.val_files

sf submit my-verl --profile h200:8 \
  --model Qwen/Qwen3.5-9B \
  --train-data data/train.parquet \
  --validation-data data/val.parquet

Platform dataset:

sf submit my-verl --profile h200:8 \
  --model Qwen/Qwen3.5-9B \
  --train-dataset alice/gsm8k-zh@v2 \
  --train-data train.parquet

Prefer data.train.dataset in config; CLI is an override. Datasets.

sf validate fails early if the required model/data declarations are missing.

Picking the RL method

Every RL method here runs the same verl.trainer.main_ppo; the recipe pins one thing, algorithm.adv_estimator, and that is what the method is.

Method	Estimator	Baseline comes from	Notes
verl/grpo	grpo	in-group mean ÷ in-group std	needs rollout.n ≥ 2
verl/rloo	rloo	mean of the other samples	unbiased; no std division. rollout.n ≥ 2
verl/reinforce-baseline	reinforce_plus_plus_baseline	group mean + global-batch whitening	same estimator as openrlhf/reinforce-baseline. rollout.n ≥ 2
verl/remax	remax	the model’s own greedy answer	rollout.n = 1, plus one extra greedy pass per step
verl/ppo	gae	a trained critic	second full-size model; watch train/value_loss
verl/dapo	grpo	in-group mean, on filtered groups	the four DAPO techniques, on by default

Writing algorithm.adv_estimator into config.yaml is rejected at submission: it is the method’s identity, not a knob. To change estimator, change method.

Loss variants (GSPO, CISPO, GMPO, clip-cov / kl-cov, GPG) are knobs — set policy_loss_mode on any of these recipes.

DAPO

verl/dapo is configuration, not a vendored trainer: 0.9 carries all four published techniques in the mainline main_ppo, so the recipe pins algorithm.filter_groups.enable and the dapo reward manager and ships the paper’s values as parameter defaults (clip 0.2/0.28, token-mean, 4096-token overlong buffer at factor 1.0, 16 samples per prompt, no KL on either path).

Two things to know before submitting one:

• Dynamic sampling judges a group on filter_metric during rollout, so the reward has to be available then: a rule-based reward function, or a reward model in its own pool. A colocated reward model scores after sampling and upstream asserts on that.
• Step time is variable by construction — the trainer keeps regenerating until enough groups qualify. For the “DAPO w/o dynamic sampling” configuration, use verl/grpo with clip_ratio_high=0.28 and loss_agg_mode=token-mean.

Rollout correction (TIS / IcePop)

The rollout engine (vLLM, bf16) and the trainer (FSDP, fp32) are not the same policy even with identical weights, and stale or refilled rollouts widen the gap. Every verl RL recipe exposes the correction:

sf submit my-verl --profile h200:8 \
  --set rollout_calculate_log_probs=true \
  --set rollout_is=sequence \
  --set rollout_is_threshold=2.0     # or 0.5_5.0 for IcePop (zeroes, not clamps, outside the band)

rollout_calculate_log_probs is not optional: without the rollout engine’s own log-probs there is no behaviour policy to compare against, and the platform rejects the combination at compile time rather than letting it silently do nothing. Leaving rollout_is unset reports the mismatch metrics without correcting anything, which is a reasonable first step on a new setup.

On-policy distillation needs two pools

verl/distillation runs the teacher inference servers in their own Ray placement group, allocated on top of the trainer’s. So the job declares two resource pools and maps the teacher role to the second one:

# spec.resources
pools:
  - {name: train, series: h100, nodes: 1, gpus_per_node: 8}
  - {name: teach, series: h100, nodes: 1, gpus_per_node: 8}
roles: {actor: train, rollout: train, teacher: teach}

The platform fills trainer.nnodes from the student pool and distillation.nnodes from the teacher pool. A single-pool submission is rejected at compile time rather than hanging on placement. Multi-pool jobs run on the kuberay and slurm backends only.

Topology

Adapter writes trainer.nnodes / trainer.n_gpus_per_node from FORGE_CLUSTER_* (from the primary pool when the job has a teacher pool).

Export / eval

sf export my-verl --checkpoint <path> --checkpoint-format verl-fsdp
# also: verl-megatron

SFT eval can attach samples back to the training run (sf eval with --run-id). See Pipelines.