Choose a method

NeMo-RL, verl, TRL, OpenRLHF, evalkit, and custom

sf methods                  # everything this deployment offers
sf methods nemo-rl/grpo     # one method's tunable parameters, with ranges

sf methods is authoritative for what the server you are connected to has actually published. The tables below are a map, not a version pin.

The console’s Methods page shows the same catalog with the parameters laid out, which is easier to read before a first submission.

NeMo-RL

Default framework for the full post-training pipeline. Current catalog release: nemo-rl@0.7.0.

Method	What it is	When
nemo-rl/sft	Supervised instruction fine-tuning	First stage; output feeds DPO / GRPO
nemo-rl/dpo	Direct preference optimization	Pairwise labels; cheaper than GRPO
nemo-rl/grpo	Group relative policy optimization	Math / reasoning RL; no critic
nemo-rl/grpo-lora	GRPO + LoRA	Tight GPU memory, or a larger base on one node. LoRA lr is 1–2 orders of magnitude higher than full FT
nemo-rl/ppo	PPO with a critic and GAE	Dense rewards or a distorted in-group baseline
nemo-rl/rm	Bradley-Terry reward model	RL reward source or LLM-judge calibration
nemo-rl/distillation	On-policy distillation	Student samples, teacher full-distribution KL
nemo-rl/maxrl	Maximum-likelihood RL	Puts weight on low-pass-rate hard problems
nemo-rl/opsd	Same-model self-distillation	Teacher = same model + reference solution

verl

ByteDance verl synchronous trainer. Catalog release: verl@0.9.0. Entrypoint and parameter-path diffs across versions live in the version matrix (Recipes).

Every RL method below runs the same official entry point (verl.trainer.main_ppo); what separates them is the advantage estimator the recipe pins. That is deliberate: the estimator is not a hyperparameter you can set in config.yaml — writing it there is rejected at submission, because a config that silently disagrees with the method is worse than an error.

Method	What it is
verl/grpo	Synchronous GRPO; image and deps pinned by the recipe
verl/dapo	GRPO plus the four DAPO techniques, on by default: clip-higher, dynamic sampling, token-level loss, overlong reward shaping
verl/ppo	GAE PPO with a critic. Dense/process rewards, or when the in-group baseline is distorted
verl/rloo	Leave-one-out baseline: unbiased, and no division by the in-group std
verl/reinforce-baseline	REINFORCE++-baseline: group mean, then global-batch whitening. Same estimator as openrlhf/reinforce-baseline
verl/remax	Baseline from the model’s own greedy answer; one sample per prompt, at the cost of an extra greedy pass
verl/distillation	On-policy distillation (OPD). Needs a second resource pool for the teacher servers
verl/sft	Official SFT trainer; GPU processes via explicit torchrun argv

Loss variants — GSPO, CISPO, GMPO, clip-cov / kl-cov, GPG — are not separate methods: they are the policy_loss_mode parameter on any of the RL recipes above. Neither is rollout correction (TIS / IcePop): every verl RL method carries the rollout_is* parameters, because the rollout engine and the trainer are never quite the same policy.

TRL

Hugging Face TRL + Accelerate. Catalog release: trl@1.10.0.

Method	What it is
trl/sft	SFTTrainer
trl/dpo	DPOTrainer
trl/grpo	GRPOTrainer online RL
trl/rm	RewardTrainer (SequenceClassification)
trl/kto	Binary good/bad labels, no pairwise pairs
trl/rloo	Leave-one-out in-group baseline; same family as GRPO, does not divide by in-group std
trl/opsd	DistillationTrainer on-policy distillation; needs a same-vocab teacher (teacher_model)

OpenRLHF

DeepSpeed ZeRO + vLLM hybrid engine (colocate). Catalog release: openrlhf@0.11.0. Images are self-built (deploy/docker/Dockerfile.openrlhf); set FORGE_IMAGE_OPENRLHF or the runtime registry. Official upstream Dockerfiles do not publish a ready image with the package itself.

Method	What it is
openrlhf/sft	Official SFT trainer (packing, multi-turn loss, continued pre-training)
openrlhf/dpo	Official DPO (IPO / cDPO, NLL regularizer)
openrlhf/rm	Bradley-Terry reward model; usable as reward_pretrain for OpenRLHF RL
openrlhf/grpo	Group-norm GRPO, no critic. no_std_norm is the Dr.GRPO setting
openrlhf/ppo	GAE PPO with critic; actor/critic/ref/reward colocated
openrlhf/reinforce-baseline	REINFORCE++-baseline (global batch norm + group mean). The same estimator as verl/reinforce-baseline, so a run moves between stacks unchanged

Evaluation and custom

Method	What it is	Docs
evalkit/benchmark	lm-eval / evalscope; scores land on the dashboard	Benchmarks
custom/custom	Runs train.sh; you bring the image and call starforge.report	Custom training

Per-framework walkthroughs: NeMo-RL, verl, TRL, OpenRLHF.

Picking a starting point

A common pipeline is SFT → DPO or GRPO → evalkit/benchmark. Pipelines.