> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Choose a method

> NeMo-RL, verl, TRL, OpenRLHF, evalkit, and custom

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf methods                  # everything this deployment offers
sf methods nemo-rl/grpo     # one method's tunable parameters, with ranges
```

`sf methods` is authoritative for what the server you are connected to has actually published. The
tables below are a map, not a version pin.

The console's [Methods page](/en/console/methods) shows the same catalog with the parameters laid
out, which is easier to read before a first submission.

## NeMo-RL

Default framework for the full post-training pipeline. Current catalog release: `nemo-rl@0.7.0`.

| Method                 | What it is                         | When                                                                                                   |
| ---------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `nemo-rl/sft`          | Supervised instruction fine-tuning | First stage; output feeds DPO / GRPO                                                                   |
| `nemo-rl/dpo`          | Direct preference optimization     | Pairwise labels; cheaper than GRPO                                                                     |
| `nemo-rl/grpo`         | Group relative policy optimization | Math / reasoning RL; no critic                                                                         |
| `nemo-rl/grpo-lora`    | GRPO + LoRA                        | Tight GPU memory, or a larger base on one node. LoRA lr is 1–2 orders of magnitude higher than full FT |
| `nemo-rl/ppo`          | PPO with a critic and GAE          | Dense rewards or a distorted in-group baseline                                                         |
| `nemo-rl/rm`           | Bradley-Terry reward model         | RL reward source or LLM-judge calibration                                                              |
| `nemo-rl/distillation` | On-policy distillation             | Student samples, teacher full-distribution KL                                                          |
| `nemo-rl/maxrl`        | Maximum-likelihood RL              | Puts weight on low-pass-rate hard problems                                                             |
| `nemo-rl/opsd`         | Same-model self-distillation       | Teacher = same model + reference solution                                                              |

## verl

ByteDance verl synchronous trainer. Catalog release: `verl@0.9.0`. Entrypoint and parameter-path diffs across versions live in the version matrix ([Recipes](/en/concepts/recipes)).

Every RL method below runs the same official entry point (`verl.trainer.main_ppo`); what separates
them is the advantage estimator the recipe pins. That is deliberate: the estimator is not a
hyperparameter you can set in `config.yaml` — writing it there is rejected at submission, because a
config that silently disagrees with the method is worse than an error.

| Method                    | What it is                                                                                                                  |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `verl/grpo`               | Synchronous GRPO; image and deps pinned by the recipe                                                                       |
| `verl/dapo`               | GRPO plus the four DAPO techniques, on by default: clip-higher, dynamic sampling, token-level loss, overlong reward shaping |
| `verl/ppo`                | GAE PPO with a critic. Dense/process rewards, or when the in-group baseline is distorted                                    |
| `verl/rloo`               | Leave-one-out baseline: unbiased, and no division by the in-group std                                                       |
| `verl/reinforce-baseline` | REINFORCE++-baseline: group mean, then global-batch whitening. Same estimator as `openrlhf/reinforce-baseline`              |
| `verl/remax`              | Baseline from the model's own greedy answer; one sample per prompt, at the cost of an extra greedy pass                     |
| `verl/distillation`       | On-policy distillation (OPD). Needs a second resource pool for the teacher servers                                          |
| `verl/sft`                | Official SFT trainer; GPU processes via explicit torchrun argv                                                              |

Loss variants — GSPO, CISPO, GMPO, clip-cov / kl-cov, GPG — are not separate methods: they are the
`policy_loss_mode` parameter on any of the RL recipes above. Neither is rollout correction (TIS /
IcePop): every verl RL method carries the `rollout_is*` parameters, because the rollout engine and the
trainer are never quite the same policy.

## TRL

Hugging Face TRL + Accelerate. Catalog release: `trl@1.10.0`.

| Method     | What it is                                                                               |
| ---------- | ---------------------------------------------------------------------------------------- |
| `trl/sft`  | SFTTrainer                                                                               |
| `trl/dpo`  | DPOTrainer                                                                               |
| `trl/grpo` | GRPOTrainer online RL                                                                    |
| `trl/rm`   | RewardTrainer (SequenceClassification)                                                   |
| `trl/kto`  | Binary good/bad labels, no pairwise pairs                                                |
| `trl/rloo` | Leave-one-out in-group baseline; same family as GRPO, does not divide by in-group std    |
| `trl/opsd` | DistillationTrainer on-policy distillation; needs a same-vocab teacher (`teacher_model`) |

## OpenRLHF

DeepSpeed ZeRO + vLLM hybrid engine (colocate). Catalog release: `openrlhf@0.11.0`. Images are self-built (`deploy/docker/Dockerfile.openrlhf`); set `FORGE_IMAGE_OPENRLHF` or the runtime registry. Official upstream Dockerfiles do not publish a ready image with the package itself.

| Method                        | What it is                                                                                                                                      |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `openrlhf/sft`                | Official SFT trainer (packing, multi-turn loss, continued pre-training)                                                                         |
| `openrlhf/dpo`                | Official DPO (IPO / cDPO, NLL regularizer)                                                                                                      |
| `openrlhf/rm`                 | Bradley-Terry reward model; usable as `reward_pretrain` for OpenRLHF RL                                                                         |
| `openrlhf/grpo`               | Group-norm GRPO, no critic. `no_std_norm` is the Dr.GRPO setting                                                                                |
| `openrlhf/ppo`                | GAE PPO with critic; actor/critic/ref/reward colocated                                                                                          |
| `openrlhf/reinforce-baseline` | REINFORCE++-baseline (global batch norm + group mean). The same estimator as `verl/reinforce-baseline`, so a run moves between stacks unchanged |

## Evaluation and custom

| Method              | What it is                                                       | Docs                                          |
| ------------------- | ---------------------------------------------------------------- | --------------------------------------------- |
| `evalkit/benchmark` | lm-eval / evalscope; scores land on the dashboard                | [Benchmarks](/en/guides/benchmarks)           |
| `custom/custom`     | Runs `train.sh`; you bring the image and call `starforge.report` | [Custom training](/en/guides/custom-training) |

Per-framework walkthroughs: [NeMo-RL](/en/guides/nemo-rl), [verl](/en/guides/verl), [TRL](/en/guides/trl), [OpenRLHF](/en/guides/openrlhf).

## Picking a starting point

```mermaid theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
flowchart TD
    A["What data do you have?"] --> B["Instruction-response"]
    A --> C["Pairwise preferences"]
    A --> D["Binary good/bad"]
    A --> E["Verifiable tasks (math/code/tools)"]
    A --> F["A stronger teacher"]
    B --> SFT["nemo-rl/sft, trl/sft, or openrlhf/sft"]
    C --> DPO["dpo on nemo-rl / trl / openrlhf<br/>or rm then RL"]
    D --> KTO["trl/kto"]
    E --> GRPO["nemo-rl/grpo (default)<br/>tight memory: grpo-lora<br/>process rewards: ppo<br/>hard problems: maxrl<br/>verl: grpo, rloo, reinforce-baseline, remax or ppo<br/>OpenRLHF: grpo or reinforce-baseline"]
    F --> DIST["nemo-rl/distillation, verl/distillation or trl/opsd<br/>no external teacher: nemo-rl/opsd"]
```

<Tip>
  A common pipeline is SFT → DPO or GRPO → `evalkit/benchmark`. [Pipelines](/en/guides/pipelines).
</Tip>