Train with NeMo-RL

sf new, config inheritance, submit, topology, and NeMo-RL-only flags

sf new my-grpo --method nemo-rl/grpo
sf validate my-grpo
sf submit my-grpo --profile h200:8

NeMo-RL is the default catalog for a full post-training stack, currently 0.7.0. Methods: nemo-rl/sft, dpo, grpo, grpo-lora, ppo, rm, distillation, maxrl, opsd.

Observability is platform, which means two things you do not have to do: pass --observability-url, and call starforge.report yourself. The adapter and the framework bridge handle both.

Create and tune

sf new my-grpo --method nemo-rl/grpo
# edit experiments/my-grpo/config.yaml  (defaults + diffs only)
sf validate my-grpo
sf methods nemo-rl/grpo

Model and datasets usually live in configs/base/ + configs/models/ fragments. Submit often needs no --model / --train-data.

Config details: config.yaml.

Submit

git add -A && git commit -m "grpo lr"
sf submit my-grpo --profile h200:8

--profile sets billed topology. The adapter writes cluster.num_nodes and cluster.gpus_per_node from FORGE_CLUSTER_* and points checkpointing.checkpoint_dir / logger.log_dir at FORGE_OUT_DIR. Do not fight those keys in the experiment file; they are overwritten.

Heterogeneous pools (train vs rollout on different GPU series) only on kuberay / slurm:

sf submit my-grpo --profile train=h200:8 --profile rollout=h100:2

When a separate rollout pool exists, the adapter sets policy.generation.colocated.enabled=false and fills colocated resource counts from FORGE_POOL_TOPOLOGY. You do not maintain a separate noncolocated overlay YAML for that.

local and agent do not run multi-pool jobs.

Entrypoint

Recipe default is the versioned NeMo-RL module/file. If the recipe declares experiment_override and that file exists in the experiment directory, that file wins (custom env / dataset experiments that ship run.py). spec.source.entrypoint in the JobSpec is rejected; you cannot override the recipe from submit JSON.

The container needs NEMO_RL_DIR (server FORGE_NEMO_RL_DIR, typically /opt/nemo-rl in the NGC image). Missing it fails compile: the server cannot proxy NeMo-RL jobs.

After training

sf export my-grpo --checkpoint <path> --checkpoint-format nemo-dcp
# formats: nemo-dcp | nemo-megatron
sf eval my-grpo --model /path/or/hub/id --eval-config configs/eval/gsm8k.yaml
sf bench run --model …          # standard benchmarks, not the same as sf eval

--then export --then eval on submit runs those after SUCCEEDED.

Tool environments

GRPO/PPO can point at modules under common/environments/. See Agent environments.