Train with TRL

Accelerate + TRL trainers, required model/data, OPSD teacher, image

sf new my-sft --method trl/sft
sf submit my-sft --profile h200:4 \
  --model Qwen/Qwen2.5-7B --train-data data/train.parquet

Hugging Face TRL 1.10.0 as its own runtime, not routed through custom. Methods: trl/sft, dpo, grpo, rm, kto, rloo, opsd.

Observability is platform. TRL’s own metric names (loss, eval_loss, learning_rate) are aliased onto the console’s series, so the charts fill in without you renaming anything.

Create

sf new my-sft --method trl/sft
sf new my-kto --method trl/kto
sf new my-opsd --method trl/opsd

Entrypoint is train.py in the experiment directory (kind: experiment). Edit that file only if you know you need a fork; hyperparameters belong in config.yaml / --set.

Submit: model and data required

Same idea as verl: --model and --train-data (and usually --validation-data) are required unless the config already has them in the bound fields.

sf submit my-sft --profile h200:8 \
  --model Qwen/Qwen3.5-9B \
  --train-data data/train.parquet \
  --validation-data data/val.parquet

trl/opsd needs a same-vocab teacher (teacher_model in the recipe params). Missing it fails validation.

Gated Hub repos: link HuggingFace on the console so submit preflight can check access. HuggingFace.

Launch

Adapter uses Accelerate (adapter_options.launcher: accelerate). GPU count still comes from --profile / FORGE_CLUSTER_GPUS_PER_NODE.

After training

Checkpoints match HuggingFace layout (checkpoint-*, final_model). Export/eval lifecycle is declared on the recipe (starforge.lifecycle.export_complete, experiment eval.py).

sf export my-sft --checkpoint <path> --checkpoint-format huggingface