Create and tune an experiment

sf new scaffolding, layered config.yaml inheritance, local validation

An experiment is a self-contained directory: experiments/<name>/, containing config.yaml (the tuning entry point), README.md, and recipe.lock.json (the method lock). Create a repository first with sf init, then run the commands below inside that repository. Disk layout and upload excludes: Lab layout. Inheritance details: config.yaml.

Create an experiment

From the method catalog

sf new my-grpo --method nemo-rl/grpo
sf new my-verl --method verl/grpo --framework-version 0.9.0
sf new my-kto  --method trl/kto

The scaffold comes from the recipe’s official template: config base, README tuning notes, and lock file all in place at once.

Fork an existing experiment

sf new my-grpo-v2 --from my-grpo

Inherits the source experiment’s full configuration and lock (--method is ignored). Ideal for incremental changes on top of an already-converged configuration.

config.yaml: write only the diff

An experiment config inherits from official bases via defaults and only writes the keys you want to change:

experiments/my-grpo/config.yaml

defaults:
  - ../../configs/base/grpo_math_1B.yaml     # official base
  - ../../configs/models/qwen3.5-9b.yaml     # model fragment

# ── tuning section ──
policy:
  optimizer:
    kwargs:
      lr: 2.0e-6
grpo:
  num_generations: 8
  kl_coef: 0.01

Hardware and distributed details (parallelism, GPU memory tuning, NCCL) are not written in the experiment config — they are delivered from the server-side registry via --profile at submission time. Switching GPU types for the same experiment requires no config change.

The top of the scaffold contains a “tuning cheat sheet” comment block listing the method’s most commonly changed keys (learning rate, batch, sequence length, validation interval, etc.) with empirical ranges.

Local validation

sf validate my-grpo

The validator checks against the recipe declaration:

• Key validity: struct mode; misspelled config keys fail immediately;
• Types and ranges: learning rate, batch, etc. are checked against declared types and value ranges;
• Batch size relationships: methods like GRPO validate the divisibility relationship between rollout batch and train batch (divisible but unequal lengths produce an off-policy warning; not divisible is an error);
• Data declarations: required --model / --train-data for verl / TRL are caught early if missing.

sf submit automatically runs the same validation; --no-validate skips it (not recommended).

Browse methods and hyperparameters

sf methods                # all methods: one-line summary + supported framework versions
sf methods nemo-rl/grpo   # tunable hyperparameters for that method: type, default, range, description
sf ls                     # list all experiment directories in this repository

The console Methods page (/methods) has a visual version of the same catalog.

Comparing runs in the console

The jobs area on a project detail page is a dense comparison matrix:

• Job name and status stay fixed while metric/config columns scroll horizontally;
• Rows and columns are both virtualized, so hundreds of runs × hundreds of columns render only nearby cells;
• All latest metrics and config values that vary across runs are visible by default; constant config columns are available from the Columns panel;
• Search jobs, filter status, search/hide/pin columns, and select 2–4 runs for review comparison.

What else can live in an experiment directory

Content	Purpose
train.sh	Custom only: training entrypoint. Logs follow stdout; curves use starforge.report. See Custom training
Custom environment / reward code	Modules in common/environments/ and common/rewards/ referenced by the config, uploaded with the job package
plugins.lock.json	References platform plugins (sf plugin install --exp); injected by the platform at submission
cluster annotation file	Legacy: default profile record; explicit --profile at submission is now recommended