Submit a training job

The full sf submit flow — resource declaration, hyperparameter overrides, data references, images, and observability

sf submit my-grpo --profile h200:8

Local validation, pack and upload, server admit (catalog / quota / preflight), queue, dispatch. Flags below are the knobs on that path.

Resources: —profile

sf submit my-exp --profile h200          # registry default shape
sf submit my-exp --profile h200:4        # 4 GPUs
sf submit my-exp --profile h200:16       # 2 full nodes
sf submit my-exp --profile train=h200:8 --profile rollout=h100:2   # heterogeneous multi-pool

--profile is the only resource entry point; see Resource Model for the semantics.

Hyperparameters: —set

sf submit my-exp -s policy.optimizer.kwargs.lr=1e-6 -s grpo.kl_coef=0.05

Repeatable; types and ranges are validated locally against the method declaration, and typos fail immediately. Only affects this submission and does not modify experiment files.

Model and data

NeMo-RL

Model and data are declared in the config (base + model fragment); no extra arguments are usually needed at submission.

verl / TRL

sf submit my-verl \
  --model Qwen/Qwen3.5-9B \
  --train-data data/train.parquet \
  --validation-data data/val.parquet

--model (HF id or path) and --train-data / --validation-data are required.

Platform datasets

sf submit my-verl \
  --train-dataset alice/gsm8k-zh@v2 \
  --train-data train.parquet          # relative filename within the dataset

When referencing a platform dataset (<owner>/<name>[@version]), the job automatically pulls it to the shared cache at startup and injects <NAME>_DATA_DIR; --train-data takes a path relative to the dataset. Prefer declaring it in the config’s data.train.dataset; CLI arguments serve only as a temporary override. See Datasets.

Project name

The console groups all runs under the name in starforge.yaml. Submission attaches it automatically; do not pass --project.

Per-job image override

sf submit my-custom --profile h200:8 \
  --image myregistry.io/train:v1

Catalog methods and custom both accept --image; custom requires it. Resolution order: --image → console framework default → runtime registry → catalog. Tags work; pin a digest in production. The registry must be on the server allowlist.

Logs follow stdout. Console curves need starforge.report in training code. The catalog custom recipe currently defaults to external observability, so those submissions still need --observability-url. Full cookbook: Custom training. Docker and allowlist: Custom images.

Provenance and waivers

Flag	Semantics
--allow-dirty	Submit even when the workspace has uncommitted changes (rejected by default to guarantee traceability to an exact commit). Untracked files are listed as warnings
--no-validate	Skip pre-submission validation
--upgrade-recipe	Upgrade the experiment lock to the current catalog before submitting (combine with --framework-version to switch versions at the same time)

The packager has a sensitive-file blacklist (.env, *.pem, id_rsa*, etc.); a hit refuses to package — secrets must not enter the job package, and the platform has a dedicated secret injection channel.

Post-training automatic actions: —then

sf submit my-exp --then export --then eval

Automatically runs export / evaluation after training succeeds; see Pipelines.

After submission

sf job ls                 # job list
sf job logs               # follow the latest job's logs (-n 0 for the full history)
sf job status <JOB_ID>    # status details
sf job stop <JOB_ID>      # stop
sf job pause <JOB_ID>     # pause (keeps the checkpoint)
sf job resume <JOB_ID>    # resume training from the latest checkpoint
sf job retry <JOB_ID>     # requeue a failed or stopped job (same run id, from the latest checkpoint)

Common reasons the server may reject a submission (all with explicit errors): catalog handshake failure (stale lock), insufficient quota (waiting in the queue is not a failure), image registry not on the allowlist, HF gated model not authorized, batch size validation failure. See Troubleshooting.