> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Submit a training job

> The full sf submit flow — resource declaration, hyperparameter overrides, data references, images, and observability

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf submit my-grpo --profile h200:8
```

Local validation, pack and upload, server admit (catalog / quota / preflight), queue, dispatch. Flags below are the knobs on that path.

## Resources: --profile

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf submit my-exp --profile h200          # registry default shape
sf submit my-exp --profile h200:4        # 4 GPUs
sf submit my-exp --profile h200:16       # 2 full nodes
sf submit my-exp --profile train=h200:8 --profile rollout=h100:2   # heterogeneous multi-pool
```

`--profile` is the only resource entry point; see [Resource Model](/en/concepts/resources) for the semantics.

## Hyperparameters: --set

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf submit my-exp -s policy.optimizer.kwargs.lr=1e-6 -s grpo.kl_coef=0.05
```

Repeatable; types and ranges are validated locally against the method declaration, and typos fail immediately. Only affects this submission and does not modify experiment files.

## Model and data

<Tabs>
  <Tab title="NeMo-RL">
    Model and data are declared in the config (base + model fragment); no extra arguments are usually needed at submission.
  </Tab>

  <Tab title="verl / TRL">
    ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
    sf submit my-verl \
      --model Qwen/Qwen3.5-9B \
      --train-data data/train.parquet \
      --validation-data data/val.parquet
    ```

    `--model` (HF id or path) and `--train-data` / `--validation-data` are required.
  </Tab>

  <Tab title="Platform datasets">
    ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
    sf submit my-verl \
      --train-dataset alice/gsm8k-zh@v2 \
      --train-data train.parquet          # relative filename within the dataset
    ```

    When referencing a platform dataset (`<owner>/<name>[@version]`), the job automatically pulls it to the shared cache at startup and injects `<NAME>_DATA_DIR`; `--train-data` takes a path relative to the dataset. Prefer declaring it in the config's `data.train.dataset`; CLI arguments serve only as a temporary override. See [Datasets](/en/guides/datasets).
  </Tab>
</Tabs>

## Project name

The console groups all runs under the `name` in `starforge.yaml`. Submission attaches it automatically; do not pass `--project`.

## Per-job image override

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf submit my-custom --profile h200:8 \
  --image myregistry.io/train:v1
```

Catalog methods and custom both accept `--image`; custom requires it. Resolution order: `--image` → console framework default → runtime registry → catalog. Tags work; pin a digest in production. The registry must be on the server allowlist.

Logs follow stdout. Console curves need `starforge.report` in training code. The catalog custom recipe currently defaults to external observability, so those submissions still need `--observability-url`. Full cookbook: [Custom training](/en/guides/custom-training). Docker and allowlist: [Custom images](/en/guides/custom-images).

## Provenance and waivers

| Flag               | Semantics                                                                                                                                                         |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--allow-dirty`    | Submit even when the workspace has uncommitted changes (rejected by default to guarantee traceability to an exact commit). Untracked files are listed as warnings |
| `--no-validate`    | Skip pre-submission validation                                                                                                                                    |
| `--upgrade-recipe` | Upgrade the experiment lock to the current catalog before submitting (combine with `--framework-version` to switch versions at the same time)                     |

<Warning>
  The packager has a sensitive-file blacklist (`.env`, `*.pem`, `id_rsa*`, etc.); a hit **refuses to package** — secrets must not enter the job package, and the platform has a dedicated secret injection channel.
</Warning>

## Post-training automatic actions: --then

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf submit my-exp --then export --then eval
```

Automatically runs export / evaluation after training succeeds; see [Pipelines](/en/guides/pipelines).

## After submission

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job ls                 # job list
sf job logs               # follow the latest job's logs (-n 0 for the full history)
sf job status <JOB_ID>    # status details
sf job stop <JOB_ID>      # stop
sf job pause <JOB_ID>     # pause (keeps the checkpoint)
sf job resume <JOB_ID>    # resume training from the latest checkpoint
sf job retry <JOB_ID>     # requeue a failed or stopped job (same run id, from the latest checkpoint)
```

Common reasons the server may reject a submission (all with explicit errors): catalog handshake failure (stale lock), insufficient quota (waiting in the queue is not a failure), image registry not on the allowlist, HF gated model not authorized, batch size validation failure. See [Troubleshooting](/en/troubleshooting).