The experiment config

defaults inheritance, override, interpolation, and what belongs in the experiment file

Catalog experiments are tuned through experiments/<name>/config.yaml. The file is YAML. Inheritance is local (relative paths), not Hydra’s full package system. sf validate and sf submit use the same resolver in starforge.config_resolve.

Custom jobs may ignore this file entirely if train.sh never reads it. Catalog adapters always do.

Write the diff, not a copy of the base

defaults:
  - ../../configs/base/grpo_math_1B.yaml
  - ../../configs/models/qwen3.5-9b.yaml

policy:
  optimizer:
    kwargs:
      lr: 2.0e-6
grpo:
  num_generations: 8
  kl_coef: 0.01

defaults entries are paths relative to this config.yaml. Each listed file is resolved recursively (that file may have its own defaults). Later files win. Keys in the experiment file win last.

The scaffold header is a comment cheat sheet: learning rate, batch, sequence length, validation interval, with ranges from the recipe. Those comments are not executed.

Four layers at submit time

Later wins:

recipe template (copied into configs/ at sf init, plus the experiment scaffold)
└─ experiments/<name>/config.yaml
   └─ hardware profile from the server registry (--profile)
      └─ sf submit --set dotted.key=value

Hardware (FSDP, GPU memory, NCCL, cluster.num_nodes) is not something you maintain in the experiment file. The adapter overwrites topology from FORGE_CLUSTER_NUM_NODES / FORGE_CLUSTER_GPUS_PER_NODE. Switching from 4 GPUs to 8 is a different --profile, not a config edit.

--set is type-checked against the recipe params locally. A typo fails on the laptop.

Replace a whole mapping: _override_

Deep merge is the default. To replace a dict instead of merging keys, set _override_: true on that mapping. The marker is stripped and does not reach the trainer.

some_block:
  _override_: true
  only_these: keys

Interpolation

${...} strings are kept as strings. sf validate skips numeric checks on those fields (it cannot see the value). OmegaConf-style ${oc.env:GSM8K_DATA_DIR}/train.jsonl is recognized for a small set of *_DATA_DIR names when checking that a local path exists. Anything else with ${ is left alone.

Struct mode

Unknown keys fail validation. That is how a misspelled kl_coef is caught before the job occupies a node. Do not add trainer-only keys that the recipe did not declare unless you are on a path that allows extras (rare; custom YAML you parse yourself).

Batch-size relationships (GRPO and similar)

Recipes that declare rollout vs train batch constraints check divisibility. Unequal but divisible lengths produce an off-policy warning. Not divisible is an error.

Local check

sf validate my-grpo
sf methods nemo-rl/grpo    # type, default, range, description per key

sf submit runs the same validator unless you pass --no-validate.