sf eval

Run a recipe’s native evaluation

sf eval my-grpo --model Qwen/Qwen2.5-7B

Runs the evaluation that ships with the recipe’s own framework — NeMo-RL’s evaluation config, verl’s SFT validation loop. This is the framework’s notion of “how is it doing”, on the framework’s terms.

That makes it different from sf bench, which runs standard academic benchmarks through an external harness so results are comparable across runs, methods, and other people’s models. Reach for sf eval while iterating on one method; reach for sf bench when comparing.

It can also run automatically after training:

sf submit my-grpo --profile h200:8 --then eval

sf eval

Run a recipe’s native evaluation

sf eval <EXP> [options]

Argument	Description
EXP	Experiment name or path (required)

Option	Description
--run-id	VeRL SFT: the training run id that receives the validation samples
--model	NeMo-RL or VeRL SFT: HF model path / Hub id
--eval-config	NeMo-RL: explicit evaluation config path
--data	verl: explicit evaluation data path
--step <int range>	VeRL SFT: the training step the exported checkpoint belongs to
--profile	Hardware profile (managed by the server registry; sf status lists the available values)
--allow-dirty	Allow uncommitted changes in the working tree
--dry-run	Only print the command that would be submitted