sf bench

Run standard benchmarks against a run or any model, and report externally produced scores.

sf bench ls                                       # what can be evaluated here
sf bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu

<EXP> is an evaluation experiment, created once with sf new my-bench --method evalkit/benchmark and reused. The workflow around these commands is in Run a benchmark.

What sf bench ls prints

Built-in benchmarks ship with the platform; published ones arrive through sf recipe sync. Each row says which runner it uses, its primary metric, and its category.

math
  gsm8k      built-in  lm-eval    exact match (strict)  GSM8K · grade-school word problems
  math       built-in  lm-eval    exact match           MATH · competition mathematics
code
  humaneval  built-in  lm-eval    pass@1                HumanEval · code generation

The runner is usually not yours to pick

Which runner, how many few-shot examples, and the sample cap are declared by the benchmark itself — so you do not have to remember that GSM8K wants 5-shot or that HumanEval needs --confirm_run_unsafe_code. --runner is only for a raw suite name the platform has no pack for.

If the benchmarks you select span two runners — say gsm8k on lm-eval and ceval on evalscope — the submission is refused and tells you to split it in two. One evaluation job is one process, and quietly splitting it would be worse than saying so.

Examples

sf new my-bench --method evalkit/benchmark              # once per project
sf bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu
sf bench run my-bench -m Qwen/Qwen2.5-7B --suites ceval
sf bench run my-bench -m run:run-4f2a91 --suites gsm8k --limit 20   # smoke test only

sf bench

Run standard benchmarks

sf bench <subcommand>

Command	Description
sf bench external	Report scores from an external harness
sf bench ls	List the available benchmarks
sf bench new	Scaffold a safety benchmark pack
sf bench run	Submit a benchmark evaluation job

sf bench external

Report scores from an external harness

sf bench external <subcommand>

Command	Description
sf bench external create	Open an external evaluation
sf bench external submit	Report a score file

sf bench external create

Open an external evaluation

sf bench external create [options]

Option	Description
--model, -m	Reference of the model under evaluation (deployment id / model version / HF id) (required)
--harness	Which harness is used, e.g. cvdp
--train-run	Which training run’s artifact is evaluated; only with this do the gate and the model version accept these scores
--note	Note

sf bench external submit

Report a score file

sf bench external submit <RUN_ID> [options]

Argument	Description
RUN_ID	The run_id that external create returned (required)

Option	Description
--file, -f	Score JSON file (required)
--model, -m	Model reference (used to group the variance)
--harness	Harness name

sf bench ls

List the available benchmarks

sf bench ls [options]

Option	Description
--category	One category only: math|code|knowledge|chinese|instruction|hardware|safety

sf bench new

Scaffold a safety benchmark pack

sf bench new <NAME> [options]

Argument	Description
NAME	Pack name, e.g. refusal-policy (required)

Option	Description
--owner	Namespace segment for the local catalog; defaults to your OS user. Not an identity claim — user packs are synced down from the platform, not pushed up from here
--rubric	<owner>/<name> of the rubric that decides what counts as a refusal
--title	Display title; defaults to the name

sf bench run

Submit a benchmark evaluation job

sf bench run <EXP> [options]

Argument	Description
EXP	The evaluation experiment (create it with sf new <name> —method evalkit/benchmark) (required)

Option	Description
--model, -m	HF model id / absolute path on shared storage / run:<run_id> (takes that run’s hf_export) (required)
--suites	Comma-separated benchmarks (e.g. gsm8k,mmlu). Run sf bench ls for the options (required)
--runner	Evaluation backend: lm-eval | evalscope. Declared by the benchmark itself by default, rarely needed
--limit <int>	Sample cap per benchmark (for smoke runs)
--extra-args	Extra arguments passed straight to the runner
--profile	Target hardware and resources in one argument: h200 (the registry default shape), h200:4 (4 GPUs), h200:16 (2 full nodes). Repeat it to split pools per role (the heterogeneous extension point): —profile train=h200:8 —profile rollout=h100:2. Comma-separate roles that share a pool: —profile actor,rollout=h200:8 —profile teacher=h200:8 (repeatable)
--allow-dirty	Allow uncommitted changes in the working tree
--repeat <int range>	Repeat the evaluation this many times to measure the evaluation’s own variance (the gate needs it to call a regression) (default 1)