sf bench

对某次 run 或任意模型跑标准评测，以及回报外部产生的分数。

sf bench ls                                       # 这里能评什么
sf bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu

<EXP> 是一个评测实验，用 sf new my-bench --method evalkit/benchmark 建一次然后复用。 围绕这些命令的完整流程见跑一次评测。

​

sf bench ls 打印什么

内置基准随平台发布，用户发布的经 sf recipe sync 同步下来。 每一行说明它用哪个 runner、主指标是什么、属于哪一类。

math
  gsm8k      built-in  lm-eval    exact match (strict)  GSM8K · grade-school word problems
  math       built-in  lm-eval    exact match           MATH · competition mathematics
code
  humaneval  built-in  lm-eval    pass@1                HumanEval · code generation

​

runner 通常轮不到你选

用哪个 runner、few-shot 给几个、样本上限多少，都由基准自己声明—— 所以你不用记住「gsm8k 要 5-shot」或者「humaneval 要加 --confirm_run_unsafe_code」。 --runner 只在评测平台没有收录的原始 suite 名时才需要。

如果你选中的基准跨了两个 runner——比如 gsm8k 用 lm-eval、ceval 用 evalscope—— 提交会被拒绝，并告诉你分两次提交。一次评测作业是一个进程， 而悄悄把它拆成两次比明说更糟。

​

示例

sf new my-bench --method evalkit/benchmark              # 每个项目做一次
sf bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu
sf bench run my-bench -m Qwen/Qwen2.5-7B --suites ceval
sf bench run my-bench -m run:run-4f2a91 --suites gsm8k --limit 20   # 仅用于冒烟

​

sf bench

Run standard benchmarks

sf bench <subcommand>

命令	说明
sf bench external	Report scores from an external harness
sf bench ls	List the available benchmarks
sf bench new	Scaffold a safety benchmark pack
sf bench run	Submit a benchmark evaluation job

​

sf bench external

Report scores from an external harness

sf bench external <subcommand>

命令	说明
sf bench external create	Open an external evaluation
sf bench external submit	Report a score file

​

sf bench external create

Open an external evaluation

sf bench external create [options]

选项	说明
--model, -m	Reference of the model under evaluation (deployment id / model version / HF id) (必填)
--harness	Which harness is used, e.g. cvdp
--train-run	Which training run’s artifact is evaluated; only with this do the gate and the model version accept these scores
--note	Note

​

sf bench external submit

Report a score file

sf bench external submit <RUN_ID> [options]

参数	说明
RUN_ID	The run_id that external create returned (必填)

选项	说明
--file, -f	Score JSON file (必填)
--model, -m	Model reference (used to group the variance)
--harness	Harness name

​

sf bench ls

List the available benchmarks

sf bench ls [options]

选项	说明
--category	One category only: math|code|knowledge|chinese|instruction|hardware|safety

​

sf bench new

Scaffold a safety benchmark pack

sf bench new <NAME> [options]

参数	说明
NAME	Pack name, e.g. refusal-policy (必填)

选项	说明
--owner	Namespace segment for the local catalog; defaults to your OS user. Not an identity claim — user packs are synced down from the platform, not pushed up from here
--rubric	<owner>/<name> of the rubric that decides what counts as a refusal
--title	Display title; defaults to the name

​

sf bench run

Submit a benchmark evaluation job

sf bench run <EXP> [options]

参数	说明
EXP	The evaluation experiment (create it with sf new <name> —method evalkit/benchmark) (必填)

选项	说明
--model, -m	HF model id / absolute path on shared storage / run:<run_id> (takes that run’s hf_export) (必填)
--suites	Comma-separated benchmarks (e.g. gsm8k,mmlu). Run sf bench ls for the options (必填)
--runner	Evaluation backend: lm-eval | evalscope. Declared by the benchmark itself by default, rarely needed
--limit <int>	Sample cap per benchmark (for smoke runs)
--extra-args	Extra arguments passed straight to the runner
--profile	Target hardware and resources in one argument: h200 (the registry default shape), h200:4 (4 GPUs), h200:16 (2 full nodes). Repeat it to split pools per role (the heterogeneous extension point): —profile train=h200:8 —profile rollout=h100:2. Comma-separate roles that share a pool: —profile actor,rollout=h200:8 —profile teacher=h200:8 (可重复)
--allow-dirty	Allow uncommitted changes in the working tree
--repeat <int range>	Repeat the evaluation this many times to measure the evaluation’s own variance (the gate needs it to call a regression) (默认 1)