> ## Documentation Index > Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # sf bench > 对某次 run 或任意模型跑标准评测，以及回报外部产生的分数。 ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench ls # 这里能评什么 sf bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu ``` `` 是一个评测实验，用 `sf new my-bench --method evalkit/benchmark` 建一次然后复用。 围绕这些命令的完整流程见[跑一次评测](/zh-Hans/guides/benchmarks)。 ## `sf bench ls` 打印什么 内置基准随平台发布，用户发布的经 `sf recipe sync` 同步下来。 每一行说明它用哪个 runner、主指标是什么、属于哪一类。 ```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} math gsm8k built-in lm-eval exact match (strict) GSM8K · grade-school word problems math built-in lm-eval exact match MATH · competition mathematics code humaneval built-in lm-eval pass@1 HumanEval · code generation ``` ## runner 通常轮不到你选 用哪个 runner、few-shot 给几个、样本上限多少，都由基准自己声明—— 所以你不用记住「gsm8k 要 5-shot」或者「humaneval 要加 `--confirm_run_unsafe_code`」。 `--runner` 只在评测平台没有收录的原始 suite 名时才需要。 如果你选中的基准跨了两个 runner——比如 `gsm8k` 用 lm-eval、`ceval` 用 evalscope—— 提交会被拒绝，并告诉你分两次提交。一次评测作业是一个进程， 而悄悄把它拆成两次比明说更糟。 ## 示例 ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf new my-bench --method evalkit/benchmark # 每个项目做一次 sf bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu sf bench run my-bench -m Qwen/Qwen2.5-7B --suites ceval sf bench run my-bench -m run:run-4f2a91 --suites gsm8k --limit 20 # 仅用于冒烟 ``` ## `sf bench` Run standard benchmarks ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench ``` | 命令 | 说明 | | ------------------- | -------------------------------------- | | `sf bench external` | Report scores from an external harness | | `sf bench ls` | List the available benchmarks | | `sf bench new` | Scaffold a safety benchmark pack | | `sf bench run` | Submit a benchmark evaluation job | ### `sf bench external` Report scores from an external harness ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench external ``` | 命令 | 说明 | | -------------------------- | --------------------------- | | `sf bench external create` | Open an external evaluation | | `sf bench external submit` | Report a score file | #### `sf bench external create` Open an external evaluation ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench external create [options] ``` | 选项 | 说明 | | --------------- | ---------------------------------------------------------------------------------------------------------------- | | `--model`, `-m` | Reference of the model under evaluation (deployment id / model version / HF id) (**必填**) | | `--harness` | Which harness is used, e.g. cvdp | | `--train-run` | Which training run's artifact is evaluated; only with this do the gate and the model version accept these scores | | `--note` | Note | #### `sf bench external submit` Report a score file ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench external submit [options] ``` | 参数 | 说明 | | -------- | -------------------------------------------------- | | `RUN_ID` | The run\_id that external create returned (**必填**) | | 选项 | 说明 | | --------------- | -------------------------------------------- | | `--file`, `-f` | Score JSON file (**必填**) | | `--model`, `-m` | Model reference (used to group the variance) | | `--harness` | Harness name | ### `sf bench ls` List the available benchmarks ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench ls [options] ``` | 选项 | 说明 | | ------------ | -------------------------------------------------------------------------------- | | `--category` | One category only: math\|code\|knowledge\|chinese\|instruction\|hardware\|safety | ### `sf bench new` Scaffold a safety benchmark pack ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench new [options] ``` | 参数 | 说明 | | ------ | --------------------------------------- | | `NAME` | Pack name, e.g. refusal-policy (**必填**) | | 选项 | 说明 | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--owner` | Namespace segment for the local catalog; defaults to your OS user. Not an identity claim -- user packs are synced down from the platform, not pushed up from here | | `--rubric` | \/\ of the rubric that decides what counts as a refusal | | `--title` | Display title; defaults to the name | ### `sf bench run` Submit a benchmark evaluation job ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench run [options] ``` | 参数 | 说明 | | ----- | --------------------------------------------------------------------------------------------- | | `EXP` | The evaluation experiment (create it with sf new \ --method evalkit/benchmark) (**必填**) | | 选项 | 说明 | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--model`, `-m` | HF model id / absolute path on shared storage / run:\ (takes that run's hf\_export) (**必填**) | | `--suites` | Comma-separated benchmarks (e.g. gsm8k,mmlu). Run `sf bench ls` for the options (**必填**) | | `--runner` | Evaluation backend: lm-eval \| evalscope. Declared by the benchmark itself by default, rarely needed | | `--limit` `` | Sample cap per benchmark (for smoke runs) | | `--extra-args` | Extra arguments passed straight to the runner | | `--profile` | Target hardware and resources in one argument: h200 (the registry default shape), h200:4 (4 GPUs), h200:16 (2 full nodes). Repeat it to split pools per role (the heterogeneous extension point): --profile train=h200:8 --profile rollout=h100:2. Comma-separate roles that share a pool: --profile actor,rollout=h200:8 --profile teacher=h200:8 (可重复) | | `--allow-dirty` | Allow uncommitted changes in the working tree | | `--repeat` `` | Repeat the evaluation this many times to measure the evaluation's own variance (the gate needs it to call a regression) (默认 `1`) |