> ## Documentation Index > Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # sf bench > Run standard benchmarks against a run or any model, and report externally produced scores. ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench ls # what can be evaluated here sf bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu ``` `` is an evaluation experiment, created once with `sf new my-bench --method evalkit/benchmark` and reused. The workflow around these commands is in [Run a benchmark](/en/guides/benchmarks). ## What `sf bench ls` prints Built-in benchmarks ship with the platform; published ones arrive through `sf recipe sync`. Each row says which runner it uses, its primary metric, and its category. ```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} math gsm8k built-in lm-eval exact match (strict) GSM8K · grade-school word problems math built-in lm-eval exact match MATH · competition mathematics code humaneval built-in lm-eval pass@1 HumanEval · code generation ``` ## The runner is usually not yours to pick Which runner, how many few-shot examples, and the sample cap are declared by the benchmark itself — so you do not have to remember that GSM8K wants 5-shot or that HumanEval needs `--confirm_run_unsafe_code`. `--runner` is only for a raw suite name the platform has no pack for. If the benchmarks you select span two runners — say `gsm8k` on lm-eval and `ceval` on evalscope — the submission is refused and tells you to split it in two. One evaluation job is one process, and quietly splitting it would be worse than saying so. ## Examples ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf new my-bench --method evalkit/benchmark # once per project sf bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu sf bench run my-bench -m Qwen/Qwen2.5-7B --suites ceval sf bench run my-bench -m run:run-4f2a91 --suites gsm8k --limit 20 # smoke test only ``` ## `sf bench` Run standard benchmarks ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench ``` | Command | Description | | ------------------- | -------------------------------------- | | `sf bench external` | Report scores from an external harness | | `sf bench ls` | List the available benchmarks | | `sf bench new` | Scaffold a safety benchmark pack | | `sf bench run` | Submit a benchmark evaluation job | ### `sf bench external` Report scores from an external harness ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench external ``` | Command | Description | | -------------------------- | --------------------------- | | `sf bench external create` | Open an external evaluation | | `sf bench external submit` | Report a score file | #### `sf bench external create` Open an external evaluation ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench external create [options] ``` | Option | Description | | --------------- | ---------------------------------------------------------------------------------------------------------------- | | `--model`, `-m` | Reference of the model under evaluation (deployment id / model version / HF id) (**required**) | | `--harness` | Which harness is used, e.g. cvdp | | `--train-run` | Which training run's artifact is evaluated; only with this do the gate and the model version accept these scores | | `--note` | Note | #### `sf bench external submit` Report a score file ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench external submit [options] ``` | Argument | Description | | -------- | -------------------------------------------------------- | | `RUN_ID` | The run\_id that external create returned (**required**) | | Option | Description | | --------------- | -------------------------------------------- | | `--file`, `-f` | Score JSON file (**required**) | | `--model`, `-m` | Model reference (used to group the variance) | | `--harness` | Harness name | ### `sf bench ls` List the available benchmarks ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench ls [options] ``` | Option | Description | | ------------ | -------------------------------------------------------------------------------- | | `--category` | One category only: math\|code\|knowledge\|chinese\|instruction\|hardware\|safety | ### `sf bench new` Scaffold a safety benchmark pack ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench new [options] ``` | Argument | Description | | -------- | --------------------------------------------- | | `NAME` | Pack name, e.g. refusal-policy (**required**) | | Option | Description | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--owner` | Namespace segment for the local catalog; defaults to your OS user. Not an identity claim -- user packs are synced down from the platform, not pushed up from here | | `--rubric` | \/\ of the rubric that decides what counts as a refusal | | `--title` | Display title; defaults to the name | ### `sf bench run` Submit a benchmark evaluation job ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench run [options] ``` | Argument | Description | | -------- | --------------------------------------------------------------------------------------------------- | | `EXP` | The evaluation experiment (create it with sf new \ --method evalkit/benchmark) (**required**) | | Option | Description | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--model`, `-m` | HF model id / absolute path on shared storage / run:\ (takes that run's hf\_export) (**required**) | | `--suites` | Comma-separated benchmarks (e.g. gsm8k,mmlu). Run `sf bench ls` for the options (**required**) | | `--runner` | Evaluation backend: lm-eval \| evalscope. Declared by the benchmark itself by default, rarely needed | | `--limit` `` | Sample cap per benchmark (for smoke runs) | | `--extra-args` | Extra arguments passed straight to the runner | | `--profile` | Target hardware and resources in one argument: h200 (the registry default shape), h200:4 (4 GPUs), h200:16 (2 full nodes). Repeat it to split pools per role (the heterogeneous extension point): --profile train=h200:8 --profile rollout=h100:2. Comma-separate roles that share a pool: --profile actor,rollout=h200:8 --profile teacher=h200:8 (repeatable) | | `--allow-dirty` | Allow uncommitted changes in the working tree | | `--repeat` `` | Repeat the evaluation this many times to measure the evaluation's own variance (the gate needs it to call a regression) (default `1`) |