> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Run a benchmark

> Score a finished run against GSM8K, MMLU, C-Eval and the rest, on one comparable dashboard.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf new my-bench --method evalkit/benchmark      # once per project
sf bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu
```

Scores land on the console's Benchmarks page, in the same matrix as every other run — comparable
across runs, methods and base models.

<Frame caption="The benchmark matrix: runs down, benchmarks across, best score highlighted.">
  <img src="https://mintcdn.com/starforge/GatXR2rI5-_Vm4_H/images/console/benchmarks.png?fit=max&auto=format&n=GatXR2rI5-_Vm4_H&q=85&s=4c566bdcc2e13ced6378a9d975671356" alt="StarForge benchmarks page" width="2160" height="1350" data-path="images/console/benchmarks.png" />
</Frame>

## What you can score

| `-m` accepts             | Example                                                               |
| ------------------------ | --------------------------------------------------------------------- |
| A run                    | `run:run-4f2a91` — resolves that run's exported weights automatically |
| A Hub model              | `Qwen/Qwen2.5-7B`                                                     |
| A path on shared storage | `/mnt/shared/models/qwen-support-v3`                                  |

Scoring a run requires it to have been exported. `--then export` at submission time saves you
remembering.

## Which benchmarks are available

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf bench ls
```

Built in: `gsm8k`, `math`, `humaneval`, `mmlu`, `ceval`, `ifeval`, plus the RTL packs. Anything
published as a [benchmark pack](/en/extend/benchmark-packs) appears here too.

Those axes are chosen deliberately — math, code, knowledge, Chinese, instruction following. Post-
training reliably improves one and quietly damages another, and running several is what makes that
visible before it reaches production rather than after.

## Common options

| Flag           | Use it for                                                                     |
| -------------- | ------------------------------------------------------------------------------ |
| `--suites`     | Comma-separated benchmark names                                                |
| `--runner`     | `lm-eval` (default) or `evalscope`, which covers Chinese benchmarks better     |
| `--limit`      | Cap samples per benchmark. Smoke tests only — never for a score you will quote |
| `--profile`    | Resources, exactly as for training                                             |
| `--extra-args` | Passed through to the runner                                                   |

<Warning>
  A `--limit` score is not comparable with a full one, and nothing downstream marks it as truncated.
  Use it to check that the pipeline works, then run without it.
</Warning>

Both runners produce the **same score contract**, so the same benchmark is directly comparable
whichever ran it. The choice of runner does not distort the number.

## Score every run automatically

Running it by hand is fine once. Making it happen on every run is what stops a regression reaching
production:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf submit my-grpo --profile h200:8 --then eval
```

Or declare an evaluation gate in the experiment config, so a run that falls below a threshold is
marked as failing its gate — see the `eval:` block in your method's template.

## Reading the matrix

Scores are ingested idempotently: re-reporting the same run and benchmark updates the row rather
than adding a second one.

Each metric carries a declared direction, so the comparison view knows which way is better without
guessing. Where a benchmark declares more than one metric, the difference between them is often the
useful part — GSM8K's strict and flexible matches diverging means the model can do the arithmetic
but is not formatting the answer as asked.

## Scoring outside the platform

When the harness cannot run here — a licensed tool, a physical test rig — report the numbers back
instead:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf bench external create -m run:run-4f2a91 --harness cvdp --train-run run-4f2a91
sf bench external submit <RUN_ID> --file scores.json
```

`create` returns a run id; `submit` reports the score file against it. `--train-run` is what lets
the evaluation gate and the model version accept these scores — without it they are recorded but
not authoritative.

They enter the same matrix, labelled as externally produced.

## Confirm it worked

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job ls          # the benchmark job, like any other
sf job logs
```

Then open the Benchmarks page. A row for your run with a score in each requested column means it
worked. An empty column means that suite failed — the job log names which.

## Next

<Columns cols={2}>
  <Card title="Write a benchmark pack" icon="package" href="/en/extend/benchmark-packs" arrow="true">
    Declare your own evaluation set: what to run, and which numbers to read.
  </Card>

  <Card title="Rubrics" icon="ruler" href="/en/guides/rubrics" arrow="true">
    Score what a rule cannot, with an LLM judge against a written standard.
  </Card>
</Columns>