Run a benchmark

Score a finished run against GSM8K, MMLU, C-Eval and the rest, on one comparable dashboard.

sf new my-bench --method evalkit/benchmark      # once per project
sf bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu

Scores land on the console’s Benchmarks page, in the same matrix as every other run — comparable across runs, methods and base models.

The benchmark matrix: runs down, benchmarks across, best score highlighted.

What you can score

-m accepts	Example
A run	run:run-4f2a91 — resolves that run’s exported weights automatically
A Hub model	Qwen/Qwen2.5-7B
A path on shared storage	/mnt/shared/models/qwen-support-v3

Scoring a run requires it to have been exported. --then export at submission time saves you remembering.

Which benchmarks are available

sf bench ls

Built in: gsm8k, math, humaneval, mmlu, ceval, ifeval, plus the RTL packs. Anything published as a benchmark pack appears here too.

Those axes are chosen deliberately — math, code, knowledge, Chinese, instruction following. Post- training reliably improves one and quietly damages another, and running several is what makes that visible before it reaches production rather than after.

Common options

Flag	Use it for
--suites	Comma-separated benchmark names
--runner	lm-eval (default) or evalscope, which covers Chinese benchmarks better
--limit	Cap samples per benchmark. Smoke tests only — never for a score you will quote
--profile	Resources, exactly as for training
--extra-args	Passed through to the runner

A --limit score is not comparable with a full one, and nothing downstream marks it as truncated. Use it to check that the pipeline works, then run without it.

Both runners produce the same score contract, so the same benchmark is directly comparable whichever ran it. The choice of runner does not distort the number.

Score every run automatically

Running it by hand is fine once. Making it happen on every run is what stops a regression reaching production:

sf submit my-grpo --profile h200:8 --then eval

Or declare an evaluation gate in the experiment config, so a run that falls below a threshold is marked as failing its gate — see the eval: block in your method’s template.

Reading the matrix

Scores are ingested idempotently: re-reporting the same run and benchmark updates the row rather than adding a second one.

Each metric carries a declared direction, so the comparison view knows which way is better without guessing. Where a benchmark declares more than one metric, the difference between them is often the useful part — GSM8K’s strict and flexible matches diverging means the model can do the arithmetic but is not formatting the answer as asked.

Scoring outside the platform

When the harness cannot run here — a licensed tool, a physical test rig — report the numbers back instead:

sf bench external create -m run:run-4f2a91 --harness cvdp --train-run run-4f2a91
sf bench external submit <RUN_ID> --file scores.json

create returns a run id; submit reports the score file against it. --train-run is what lets the evaluation gate and the model version accept these scores — without it they are recorded but not authoritative.

They enter the same matrix, labelled as externally produced.

Confirm it worked

sf job ls          # the benchmark job, like any other
sf job logs

Then open the Benchmarks page. A row for your run with a score in each requested column means it worked. An empty column means that suite failed — the job log names which.

Next

Write a benchmark pack

Declare your own evaluation set: what to run, and which numbers to read.

Rubrics

Score what a rule cannot, with an LLM judge against a written standard.