Benchmark packs

Declare an evaluation set: what to run, which numbers to read, and how to read them.

sf bench new my-bench            # scaffold a pack
sf plugin publish ./my-bench     # publish it
sf bench ls                      # it is now in the catalog
sf bench run my-bench -m run:run-4f2a91

A benchmark pack answers three questions in YAML, and contains no code: what to run, which numbers to read, and how to run it by default. The thing that actually executes is a runner that ships with the SDK.

Layout

my-bench/
├── plugin.yaml       # kind: benchmark
├── benchmark.yaml    # the declaration
└── README.md

benchmark.yaml

benchmark.yaml

schema: forge/benchmark/v1
name: gsm8k
title: GSM8K · Grade-school math word problems
category: math
summary: >
  8.5K grade-school math word problems requiring 2-8 reasoning steps. The usual entry
  benchmark for math post-training: a clean signal, fast to run, and well correlated with
  the RL reward curve, which makes it a good spot check mid-training.

runner: lm-eval
suites: [gsm8k]
num_fewshot: 5
batch_size: auto
keywords: [math, reasoning, cot]

metrics:
  - key: exact_match,strict-match
    label: Exact match (strict)
    primary: true
  - key: exact_match,flexible-extract
    label: Exact match (flexible)

i18n:
  zh:
    title: GSM8K · 小学数学应用题
    summary: >
      8.5K 道小学数学应用题，需要 2-8 步推理。

Fields

name · string · required

Leaf name. A published pack is referenced as <owner>.<name>; built-ins keep the bare name.

runner · lm-eval | evalscope | rtl | judge | safety · required

Which harness executes the evaluation. See the table below.

suites · string[] · required

The suite names the runner understands. For lm-eval these are its task names.

category · string · default: "general"

Dashboard grouping — math, code, knowledge, chinese, instruction, and so on.

num_fewshot, batch_size, limit, extra_args · mixed

Defaults for running it. batch_size defaults to auto, limit caps the number of samples. A user can override all of these at submission; what you declare is what happens when they do not.

This is the point of the pack: nobody should have to remember that GSM8K wants 5-shot or that HumanEval needs --confirm_run_unsafe_code.

rubric · <owner>/<name>

Only read by the judge runner, and for that runner it is the benchmark’s definition. Swap the rubric and it is a different benchmark — which is why it is a declared field rather than a flag buried in extra_args that nobody can see from the catalog.

Metrics

Each entry says which number to pull out of the runner’s report and how to present it.

key · string · required

The raw metric name the runner emits, verbatim — exact_match,strict-match for lm-eval. This is the machine-facing name and it is not translatable.

label · string

What the dashboard column says. Defaults to key, which is fine for a machine and poor for a human.

direction · higher | lower | neutral · default: "higher"

Which way is better. The comparison view colours regressions by this.

ratio · bool · default: "true"

Whether the value is a 0–1 proportion, so the dashboard renders it as a percentage.

primary · bool · default: "false"

The metric cross-run comparison defaults to. Exactly one should be primary; if none is, the first is used.

Declare more than one metric where the difference between them is diagnostic. GSM8K’s strict and flexible matches are the standard example: a low strict score with a high flexible score means the model can do the arithmetic but is not formatting the answer as asked — a prompt problem, not a capability problem. One number would hide that.

The five runners

Runner	What it runs	Use it for
lm-eval	EleutherAI lm-evaluation-harness	Academic standard benchmarks. The default choice
evalscope	ModelScope evalscope	Chinese benchmarks — C-Eval, CMMLU — which it covers better
rtl	An RTL/hardware harness pinned in the evalkit image	Verilog and hardware-design suites
judge	Generation with vLLM, scored by the platform judge	Anything a rule cannot score. Needs a rubric
safety	A refusal suite scored against a rubric	Whether the model declines what it should

lm-eval and evalscope decide correctness with a rule — an exact match, a parsed answer. judge and safety decide it with a rubric your team wrote. The platform supplies neither the suite’s policy nor the rubric’s content for the last two: both belong to the deployment, and a built-in would be the platform declaring a verdict it has no standing to declare.

Bilingual display copy

Write the manifest in English and carry translations in an i18n: block, at the pack level and per metric:

metrics:
  - key: exact_match,strict-match
    label: Exact match (strict)
    i18n:
      zh:
        label: 精确匹配（严格）
    primary: true

Only display copy is translatable. The id, the suites and the metric keys are the contract and must read the same in every language.

Confirm it worked

sf bench ls                                    # your pack, with its runner and metrics
sf bench run my-bench -m run:run-4f2a91        # score a finished run

Scores land on the console’s Benchmarks page, in the same matrix as every built-in benchmark, so your pack’s numbers are directly comparable with GSM8K and MMLU on the same run.

Externally scored benchmarks

When the scoring happens somewhere the platform cannot reach — a licensed harness, a physical test rig — report the numbers back instead of running them here:

sf bench external create --name my-rig-eval -m run:run-4f2a91
sf bench external submit <EVAL_ID> --scores scores.json

The scores enter the same matrix and are labelled as externally produced. See running benchmarks for the workflow around it.

What ships built in

gsm8k, math, humaneval, mmlu, ceval, ifeval, plus the RTL packs (rtl-repo, rtllm-v2, verilogeval-v2, verilogeval-v2-completion, cvdp).

Those axes are chosen deliberately: math, code, knowledge regression, Chinese, and instruction following. Post-training most often improves one thing while quietly breaking another, and the point of running several is to make that visible on one screen.