> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark packs

> Declare an evaluation set: what to run, which numbers to read, and how to read them.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf bench new my-bench            # scaffold a pack
sf plugin publish ./my-bench     # publish it
sf bench ls                      # it is now in the catalog
sf bench run my-bench -m run:run-4f2a91
```

A benchmark pack answers three questions in YAML, and contains no code: **what to run**, **which
numbers to read**, and **how to run it by default**. The thing that actually executes is a runner
that ships with the SDK.

## Layout

```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
my-bench/
├── plugin.yaml       # kind: benchmark
├── benchmark.yaml    # the declaration
└── README.md
```

## benchmark.yaml

```yaml benchmark.yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
schema: forge/benchmark/v1
name: gsm8k
title: GSM8K · Grade-school math word problems
category: math
summary: >
  8.5K grade-school math word problems requiring 2-8 reasoning steps. The usual entry
  benchmark for math post-training: a clean signal, fast to run, and well correlated with
  the RL reward curve, which makes it a good spot check mid-training.

runner: lm-eval
suites: [gsm8k]
num_fewshot: 5
batch_size: auto
keywords: [math, reasoning, cot]

metrics:
  - key: exact_match,strict-match
    label: Exact match (strict)
    primary: true
  - key: exact_match,flexible-extract
    label: Exact match (flexible)

i18n:
  zh:
    title: GSM8K · 小学数学应用题
    summary: >
      8.5K 道小学数学应用题，需要 2-8 步推理。
```

### Fields

<ParamField path="name" type="string" required>
  Leaf name. A published pack is referenced as `<owner>.<name>`; built-ins keep the bare name.
</ParamField>

<ParamField path="runner" type="lm-eval | evalscope | rtl | judge | safety" required>
  Which harness executes the evaluation. See the table below.
</ParamField>

<ParamField path="suites" type="string[]" required>
  The suite names the runner understands. For `lm-eval` these are its task names.
</ParamField>

<ParamField path="category" type="string" default="general">
  Dashboard grouping — `math`, `code`, `knowledge`, `chinese`, `instruction`, and so on.
</ParamField>

<ParamField path="num_fewshot, batch_size, limit, extra_args" type="mixed">
  Defaults for running it. `batch_size` defaults to `auto`, `limit` caps the number of samples. A
  user can override all of these at submission; what you declare is what happens when they do not.

  This is the point of the pack: nobody should have to remember that GSM8K wants 5-shot or that
  HumanEval needs `--confirm_run_unsafe_code`.
</ParamField>

<ParamField path="rubric" type="<owner>/<name>">
  Only read by the `judge` runner, and for that runner it *is* the benchmark's definition. Swap the
  rubric and it is a different benchmark — which is why it is a declared field rather than a flag
  buried in `extra_args` that nobody can see from the catalog.
</ParamField>

### Metrics

Each entry says which number to pull out of the runner's report and how to present it.

<ParamField path="key" type="string" required>
  The raw metric name the runner emits, verbatim — `exact_match,strict-match` for lm-eval. This is
  the machine-facing name and it is not translatable.
</ParamField>

<ParamField path="label" type="string">
  What the dashboard column says. Defaults to `key`, which is fine for a machine and poor for a
  human.
</ParamField>

<ParamField path="direction" type="higher | lower | neutral" default="higher">
  Which way is better. The comparison view colours regressions by this.
</ParamField>

<ParamField path="ratio" type="bool" default="true">
  Whether the value is a 0–1 proportion, so the dashboard renders it as a percentage.
</ParamField>

<ParamField path="primary" type="bool" default="false">
  The metric cross-run comparison defaults to. Exactly one should be primary; if none is, the first
  is used.
</ParamField>

<Tip>
  Declare more than one metric where the difference between them is diagnostic. GSM8K's strict and
  flexible matches are the standard example: a low strict score with a high flexible score means the
  model can do the arithmetic but is not formatting the answer as asked — a prompt problem, not a
  capability problem. One number would hide that.
</Tip>

## The five runners

| Runner      | What it runs                                        | Use it for                                                  |
| ----------- | --------------------------------------------------- | ----------------------------------------------------------- |
| `lm-eval`   | EleutherAI lm-evaluation-harness                    | Academic standard benchmarks. The default choice            |
| `evalscope` | ModelScope evalscope                                | Chinese benchmarks — C-Eval, CMMLU — which it covers better |
| `rtl`       | An RTL/hardware harness pinned in the evalkit image | Verilog and hardware-design suites                          |
| `judge`     | Generation with vLLM, scored by the platform judge  | Anything a rule cannot score. Needs a `rubric`              |
| `safety`    | A refusal suite scored against a rubric             | Whether the model declines what it should                   |

<Note>
  `lm-eval` and `evalscope` decide correctness with a rule — an exact match, a parsed answer. `judge`
  and `safety` decide it with a rubric your team wrote. The platform supplies neither the suite's
  policy nor the rubric's content for the last two: both belong to the deployment, and a built-in
  would be the platform declaring a verdict it has no standing to declare.
</Note>

## Bilingual display copy

Write the manifest in English and carry translations in an `i18n:` block, at the pack level and per
metric:

```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
metrics:
  - key: exact_match,strict-match
    label: Exact match (strict)
    i18n:
      zh:
        label: 精确匹配（严格）
    primary: true
```

Only display copy is translatable. The id, the suites and the metric keys are the contract and must
read the same in every language.

## Confirm it worked

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf bench ls                                    # your pack, with its runner and metrics
sf bench run my-bench -m run:run-4f2a91        # score a finished run
```

Scores land on the console's Benchmarks page, in the same matrix as every built-in benchmark, so
your pack's numbers are directly comparable with GSM8K and MMLU on the same run.

## Externally scored benchmarks

When the scoring happens somewhere the platform cannot reach — a licensed harness, a physical test
rig — report the numbers back instead of running them here:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf bench external create --name my-rig-eval -m run:run-4f2a91
sf bench external submit <EVAL_ID> --scores scores.json
```

The scores enter the same matrix and are labelled as externally produced. See
[running benchmarks](/en/guides/benchmarks) for the workflow around it.

## What ships built in

`gsm8k`, `math`, `humaneval`, `mmlu`, `ceval`, `ifeval`, plus the RTL packs (`rtl-repo`,
`rtllm-v2`, `verilogeval-v2`, `verilogeval-v2-completion`, `cvdp`).

Those axes are chosen deliberately: math, code, knowledge regression, Chinese, and instruction
following. Post-training most often improves one thing while quietly breaking another, and the point
of running several is to make that visible on one screen.