Verifiers

The three ways a task’s completion is decided, and the exact contract of each.

manifest.json

{ "verifier": { "kind": "rubric", "ref": "alice/arithmetic-correctness" } }

A verifier decides whether a task was completed. It is always a reference to something that already owns that answer — never something the platform implements.

The three kinds

Kind	ref is	Who owns the judgement
rubric	<owner>/<name> of a Rubric	Your team’s written standard
plugin	module:callable in a kind: environment plugin	Your code
endpoint	An HTTP URL	A service you run

There is deliberately no fourth kind, and in particular no built-in.

Why the platform is never a verifier

The last time this repository grew a built-in scorer, it wrote its own simulation pipeline with its own weights, arrived at a different judgement from the evaluation side, and had to be reverted whole. A verifier living in the control plane would be a second definition of “correct” no matter how small it started, and the two definitions would disagree exactly when it mattered.

Contracts

Whatever the kind, the resolved verifier looks the same from the platform’s side:

VerifyFn = Callable[[Mapping[str, Any], Any], float]

Two arguments: the task, and whatever the harness submitted as the trajectory. One return: a reward.

rubric

{ "verifier": { "kind": "rubric", "ref": "alice/arithmetic-correctness" } }

The platform judge scores the trajectory against the named rubric version. It receives the task’s prompt, the flattened trajectory text, and the task’s reference if there is one.

The trajectory is flattened for you — a string stays a string, a message object contributes its content or text, and a list is joined. Harnesses differ in shape and the verifier contract is about what the model produced, so the folding happens once here rather than in every author’s code.

plugin

{ "verifier": { "kind": "plugin", "ref": "verify:score_arithmetic" } }

verify.py

from typing import Any, Mapping

def score_arithmetic(task: Mapping[str, Any], trajectory: Any) -> float:
    expected = str(task.get("reference", "")).strip()
    answer = str(trajectory).strip().split()[-1] if trajectory else ""
    return 1.0 if answer == expected else 0.0

Imported from a kind: environment plugin the launcher has already vetted through the digest gates. Imported lazily and cached: an import error surfaces on the first task, where it reads as a deployment problem, rather than at startup where it would look like the environment itself is broken.

Must return something float() accepts. Raising is fine and correct — see below.

endpoint

{ "verifier": { "kind": "endpoint", "ref": "https://verifier.corp/score" } }

The platform POSTs JSON and expects a reward back:

Request

{ "task": { "prompt": "...", "reference": "..." }, "trajectory": "..." }

Response

{ "reward": 0.83 }

A response without a reward field, or with a non-numeric one, is an error rather than a zero. The default timeout is 60 seconds.

A failed verification is an error, never a zero

If your verifier cannot answer — the endpoint is down, the module will not import, the rubric is gone — the platform raises. It does not report a reward of 0.0.

This is the single most important rule for anyone writing a verifier, and it is worth being blunt about why: a silently zeroed reward is indistinguishable from an answer that was actually wrong. A training run whose verifier quietly broke at step 400 would keep producing a reward curve, keep looking plausible, and teach the model from noise for the remaining hours of the job.

So in your own verifier code:

def score(task, trajectory):
    if not task.get("reference"):
        raise ValueError("this task has no reference; it cannot be scored")   # correct
        # return 0.0                                                          # wrong
    ...

Errors are wrapped as VerificationError with the reference that failed, so the job log names which verifier stopped and why.

Picking one

Use a rubric

When correctness is a matter of judgement — quality, tone, whether an explanation is sound. Also when you want the same standard to score both training rewards and benchmarks.

Use a plugin

When correctness is computable and the computation is cheap and local: an exact match, a parser, a unit test.

Use an endpoint

When something already decides this — a compiler service, a simulator, an internal grading system. Do not reimplement it in a plugin.