> ## Documentation Index > Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Verifiers > The three ways a task's completion is decided, and the exact contract of each. ```json manifest.json theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} { "verifier": { "kind": "rubric", "ref": "alice/arithmetic-correctness" } } ``` A verifier decides whether a task was completed. It is always a **reference** to something that already owns that answer — never something the platform implements. ## The three kinds | Kind | `ref` is | Who owns the judgement | | ---------- | -------------------------------------------------- | ---------------------------- | | `rubric` | `/` of a [Rubric](/en/extend/rubrics) | Your team's written standard | | `plugin` | `module:callable` in a `kind: environment` plugin | Your code | | `endpoint` | An HTTP URL | A service you run | There is deliberately no fourth kind, and in particular no built-in. The last time this repository grew a built-in scorer, it wrote its own simulation pipeline with its own weights, arrived at a different judgement from the evaluation side, and had to be reverted whole. A verifier living in the control plane would be a second definition of "correct" no matter how small it started, and the two definitions would disagree exactly when it mattered. ## Contracts Whatever the kind, the resolved verifier looks the same from the platform's side: ```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} VerifyFn = Callable[[Mapping[str, Any], Any], float] ``` Two arguments: the task, and whatever the harness submitted as the trajectory. One return: a reward. ```json theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} { "verifier": { "kind": "rubric", "ref": "alice/arithmetic-correctness" } } ``` The platform judge scores the trajectory against the named rubric version. It receives the task's `prompt`, the flattened trajectory text, and the task's `reference` if there is one. The trajectory is flattened for you — a string stays a string, a message object contributes its `content` or `text`, and a list is joined. Harnesses differ in shape and the verifier contract is about what the model produced, so the folding happens once here rather than in every author's code. ```json theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} { "verifier": { "kind": "plugin", "ref": "verify:score_arithmetic" } } ``` ```python verify.py theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} from typing import Any, Mapping def score_arithmetic(task: Mapping[str, Any], trajectory: Any) -> float: expected = str(task.get("reference", "")).strip() answer = str(trajectory).strip().split()[-1] if trajectory else "" return 1.0 if answer == expected else 0.0 ``` Imported from a `kind: environment` plugin the launcher has already vetted through the digest gates. Imported lazily and cached: an import error surfaces on the first task, where it reads as a deployment problem, rather than at startup where it would look like the environment itself is broken. Must return something `float()` accepts. Raising is fine and correct — see below. ```json theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} { "verifier": { "kind": "endpoint", "ref": "https://verifier.corp/score" } } ``` The platform POSTs JSON and expects a reward back: ```json Request theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} { "task": { "prompt": "...", "reference": "..." }, "trajectory": "..." } ``` ```json Response theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} { "reward": 0.83 } ``` A response without a `reward` field, or with a non-numeric one, is an error rather than a zero. The default timeout is 60 seconds. ## A failed verification is an error, never a zero If your verifier cannot answer — the endpoint is down, the module will not import, the rubric is gone — the platform raises. It does not report a reward of `0.0`. This is the single most important rule for anyone writing a verifier, and it is worth being blunt about why: a silently zeroed reward is **indistinguishable from an answer that was actually wrong**. A training run whose verifier quietly broke at step 400 would keep producing a reward curve, keep looking plausible, and teach the model from noise for the remaining hours of the job. So in your own verifier code: ```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} def score(task, trajectory): if not task.get("reference"): raise ValueError("this task has no reference; it cannot be scored") # correct # return 0.0 # wrong ... ``` Errors are wrapped as `VerificationError` with the reference that failed, so the job log names which verifier stopped and why. ## Picking one When correctness is a matter of judgement — quality, tone, whether an explanation is sound. Also when you want the same standard to score both training rewards and benchmarks. When correctness is computable and the computation is cheap and local: an exact match, a parser, a unit test. When something already decides this — a compiler service, a simulator, an internal grading system. Do not reimplement it in a plugin.