Rubrics

Write down what your team means by a correct answer, then use it as a reward and as a benchmark.

A rubric is your team’s written standard for what counts as a correct answer: named criteria, their weights, and a scale. It is owned, versioned, and referenced as <owner>/<name>.

curl -X PUT https://starforge.your-company.com/api/rubrics/alice/answer-quality \
  -H "Authorization: Bearer $SF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "description": "Scoring customer-support answers.",
    "visibility": "private",
    "scale_min": 0,
    "scale_max": 5,
    "criteria": [
      {"name": "Factually correct", "weight": 3.0,
       "description": "Every claim about the product is true and current."},
      {"name": "Answers the question asked", "weight": 2.0,
       "description": "Addresses the customer'\''s actual problem, not an adjacent one."},
      {"name": "Cites the handbook", "weight": 1.0,
       "description": "Points to the section a human could check."}
    ]
  }'

The console’s Rubrics page does the same thing with a form. Use whichever you prefer — most people draft in the console and version through the API.

Shape

criteria · array · required

Each is {"name": string, "weight": number, "description": string}.

Write the description as an instruction to a careful reader, not a label. “Every claim about the product is true and current” tells a judge what to look for; “Accuracy” does not.

scale_min / scale_max · number

The range a criterion is scored on. 0–5 is a common choice; 0–1 makes the weighted total read as a proportion.

visibility · private | public · default: "private"

private: the owner and administrators can read it. public: everyone reads, the owner still writes.

description · string

What this rubric is for. Shown in listings, and worth writing — a rubric with a vague name and no description is the one somebody duplicates six months later.

Where a rubric is used

The same rubric serves three places, which is the reason it is an object rather than a prompt pasted into two configs:

Used as	How	Result
A training reward	An environment verifier with kind: rubric	Reward per rollout
A benchmark	A benchmark pack with runner: judge and a rubric	A score on the dashboard
A safety evaluation	A pack with runner: safety	Whether the model declines what it should

Because it is one object, the standard your model trains against and the standard it is scored against cannot silently drift apart.

Versions

Every edit increments the version, and the full text of the old version is kept.

curl -H "Authorization: Bearer $SF_TOKEN" \
  https://starforge.your-company.com/api/rubrics/alice/answer-quality/revisions

That matters more than it sounds. rubrics holds the current version and overwrites its criteria on every edit — which would make “versioned” true of the number and false of the content. A run that scored against v2 could not show what v2 actually said once the rubric reached v5. Keeping the revision text is the difference between a version counter and an auditable asset.

curl -H "Authorization: Bearer $SF_TOKEN" \
  https://starforge.your-company.com/api/rubrics/alice/answer-quality/usage

Lists which runs cited which version, and whether as a training reward or as a benchmark score.

Editing a rubric invalidates the judge’s cached scores for it — the cache key carries the owner and the version. Sharpening a criterion does not silently leave old scores in place.

Why rubrics have owners

A globally unique name would freeze every rubric at version one

What counts as a correct answer is each team’s judgement, not the platform’s. If rubric names were global and writing needed an administrator, a researcher who wanted to sharpen one criterion would have to file a request — so they would not, and the rubric would stop at version one.

A rubric stuck at version one is one that training and evaluation both route around by writing their own copy of the scoring logic, which is exactly the drift the object exists to prevent. So ownership follows the dataset model: the owner holds the write permission, visibility decides who else can read.

Writing a rubric that works

• Few criteria, clearly separated. Three well-described criteria beat eight overlapping ones. A judge asked to score “Clarity” and “Readability” separately will produce two copies of the same number.
• Weight by what you would actually trade off. If a factually wrong but beautifully written answer is worthless, the factual criterion needs a weight that says so.
• Describe the failure, not just the success. “Does not invent product features that do not exist” is easier to apply consistently than “Accurate”.
• Version deliberately. Sharpen one criterion at a time so the usage record tells you which change moved the scores.

Confirm it worked

curl -H "Authorization: Bearer $SF_TOKEN" \
  "https://starforge.your-company.com/api/rubrics/resolve?ref=alice/answer-quality"

Resolves the reference the way a job would. If that answers, an environment or benchmark pack can name it.