> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Collect preference annotations

> Pair two runs' answers → annotate in Argilla → pull back as DPO training data

LLM-as-judge scores automatically, but some judgements only a person can make. This chain turns
those judgements back into training data.

## Which part the platform does

The annotation UI, permissions, progress tracking and inter-annotator agreement are all
[Argilla](https://argilla.io)'s job. Rebuilding that is another product's worth of work, and the
platform does not attempt it.

What the platform does is the two ends Argilla cannot:

<Steps>
  <Step title="Out: pairing">
    "On the same prompts, what did model A and model B each answer" — only the platform can answer
    that, because it holds both runs' validation samples.
  </Step>

  <Step title="(Middle: Argilla)">
    A person picks "A is better / B is better / can't tell" record by record.
  </Step>

  <Step title="Back: into training data">
    Annotated preferences come back as `prompt / chosen / rejected` DPO rows and land as a new
    version of a platform dataset, ready for the next run to reference.
  </Step>
</Steps>

## Using it

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf annotate status                              # confirm the integration is configured
sf annotate push <RUN_A> <RUN_B> --limit 200    # pair and push for annotation
# …people annotate in Argilla…
sf annotate pull forge-<RUN_A>-vs-<RUN_B> --into qa-preference
```

The last step lands the preference data as a new version of `qa-preference`, through exactly the
same path as `sf dataset push` — same immutable versions, fingerprint and quality report. Then:

```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# next DPO experiment's config
data:
  train:
    dataset: alice/qa-preference@20260828-120000
```

## Deliberate choices

**Two runs are required.** Preference annotation asks "which of A and B is better"; a single run's
samples can only be rated, not compared — and DPO needs an ordering.

**Aligned by prompt, not by sample index.** Two validation passes do not guarantee the same
sampling order; aligning by index would pair A's third question with B's third question, and the
resulting preferences would mean nothing.

**A/B order is not tied to the run.** If A always came from the same run, annotators would quickly
recognise the style and develop a systematic bias. Order is derived from the prompt — stable across
re-opens, but independent of which run is which.

**Pairs with identical answers are dropped.** Asking a person to choose between two identical
answers only spends annotation budget and produces random noise.

**"Can't tell" produces no training data.** DPO needs an ordering; forcing ties into it gives the
model a fabricated signal. The number skipped is printed, so it never looks like lost data.

## Admins: configuration

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
FORGE_ARGILLA_URL: "http://argilla.internal:6900"
FORGE_ARGILLA_API_KEY: "<argilla API key>"
FORGE_ARGILLA_WORKSPACE: "argilla"     # default; the workspace must exist in Argilla already
```

<Note>
  The Argilla address is **deployment configuration**, at the same trust level as the database and
  object-storage addresses — on an internal deployment it lives on a private network by definition.
  So the webhook rule of "reject private addresses" is deliberately not applied here: that rule
  exists for **user-supplied** addresses, and copying it over would make this integration unable to
  connect on exactly the networks where it is deployed.
</Note>