Collect preference annotations

Pair two runs’ answers → annotate in Argilla → pull back as DPO training data

LLM-as-judge scores automatically, but some judgements only a person can make. This chain turns those judgements back into training data.

Which part the platform does

The annotation UI, permissions, progress tracking and inter-annotator agreement are all Argilla’s job. Rebuilding that is another product’s worth of work, and the platform does not attempt it.

What the platform does is the two ends Argilla cannot:

1. Out: pairing

   “On the same prompts, what did model A and model B each answer” — only the platform can answer that, because it holds both runs’ validation samples.

2. (Middle: Argilla)

   A person picks “A is better / B is better / can’t tell” record by record.

3. Back: into training data

   Annotated preferences come back as prompt / chosen / rejected DPO rows and land as a new version of a platform dataset, ready for the next run to reference.

Using it

sf annotate status                              # confirm the integration is configured
sf annotate push <RUN_A> <RUN_B> --limit 200    # pair and push for annotation
# …people annotate in Argilla…
sf annotate pull forge-<RUN_A>-vs-<RUN_B> --into qa-preference

The last step lands the preference data as a new version of qa-preference, through exactly the same path as sf dataset push — same immutable versions, fingerprint and quality report. Then:

# next DPO experiment's config
data:
  train:
    dataset: alice/qa-preference@20260828-120000

Deliberate choices

Two runs are required. Preference annotation asks “which of A and B is better”; a single run’s samples can only be rated, not compared — and DPO needs an ordering.

Aligned by prompt, not by sample index. Two validation passes do not guarantee the same sampling order; aligning by index would pair A’s third question with B’s third question, and the resulting preferences would mean nothing.

A/B order is not tied to the run. If A always came from the same run, annotators would quickly recognise the style and develop a systematic bias. Order is derived from the prompt — stable across re-opens, but independent of which run is which.

Pairs with identical answers are dropped. Asking a person to choose between two identical answers only spends annotation budget and produces random noise.

“Can’t tell” produces no training data. DPO needs an ordering; forcing ties into it gives the model a fabricated signal. The number skipped is printed, so it never looks like lost data.

Admins: configuration

FORGE_ARGILLA_URL: "http://argilla.internal:6900"
FORGE_ARGILLA_API_KEY: "<argilla API key>"
FORGE_ARGILLA_WORKSPACE: "argilla"     # default; the workspace must exist in Argilla already

The Argilla address is deployment configuration, at the same trust level as the database and object-storage addresses — on an internal deployment it lives on a private network by definition. So the webhook rule of “reject private addresses” is deliberately not applied here: that rule exists for user-supplied addresses, and copying it over would make this integration unable to connect on exactly the networks where it is deployed.