sf annotate

Manage preference annotation

sf annotate status                                   # is the integration configured
sf annotate push run-4f2a91 --limit 200              # send answers out for labelling
sf annotate pull run-4f2a91 --dataset alice/prefs    # bring judgements back as a dataset

Human preference annotation, through an Argilla instance the deployment configures. The loop is: take a run’s generated answers out for people to judge, then bring the judgements back as a versioned dataset you can train DPO on.

The platform is the transport and the governance here, not the annotation tool. sf annotate status tells you whether an Argilla instance is wired up at all — start there, because everything else fails confusingly without one.

See Annotation for the workflow around these commands.

sf annotate

Manage preference annotation

sf annotate <subcommand>

Command	Description
sf annotate pull	Pull annotations back as a dataset
sf annotate push	Send answers out for annotation
sf annotate status	Check the Argilla integration

sf annotate pull

Pull annotations back as a dataset

sf annotate pull <DATASET> [options]

Argument	Description
DATASET	Argilla dataset name (required)

Option	Description
--into	Land them as a new version of this platform dataset; omit to only print the statistics
--version, -v	Dataset version; defaults to the date

sf annotate push

Send answers out for annotation

sf annotate push <RUN_A> <RUN_B> [options]

Argument	Description
RUN_A	Run id of the first training run (required)
RUN_B	Run id of the second training run (required)

Option	Description
--dataset, -d	Argilla dataset name; synthesised when empty
--limit <int range>	How many pairs to push at most (default 200)

sf annotate status

Check the Argilla integration

sf annotate status

No options beyond --help.