sf dataset

Datasets: local preprocessing, versioned uploads, visibility management

See Dataset Management for the workflow walkthrough.

sf dataset prepare

sf dataset prepare              # Without arguments: list available preprocessing scripts
sf dataset prepare <name>       # Run common/data/prepare_<name>.py

Discovers common/data/prepare_*.py by convention and runs it locally to produce the data directory.

sf dataset push

sf dataset push <DATASET> <VERSION> <PATH> [--public] [--readme README.md]

Argument	Description
DATASET	Dataset name (placed under your own namespace); admins can use <owner>/<name> to work across namespaces
VERSION	Version, e.g. v1 / 20260812; immutable — a version with the same name cannot be overwritten
PATH	Local directory
--public	Make it public on first creation (referenceable platform-wide)
--readme	Description file; defaults to <PATH>/README.md when present

Upload features: streaming chunks, per-file SHA256, resumable upload (rerunning after an interruption automatically skips files already uploaded).

Drop a README.md in the directory and push publishes it as the dataset card (below).

sf dataset card

sf dataset card <DATASET>                     # Print the current card
sf dataset card <DATASET> -f README.md        # Update the card from this markdown file

The card belongs to the dataset, not to one version: fixing the description does not force you to push a new version of the data. Its first paragraph becomes the one-line description shown on every row of sf dataset ls and of the console list.

sf dataset check — catch contamination before you push

sf dataset check ./out/my-train --against ./eval/gsm8k-test.jsonl
sf dataset check ./out/my-train --against ./eval/test.jsonl --fields question

What this guards against is accidentally training on the test set: slicing train/test from the same public dataset, or picking up someone’s cleaned mixture that happens to contain GSM8K. The damage is a score that is inflated and nobody notices — the model scores 90 on questions it memorised, you ship on that number, and production does not match.

It reports which rows, because the action you need to take is “delete these”:

  !  3 / 12000 training rows overlap the eval set (0.0%)
     · Natalia sold clips to 48 of her friends in April, and then she sold…
  →  These rows inflate your eval scores. Remove them, or use a different eval set.

Exit code is 1 when there is overlap — drop it into CI and a contaminated push gets stopped.

--fields narrows it to the question text: the same word appearing in an answer is not contamination.

Matching uses normalised 13-grams (the window used in the GPT-3 and PaLM contamination analyses). Case, punctuation and full-width/half-width forms do not affect the verdict; Chinese is shingled by character, since splitting on spaces would turn a whole sentence into one token and find nothing.

sf dataset quality — is this data clean in itself

sf dataset quality ./out/my-train

12000 records
  duplicates   842 (7.0%), of which 310 byte-identical
  empty        0 (0.0%)
  text length  median 180 · p95 1400 · longest 9000 characters
  fields       answer, prompt
  !  duplicate rate 7.0% is high
  →  the model will overfit to the repeated samples. Deduplicate before pushing,
     or confirm this is a deliberate sampling weight.

sf dataset push computes this automatically and stores it with the version, where the console shows it at the top of the dataset page. This command is for looking before you push.

Four numbers, each guarding against a different failure:

Metric	What it guards against
Duplicate rate	A training set that is 30% repeats overfits to those repeats, and nothing else on the platform would ever show you that
Length p95	Whether max_seq_len is enough is decided by p95, not the mean. The mean is dragged down by short samples, and 5% of the data gets silently truncated
Empty records	Wholly empty samples dilute the gradient signal
Field drift	Half the records have an answer field and half do not; the half that does not reads None during training

It counts only what a machine can definitively count. “Is this a good answer” is a judgement made by a person or a model, and belongs to preference annotation instead.

Near-duplicates use a small MinHash. Template-generated data — fifty records differing by one number — is reported as duplicated, and that is not a false positive. It is precisely the kind of thing this report exists to say out loud.

Contamination fingerprints (comparing across the wire)

sf dataset push also computes an n-gram fingerprint (a sampled set of hashes, a few hundred KB) and uploads it with the version. Push already reads every file once to compute sha256, so shingling costs almost nothing there — whereas computing it later means pulling gigabytes back out of object storage, which is expensive enough that nobody ever does it.

With fingerprints on both sides, comparing two datasets is just a set intersection:

# Console or API: how much does the training set overlap the eval set?
GET /api/datasets/<owner>/<name>/contamination?against=<owner>/<eval-name>

For data that does not live on the platform (a benchmark’s test split, say):

sf dataset fingerprint ./gsm8k-test.jsonl -o gsm8k.fingerprint.json

Fingerprint comparison is an estimate: sampling only allows false negatives, and a very small dataset may have nothing sampled at all. Use sf dataset check for small sets — it names rows, the fingerprint only gives a ratio. Versions pushed before this feature existed have no fingerprint, and the endpoint says so explicitly rather than returning a false “no contamination”.

sf dataset ls / visibility

sf dataset ls                              # Visible datasets (public + your own)
sf dataset ls <DATASET> [-v <VERSION>]     # File manifest of one version
sf dataset visibility <DATASET> --public   # Change visibility (owner or admin)
sf dataset visibility <DATASET> --private

Each ls row carries the latest version, file count, size, formats and update time:

alice/gsm8k-zh                   [public] v2 · 3 files · 39.1 MB · parquet · 3 days ago
                                 Chinese math problems rewritten from GSM8K, deduplicated.

Referencing in Training

sf submit my-exp --train-dataset alice/gsm8k-zh@v2 --train-data train.parquet

On job start the dataset is automatically distributed to the cluster’s shared cache and <NAME>_DATA_DIR is injected; it is recommended to pin the reference in the experiment config under data.train.dataset.

sf dataset

Manage datasets

sf dataset <subcommand>

Command	Description
sf dataset card	Show/update a dataset description (README.md)
sf dataset check	Check training data for evaluation overlap
sf dataset fingerprint	Compute a dataset fingerprint
sf dataset ls	List the visible datasets
sf dataset prepare	Preprocess a dataset locally
sf dataset push	Upload one dataset version
sf dataset quality	Produce a dataset quality report
sf dataset visibility	Change a dataset’s visibility (owner or admin)

sf dataset card

Show/update a dataset description (README.md)

sf dataset card <DATASET> [options]

Argument	Description
DATASET	Dataset id (<owner>/<name>) (required)

Option	Description
--file, -f	Update the description from this markdown file; omit to print the current one

sf dataset check

Check training data for evaluation overlap

sf dataset check <PATH> [options]

Argument	Description
PATH	Training data directory or file (jsonl / json / csv / parquet) (required)

Option	Description
--against, -a	Evaluation set directory or file; the question is whether its questions leaked into training (required)
--fields	Only these fields (comma separated), e.g. question,prompt. Defaults to every string field

sf dataset fingerprint

Compute a dataset fingerprint

sf dataset fingerprint <PATH> [options]

Argument	Description
PATH	Data directory or file (required)

Option	Description
--out, -o	Write it to this file; omit to print a summary
--fields	Only these fields (comma separated)

sf dataset ls

List the visible datasets

sf dataset ls [DATASET] [options]

Argument	Description
DATASET	Dataset id (<owner>/<name>); omit to list everything visible

Option	Description
--version, -v	Show one version’s file list; defaults to the latest

sf dataset prepare

Preprocess a dataset locally

sf dataset prepare [DATASET]

Argument	Description
DATASET	Dataset name; leave empty to list the options

No options beyond --help.

sf dataset push

Upload one dataset version

sf dataset push <DATASET> <VERSION> <PATH> [options]

Argument	Description
DATASET	Dataset name (lands in your own namespace), or a full <owner>/<name> (an admin can cross namespaces) (required)
VERSION	Version, e.g. v1 / 20260812 (required)
PATH	Local directory (required)

Option	Description
--public	Make it public on first creation (anyone can reference it in training)
--readme	Dataset description (markdown); defaults to the README.md in the directory

sf dataset quality

Produce a dataset quality report

sf dataset quality <PATH> [options]

Argument	Description
PATH	Data directory or file (required)

Option	Description
--fields	Only these fields (comma separated)

sf dataset visibility

Change a dataset’s visibility (owner or admin)

sf dataset visibility <DATASET> [options]

Argument	Description
DATASET	Dataset id (<owner>/<name>) (required)

Option	Description
--public	Make it public
--private	Make it private