Manage datasets

Local preprocessing → versioned upload → automatic distribution to the job’s shared cache

sf dataset prepare gsm8k-zh                     # preprocess locally
sf dataset push gsm8k-zh v2 ./out/gsm8k-zh      # upload as version v2
sf submit my-grpo --train-dataset alice/gsm8k-zh@v2 --train-data train.parquet

A platform dataset solves three problems at once: versioning, so training points at an exact version rather than at whatever was on disk; distribution, so a job pulls it into the cluster’s shared cache at startup instead of somebody scp-ing it to every node; and permission, so who may use it is a property of the dataset rather than of the filesystem.

Three steps

1. Preprocess locally

   sf dataset prepare              # list available preprocessing scripts
   sf dataset prepare gsm8k-zh     # run common/data/prepare_gsm8k_zh.py
   

   Preprocessing scripts live by convention at common/data/prepare_*.py and produce a local directory (parquet / jsonl).

2. Versioned upload

   sf dataset push gsm8k-zh v1 ./out/gsm8k-zh        # private (default)
   sf dataset push gsm8k-zh v1 ./out/gsm8k-zh --public   # public for platform-wide reference
   

   Uploads to object storage (MinIO / S3) with streamed chunking + SHA256 verification + resumable upload (rerunning after an interruption skips already-uploaded files). Versions are immutable — a version of the same name cannot be overwritten.

3. Reference in training

   Prefer writing it into the experiment config:

   data:
     train:
       dataset: alice/gsm8k-zh@v2      # <owner>/<name>[@version]; omitting the version takes the latest
       file: train.parquet
   

   Or override temporarily at submission:

   sf submit my-exp --train-dataset alice/gsm8k-zh@v2 --train-data train.parquet
   

   At job startup, the platform pulls the dataset to the cluster shared cache (with integrity verification; corruption triggers an automatic re-pull) and injects the <NAME>_DATA_DIR environment variable; --train-data takes just the relative filename within the dataset.

Writing a dataset card

A dataset with nothing but a name is a black box to everyone except whoever uploaded it — and picking the wrong data costs hundreds of GPU hours. Put the description in a README.md inside the data directory and sf dataset push publishes it along with the data:

# gsm8k-zh

Chinese math problems rewritten from GSM8K, deduplicated by question text.

## Columns

| Column | Type | Description |
| --- | --- | --- |
| `question` | string | The problem statement |
| `answer` | string | Reference solution including the reasoning |

## Provenance

`common/data/prepare_gsm8k_zh.py`; 200 samples spot-checked by hand after translation.

Updating the card later does not require pushing new data:

sf dataset card alice/gsm8k-zh -f README.md

Its first paragraph becomes the one-line description on every list row; the body is rendered in the console’s Card tab.

Management

sf dataset ls                                  # visible datasets (public + your own)
sf dataset visibility gsm8k-zh --public        # change visibility (owner or admin)

Every row of the console Data → Datasets tab (/data) carries the description, latest version, file count, size, formats and update time — no need to open datasets one by one to compare them. Opening one gives you four tabs:

Tab	The question it answers
Card	What this data is, which columns it has, where it came from
Files	What this version contains, how large, with which checksums
Preview	Column types, how many empties, roughly how long the text is (is max_seq_len enough?)
Usage	How many jobs reference it — i.e. whether it is safe to roll or clean up

Parquet

Parquet is the most common training-data format here, and its preview works out of the box (pyarrow ships with the control plane; it is not an optional extra). It gives you two things jsonl and csv cannot:

• The total row count, read from the footer at the end of the file — a few KB. jsonl and csv would need a full scan, so those formats show ”—” here rather than a guessed number.
• Declared column types (int64 / string / timestamp[us] …). More reliable than types inferred from the first few rows: a sample that happens to be all integers does not make the column an integer column, and an all-null sample does not mean the column has no type.

The platform reads only the footer and the first row group, with a cap on total bytes. Files with very large row groups (64–128 MB per group is a common writer setting) yield no sample rows; columns and row count are still shown, with a note where the samples would be — those two cost only the footer’s few KB and should not be thrown away along with the samples.

Column lengths in the preview are measured before truncation. Sample cells are cut to 160 characters on their way to the browser, so an average computed from the samples would always equal the truncation cap — a number you cannot size max_seq_len with.

“Referenced by N jobs” is platform-wide; the job list below it still only shows jobs you can already see.

The dataset manifest (with presigned URLs) is generated at submission and written into the job package — the job side needs no object storage credentials and no access to the console; one fewer runtime dependency means one fewer failure point.

Scanning for sensitive values

After a version is published, the reconcile pass scans it once and the result appears on the dataset page.

It produces a report and never rewrites what you uploaded. Every comparable tool draws this line — Cloud DLP’s inspect and deidentify are two calls you make separately, Macie reports and does not touch the object — and the reason is sharper here. A reflow buffer is something the platform produced, so rewriting one is its business. A training set is your material: rewriting it silently changes what a model learns, cannot be undone, and leaves nobody able to explain a result afterwards.

To clean it, use a Processing Run: read the material, run your own treatment, publish a new version. Your code, in the platform, with provenance.

It samples, and says how much

A hundred-gigabyte dataset cannot be read on the way to a submission. The scan reads bounded windows and reports how many rows it read. A sampled scan can never say “clean” — only “nothing in what I looked at” — and a report that omits its sample size is one somebody will eventually quote as proof.

Windows are spread across each file rather than taken from the head. A bad export is appended and a test fixture sits at the top, so a head-only sample systematically misses the case most worth catching.

Confidence, not a boolean

Every rule carries a severity. api_key, private_key and a checksum-valid identity number are high; email and phone are medium; the generic long-token rule is low — it hits every base64 blob and git SHA in a code corpus.

So the blocking policy (FORGE_DATA_SCAN_POLICY=block) only fires on high. A gate that refuses work over low-confidence heuristics is one somebody turns off within a week, taking the useful half with it.

Your own terms

FORGE_DATA_SCAN_DENY_TERMS adds terms this deployment treats as sensitive (a customer number, a case id); FORGE_DATA_SCAN_ALLOW_TERMS suppresses known false positives.

Terms, never patterns. Pasting a regex into a settings box is how a config change becomes a catastrophic backtrack that hangs a worker — Python’s re has no timeout. The platform escapes and compiles them. Presidio’s deny lists work the same way, for the same reason.

Volumes (not train/val)

Material a job reads but does not train on — internal documents, reference PDFs, anything a script should see — lives in a Volume: a directory you create in the console and drop files into. It has no versions; its current contents are its contents.

data:
  volumes:
    - alice/legal-docs

Each volume appears inside the container at $VOLUMES_DIR/<name>, read-only. Nobody can download its files unless the owner turns that on, and there is no preview at any setting. The console’s Volumes page is the reference.

Relationship to HuggingFace datasets

Writing an HF dataset id directly in the config (e.g. nvidia/OpenMathInstruct-2) also works: the platform runs an HF preflight at submission (gated datasets verify your authorization). Platform datasets suit internal data and preprocessed derived data — no need to push to HF, and permissions stay on the internal network.