> ## Documentation Index > Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # sf dataset > Datasets: local preprocessing, versioned uploads, visibility management See [Dataset Management](/en/guides/datasets) for the workflow walkthrough. ## sf dataset prepare ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset prepare # Without arguments: list available preprocessing scripts sf dataset prepare # Run common/data/prepare_.py ``` Discovers `common/data/prepare_*.py` by convention and runs it locally to produce the data directory. ## sf dataset push ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset push [--public] [--readme README.md] ``` | Argument | Description | | ---------- | --------------------------------------------------------------------------------------------------------- | | `DATASET` | Dataset name (placed under your own namespace); admins can use `/` to work across namespaces | | `VERSION` | Version, e.g. `v1` / `20260812`; **immutable** — a version with the same name cannot be overwritten | | `PATH` | Local directory | | `--public` | Make it public on first creation (referenceable platform-wide) | | `--readme` | Description file; defaults to `/README.md` when present | Upload features: streaming chunks, per-file SHA256, **resumable upload** (rerunning after an interruption automatically skips files already uploaded). Drop a `README.md` in the directory and `push` publishes it as the **dataset card** (below). ## sf dataset card ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset card # Print the current card sf dataset card -f README.md # Update the card from this markdown file ``` The card belongs to the dataset, not to one version: fixing the description does not force you to push a new version of the data. Its first paragraph becomes the one-line description shown on every row of `sf dataset ls` and of the console list. ## sf dataset check — catch contamination before you push ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset check ./out/my-train --against ./eval/gsm8k-test.jsonl sf dataset check ./out/my-train --against ./eval/test.jsonl --fields question ``` What this guards against is **accidentally training on the test set**: slicing train/test from the same public dataset, or picking up someone's cleaned mixture that happens to contain GSM8K. The damage is a score that is inflated and nobody notices — the model scores 90 on questions it memorised, you ship on that number, and production does not match. It reports **which rows**, because the action you need to take is "delete these": ```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} ! 3 / 12000 training rows overlap the eval set (0.0%) · Natalia sold clips to 48 of her friends in April, and then she sold… → These rows inflate your eval scores. Remove them, or use a different eval set. ``` Exit code is 1 when there is overlap — drop it into CI and a contaminated push gets stopped. `--fields` narrows it to the question text: the same word appearing in an answer is not contamination. Matching uses normalised 13-grams (the window used in the GPT-3 and PaLM contamination analyses). Case, punctuation and full-width/half-width forms do not affect the verdict; Chinese is shingled by character, since splitting on spaces would turn a whole sentence into one token and find nothing. ## sf dataset quality — is this data clean in itself ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset quality ./out/my-train ``` ```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} 12000 records duplicates 842 (7.0%), of which 310 byte-identical empty 0 (0.0%) text length median 180 · p95 1400 · longest 9000 characters fields answer, prompt ! duplicate rate 7.0% is high → the model will overfit to the repeated samples. Deduplicate before pushing, or confirm this is a deliberate sampling weight. ``` `sf dataset push` computes this automatically and stores it with the version, where the console shows it at the top of the dataset page. This command is for looking before you push. Four numbers, each guarding against a different failure: | Metric | What it guards against | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Duplicate rate** | A training set that is 30% repeats overfits to those repeats, and nothing else on the platform would ever show you that | | **Length p95** | Whether `max_seq_len` is enough is decided by p95, not the mean. The mean is dragged down by short samples, and 5% of the data gets silently truncated | | **Empty records** | Wholly empty samples dilute the gradient signal | | **Field drift** | Half the records have an `answer` field and half do not; the half that does not reads `None` during training | It counts only what a machine can definitively count. "Is this a good answer" is a judgement made by a person or a model, and belongs to [preference annotation](/en/guides/annotation) instead. Near-duplicates use a small MinHash. Template-generated data — fifty records differing by one number — is reported as duplicated, and **that is not a false positive**. It is precisely the kind of thing this report exists to say out loud. ## Contamination fingerprints (comparing across the wire) `sf dataset push` also computes an **n-gram fingerprint** (a sampled set of hashes, a few hundred KB) and uploads it with the version. Push already reads every file once to compute sha256, so shingling costs almost nothing there — whereas computing it later means pulling gigabytes back out of object storage, which is expensive enough that nobody ever does it. With fingerprints on both sides, comparing two datasets is just a set intersection: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} # Console or API: how much does the training set overlap the eval set? GET /api/datasets///contamination?against=/ ``` For data that does not live on the platform (a benchmark's test split, say): ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset fingerprint ./gsm8k-test.jsonl -o gsm8k.fingerprint.json ``` Fingerprint comparison is an **estimate**: sampling only allows false negatives, and a very small dataset may have nothing sampled at all. Use `sf dataset check` for small sets — it names rows, the fingerprint only gives a ratio. Versions pushed before this feature existed have no fingerprint, and the endpoint says so explicitly rather than returning a false "no contamination". ## sf dataset ls / visibility ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset ls # Visible datasets (public + your own) sf dataset ls [-v ] # File manifest of one version sf dataset visibility --public # Change visibility (owner or admin) sf dataset visibility --private ``` Each `ls` row carries the latest version, file count, size, formats and update time: ```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} alice/gsm8k-zh [public] v2 · 3 files · 39.1 MB · parquet · 3 days ago Chinese math problems rewritten from GSM8K, deduplicated. ``` ## Referencing in Training ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf submit my-exp --train-dataset alice/gsm8k-zh@v2 --train-data train.parquet ``` On job start the dataset is automatically distributed to the cluster's shared cache and `_DATA_DIR` is injected; it is recommended to pin the reference in the experiment config under `data.train.dataset`. ## `sf dataset` Manage datasets ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset ``` | Command | Description | | ------------------------ | ---------------------------------------------- | | `sf dataset card` | Show/update a dataset description (README.md) | | `sf dataset check` | Check training data for evaluation overlap | | `sf dataset fingerprint` | Compute a dataset fingerprint | | `sf dataset ls` | List the visible datasets | | `sf dataset prepare` | Preprocess a dataset locally | | `sf dataset push` | Upload one dataset version | | `sf dataset quality` | Produce a dataset quality report | | `sf dataset visibility` | Change a dataset's visibility (owner or admin) | ### `sf dataset card` Show/update a dataset description (README.md) ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset card [options] ``` | Argument | Description | | --------- | -------------------------------------------- | | `DATASET` | Dataset id (\/\) (**required**) | | Option | Description | | -------------- | ----------------------------------------------------------------------------- | | `--file`, `-f` | Update the description from this markdown file; omit to print the current one | ### `sf dataset check` Check training data for evaluation overlap ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset check [options] ``` | Argument | Description | | -------- | ----------------------------------------------------------------------------- | | `PATH` | Training data directory or file (jsonl / json / csv / parquet) (**required**) | | Option | Description | | ----------------- | ----------------------------------------------------------------------------------------------------------- | | `--against`, `-a` | Evaluation set directory or file; the question is whether its questions leaked into training (**required**) | | `--fields` | Only these fields (comma separated), e.g. question,prompt. Defaults to every string field | ### `sf dataset fingerprint` Compute a dataset fingerprint ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset fingerprint [options] ``` | Argument | Description | | -------- | ------------------------------------- | | `PATH` | Data directory or file (**required**) | | Option | Description | | ------------- | ---------------------------------------------- | | `--out`, `-o` | Write it to this file; omit to print a summary | | `--fields` | Only these fields (comma separated) | ### `sf dataset ls` List the visible datasets ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset ls [DATASET] [options] ``` | Argument | Description | | --------- | -------------------------------------------------------------- | | `DATASET` | Dataset id (\/\); omit to list everything visible | | Option | Description | | ----------------- | ---------------------------------------------------- | | `--version`, `-v` | Show one version's file list; defaults to the latest | ### `sf dataset prepare` Preprocess a dataset locally ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset prepare [DATASET] ``` | Argument | Description | | --------- | --------------------------------------------- | | `DATASET` | Dataset name; leave empty to list the options | No options beyond `--help`. ### `sf dataset push` Upload one dataset version ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset push [options] ``` | Argument | Description | | --------- | --------------------------------------------------------------------------------------------------------------------- | | `DATASET` | Dataset name (lands in your own namespace), or a full \/\ (an admin can cross namespaces) (**required**) | | `VERSION` | Version, e.g. v1 / 20260812 (**required**) | | `PATH` | Local directory (**required**) | | Option | Description | | ---------- | -------------------------------------------------------------------------- | | `--public` | Make it public on first creation (anyone can reference it in training) | | `--readme` | Dataset description (markdown); defaults to the README.md in the directory | ### `sf dataset quality` Produce a dataset quality report ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset quality [options] ``` | Argument | Description | | -------- | ------------------------------------- | | `PATH` | Data directory or file (**required**) | | Option | Description | | ---------- | ----------------------------------- | | `--fields` | Only these fields (comma separated) | ### `sf dataset visibility` Change a dataset's visibility (owner or admin) ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf dataset visibility [options] ``` | Argument | Description | | --------- | -------------------------------------------- | | `DATASET` | Dataset id (\/\) (**required**) | | Option | Description | | ----------- | --------------- | | `--public` | Make it public | | `--private` | Make it private |