> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Manage datasets

> Local preprocessing → versioned upload → automatic distribution to the job's shared cache

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf dataset prepare gsm8k-zh                     # preprocess locally
sf dataset push gsm8k-zh v2 ./out/gsm8k-zh      # upload as version v2
sf submit my-grpo --train-dataset alice/gsm8k-zh@v2 --train-data train.parquet
```

A platform dataset solves three problems at once: **versioning**, so training points at an exact
version rather than at whatever was on disk; **distribution**, so a job pulls it into the cluster's
shared cache at startup instead of somebody scp-ing it to every node; and **permission**, so who may
use it is a property of the dataset rather than of the filesystem.

## Three steps

<Steps>
  <Step title="Preprocess locally">
    ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
    sf dataset prepare              # list available preprocessing scripts
    sf dataset prepare gsm8k-zh     # run common/data/prepare_gsm8k_zh.py
    ```

    Preprocessing scripts live by convention at `common/data/prepare_*.py` and produce a local directory (parquet / jsonl).
  </Step>

  <Step title="Versioned upload">
    ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
    sf dataset push gsm8k-zh v1 ./out/gsm8k-zh        # private (default)
    sf dataset push gsm8k-zh v1 ./out/gsm8k-zh --public   # public for platform-wide reference
    ```

    Uploads to object storage (MinIO / S3) with streamed chunking + SHA256 verification + resumable upload (rerunning after an interruption skips already-uploaded files). Versions are **immutable** — a version of the same name cannot be overwritten.
  </Step>

  <Step title="Reference in training">
    Prefer writing it into the experiment config:

    ```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
    data:
      train:
        dataset: alice/gsm8k-zh@v2      # <owner>/<name>[@version]; omitting the version takes the latest
        file: train.parquet
    ```

    Or override temporarily at submission:

    ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
    sf submit my-exp --train-dataset alice/gsm8k-zh@v2 --train-data train.parquet
    ```

    At job startup, the platform pulls the dataset to the **cluster shared cache** (with integrity verification; corruption triggers an automatic re-pull) and injects the `<NAME>_DATA_DIR` environment variable; `--train-data` takes just the relative filename within the dataset.
  </Step>
</Steps>

## Writing a dataset card

A dataset with nothing but a name is a black box to everyone except whoever uploaded it — and
picking the wrong data costs hundreds of GPU hours. Put the description in a `README.md` inside
the data directory and `sf dataset push` publishes it along with the data:

```markdown theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# gsm8k-zh

Chinese math problems rewritten from GSM8K, deduplicated by question text.

## Columns

| Column | Type | Description |
| --- | --- | --- |
| `question` | string | The problem statement |
| `answer` | string | Reference solution including the reasoning |

## Provenance

`common/data/prepare_gsm8k_zh.py`; 200 samples spot-checked by hand after translation.
```

Updating the card later does not require pushing new data:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf dataset card alice/gsm8k-zh -f README.md
```

Its first paragraph becomes the one-line description on every list row; the body is rendered in
the console's **Card** tab.

## Management

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf dataset ls                                  # visible datasets (public + your own)
sf dataset visibility gsm8k-zh --public        # change visibility (owner or admin)
```

Every row of the console **Data → Datasets** tab (`/data`) carries the description, latest version,
file count, size, formats and update time — no need to open datasets one by one to compare them.
Opening one gives you four tabs:

| Tab     | The question it answers                                                                 |
| ------- | --------------------------------------------------------------------------------------- |
| Card    | What this data is, which columns it has, where it came from                             |
| Files   | What this version contains, how large, with which checksums                             |
| Preview | Column types, how many empties, roughly how long the text is (is `max_seq_len` enough?) |
| Usage   | How many jobs reference it — i.e. whether it is safe to roll or clean up                |

### Parquet

Parquet is the most common training-data format here, and its preview works out of the box
(`pyarrow` ships with the control plane; it is not an optional extra). It gives you two things
jsonl and csv cannot:

* **The total row count**, read from the footer at the end of the file — a few KB. jsonl and csv
  would need a full scan, so those formats show "—" here rather than a guessed number.
* **Declared column types** (`int64` / `string` / `timestamp[us]` …). More reliable than types
  inferred from the first few rows: a sample that happens to be all integers does not make the
  column an integer column, and an all-null sample does not mean the column has no type.

The platform reads only the footer and the first row group, with a cap on total bytes. **Files
with very large row groups (64–128 MB per group is a common writer setting) yield no sample
rows**; columns and row count are still shown, with a note where the samples would be — those two
cost only the footer's few KB and should not be thrown away along with the samples.

<Note>
  Column lengths in the preview are measured **before truncation**. Sample cells are cut to 160
  characters on their way to the browser, so an average computed from the samples would always
  equal the truncation cap — a number you cannot size `max_seq_len` with.

  "Referenced by N jobs" is platform-wide; the job list below it still only shows jobs you can
  already see.
</Note>

<Note>
  The dataset manifest (with presigned URLs) is generated **at submission** and written into the job package — the job side needs no object storage credentials and no access to the console; one fewer runtime dependency means one fewer failure point.
</Note>

## Scanning for sensitive values

After a version is published, the reconcile pass scans it once and the result
appears on the dataset page.

**It produces a report and never rewrites what you uploaded.** Every comparable
tool draws this line -- Cloud DLP's `inspect` and `deidentify` are two calls you
make separately, Macie reports and does not touch the object -- and the reason is
sharper here. A reflow buffer is something the platform produced, so rewriting
one is its business. **A training set is your material**: rewriting it silently
changes what a model learns, cannot be undone, and leaves nobody able to explain
a result afterwards.

To clean it, use a **Processing Run**: read the material, run your own
treatment, publish a new version. Your code, in the platform, with provenance.

### It samples, and says how much

A hundred-gigabyte dataset cannot be read on the way to a submission. The scan
reads bounded windows and **reports how many rows it read**. A sampled scan can
never say "clean" -- only "nothing in what I looked at" -- and a report that omits
its sample size is one somebody will eventually quote as proof.

Windows are **spread across each file** rather than taken from the head. A bad
export is appended and a test fixture sits at the top, so a head-only sample
systematically misses the case most worth catching.

### Confidence, not a boolean

Every rule carries a severity. `api_key`, `private_key` and a checksum-valid
identity number are `high`; email and phone are `medium`; **the generic
long-token rule is `low` -- it hits every base64 blob and git SHA in a code
corpus.**

So the blocking policy (`FORGE_DATA_SCAN_POLICY=block`) **only fires on `high`**.
A gate that refuses work over low-confidence heuristics is one somebody turns off
within a week, taking the useful half with it.

### Your own terms

`FORGE_DATA_SCAN_DENY_TERMS` adds terms this deployment treats as sensitive (a
customer number, a case id); `FORGE_DATA_SCAN_ALLOW_TERMS` suppresses known
false positives.

**Terms, never patterns.** Pasting a regex into a settings box is how a config
change becomes a catastrophic backtrack that hangs a worker -- Python's `re` has
no timeout. The platform escapes and compiles them. Presidio's deny lists work
the same way, for the same reason.

## Volumes (not train/val)

Material a job reads but does not train on — internal documents, reference PDFs,
anything a script should see — lives in a **Volume**: a directory you create in
the console and drop files into. It has no versions; its current contents are
its contents.

```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
data:
  volumes:
    - alice/legal-docs
```

Each volume appears inside the container at `$VOLUMES_DIR/<name>`, read-only.
Nobody can download its files unless the owner turns that on, and there is no
preview at any setting. The console's Volumes page is the reference.

## Relationship to HuggingFace datasets

Writing an HF dataset id directly in the config (e.g. `nvidia/OpenMathInstruct-2`) also works: the platform runs an HF preflight at submission (gated datasets verify your authorization). Platform datasets suit **internal data** and **preprocessed derived data** — no need to push to HF, and permissions stay on the internal network.