Storage layout

One root, everything derived from it, and which parts you can safely reclaim.

FORGE_STORAGE_ROOT=/mnt/shared/starforge

Every path the platform writes derives from that one setting. Nothing else is configured, and nothing is written outside it.

The layout

<root>/
├── cache/
│   ├── hf/            HF_HOME — model weights, shared across every run
│   ├── datasets/      platform datasets pulled for a job
│   └── corpora/       governed corpora mounted read-only
├── runs/
│   └── <user>/<experiment>/<run_id>/
│       ├── work/      the job's working copy
│       ├── out/       checkpoints, exports, evaluation reports
│       └── logs/      archived logs
├── packages/          job packages, when object storage is off
├── reflow/            captured traffic buffers, per deployment
└── state/             control-plane bookkeeping

The invariant that matters

The root must resolve to the same absolute path on the console, on every node, and inside every container. A shared filesystem, an RWX PVC, or a same-path bind mount all satisfy it.

The path is expanded but deliberately not resolved through symlinks. The root is frequently a symlink or an NFS mount whose real path differs between the console and a node, and the whole design rests on the string being identical on both sides. Resolving it would break exactly the deployments it is meant to support.

A relative root is refused at startup rather than resolved against whatever directory the process happened to start in.

What is safe to reclaim

Directory	Reclaimable	Why
cache/hf	Yes	Reconstructible from the hub or the mirror. Costs a re-download
cache/datasets	Yes	Reconstructible from object storage
cache/corpora	Yes	Same
runs/**/work	Yes, after the run ends	The working copy; the package it came from is still in object storage
runs/**/out	No	Checkpoints and exports. Gone means gone
runs/**/logs	Careful	The only copy once the job is terminal
packages/	Yes, once object storage is on
state/	No	Control-plane bookkeeping

The caches are shared across runs on purpose. Without a shared HF cache, every job re-downloads tens of gigabytes of weights, which on an internal link costs more than the training does.

Where the database is not

FORGE_DB_PATH is deliberately not under the storage root. That root is usually a shared filesystem, and SQLite over NFS has well-known locking problems — the ledger would risk corruption and every console replica would contend for one file. The database is control-plane state, not job storage, so it stays local to the process.

A team deployment sets FORGE_DB_URL to Postgres and the question stops mattering.

Corpora live outside

FORGE_CORPUS_ROOT must sit outside the storage root, and startup refuses a configuration where it does not. Corpora are governed read-only material with different access rules from the platform’s own writes; nesting them would make one reclamation policy apply to both.

Accounting and pressure

Setting	What it does
FORGE_STORAGE_SCAN_INTERVAL_S	How often the root is walked for per-user usage. 0 means storage quotas report “not measured”
FORGE_DISK_WATERMARK_PCT	Stop dequeuing above this usage, to protect the checkpoint writes of jobs already running

The watermark is worth setting. A full disk breaks every running job at once, not just the next one to start — and the jobs it breaks are the ones that have been running longest and have the most to lose.

Storage quota is measured, not reserved, so it is eventually consistent — a user can briefly exceed it between scans. The GPU gate is exact because allocation is a decision the platform makes; storage is a fact it observes.