Backup and restore

What actually has to survive, and what you can afford to lose.

Three tiers, and knowing which tier a thing is in decides how much effort it deserves.

Tier 1 — irreplaceable

What	Where	Notes
The database	Postgres, or FORGE_DB_PATH for SQLite	Every job record, quota, rubric, model version, audit entry. Losing it loses the platform’s memory, not its files
<root>/runs/**/out	Storage root	Checkpoints, exports, evaluation reports. Nothing reconstructs these
<root>/state	Storage root	Control-plane bookkeeping
Object storage	Your S3 bucket	Datasets, job packages, archives. Back it up as a bucket, not through the platform
FORGE_WEB_JWT_SECRET and FORGE_SECRET_ENC_KEY	Your secret store	Losing the encryption key makes every stored integration credential unreadable

Tier 2 — expensive to lose, reconstructible

What	Cost of losing it
<root>/cache/hf	Every job re-downloads its weights. On an internal link that can cost more than the training
<root>/cache/datasets	Re-pulled from object storage on next use
<root>/runs/**/logs	Gone for good once a job is terminal, but rarely load-bearing after a run is understood

Tier 3 — do not bother

<root>/cache/corpora, <root>/packages (when object storage is on), and <root>/runs/**/work for finished runs. All reconstructible, all reclaimed under disk pressure anyway.

Backing up the database

Postgres

pg_dump --format=custom --file=starforge-$(date +%F).dump "$FORGE_DB_URL"

Schema migrations run automatically at startup, so a dump restored into a newer build will be migrated forward. Restoring a dump into an older build is not supported — keep a note of the version each dump came from.

SQLite

sqlite3 .forge/web.db ".backup 'starforge-$(date +%F).db'"

Use .backup rather than copying the file: a plain cp of a live SQLite database can capture a torn write.

Restoring

1. Stop the console

   Or put it in maintenance mode first and let jobs drain — see maintenance.

2. Restore the database and the storage root

   They have to be consistent with each other. A database that references a run directory which was restored from an older snapshot will show artifacts that are not there.

3. Restore the secrets

   Especially FORGE_SECRET_ENC_KEY. Without the original, stored integration credentials decrypt to nothing and every user has to re-link.

4. Start, and check

   GET /api/version answers, the job list renders, and one finished run still shows its artifacts.

Test the restore. A backup nobody has restored is a hypothesis, and the failure mode — a missing encryption key discovered during an incident — is the expensive kind.

What is not in scope

Training data that lives outside the platform, and container images. Both are somebody else’s backup: the source of truth for a dataset you pushed is the object store, and for an image it is your registry.