> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Backup and restore

> What actually has to survive, and what you can afford to lose.

Three tiers, and knowing which tier a thing is in decides how much effort it deserves.

## Tier 1 — irreplaceable

| What                                              | Where                                   | Notes                                                                                                             |
| ------------------------------------------------- | --------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| The database                                      | Postgres, or `FORGE_DB_PATH` for SQLite | Every job record, quota, rubric, model version, audit entry. Losing it loses the platform's memory, not its files |
| `<root>/runs/**/out`                              | Storage root                            | Checkpoints, exports, evaluation reports. Nothing reconstructs these                                              |
| `<root>/state`                                    | Storage root                            | Control-plane bookkeeping                                                                                         |
| Object storage                                    | Your S3 bucket                          | Datasets, job packages, archives. Back it up as a bucket, not through the platform                                |
| `FORGE_WEB_JWT_SECRET` and `FORGE_SECRET_ENC_KEY` | Your secret store                       | Losing the encryption key makes every stored integration credential unreadable                                    |

## Tier 2 — expensive to lose, reconstructible

| What                    | Cost of losing it                                                                            |
| ----------------------- | -------------------------------------------------------------------------------------------- |
| `<root>/cache/hf`       | Every job re-downloads its weights. On an internal link that can cost more than the training |
| `<root>/cache/datasets` | Re-pulled from object storage on next use                                                    |
| `<root>/runs/**/logs`   | Gone for good once a job is terminal, but rarely load-bearing after a run is understood      |

## Tier 3 — do not bother

`<root>/cache/corpora`, `<root>/packages` (when object storage is on), and `<root>/runs/**/work`
for finished runs. All reconstructible, all reclaimed under disk pressure anyway.

## Backing up the database

<Tabs>
  <Tab title="Postgres">
    ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
    pg_dump --format=custom --file=starforge-$(date +%F).dump "$FORGE_DB_URL"
    ```

    Schema migrations run automatically at startup, so a dump restored into a newer build will be
    migrated forward. Restoring a dump into an *older* build is not supported — keep a note of the
    version each dump came from.
  </Tab>

  <Tab title="SQLite">
    ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
    sqlite3 .forge/web.db ".backup 'starforge-$(date +%F).db'"
    ```

    Use `.backup` rather than copying the file: a plain `cp` of a live SQLite database can capture a
    torn write.
  </Tab>
</Tabs>

## Restoring

<Steps>
  <Step title="Stop the console">
    Or put it in maintenance mode first and let jobs drain — see [maintenance](/en/ops/maintenance).
  </Step>

  <Step title="Restore the database and the storage root">
    They have to be consistent with each other. A database that references a run directory which was
    restored from an older snapshot will show artifacts that are not there.
  </Step>

  <Step title="Restore the secrets">
    Especially `FORGE_SECRET_ENC_KEY`. Without the original, stored integration credentials decrypt to
    nothing and every user has to re-link.
  </Step>

  <Step title="Start, and check">
    `GET /api/version` answers, the job list renders, and one finished run still shows its artifacts.
  </Step>
</Steps>

<Warning>
  Test the restore. A backup nobody has restored is a hypothesis, and the failure mode — a missing
  encryption key discovered during an incident — is the expensive kind.
</Warning>

## What is not in scope

Training data that lives outside the platform, and container images. Both are somebody else's
backup: the source of truth for a dataset you pushed is the object store, and for an image it is your
registry.