> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Maintenance mode

> Drain the cluster without losing work, then resume where everything left off.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf admin maintenance drain --note "Upgrading the Ray cluster image, back by 14:00"
sf admin maintenance status
sf admin maintenance resume
```

Maintenance mode exists so that upgrading a cluster does not mean throwing away the jobs running on
it.

## What draining does

<Steps>
  <Step title="New submissions queue, but nothing dispatches">
    A user can still submit. Their job waits, and the note you wrote is shown to them.
  </Step>

  <Step title="Running jobs are paused, keeping their checkpoints">
    Each is marked `PAUSED` with auto-resume set, under its original run id.
  </Step>

  <Step title="You do the work">
    Replace images, upgrade the operator, patch nodes.
  </Step>

  <Step title="Resume">
    The queue worker resubmits each paused job under its original run id, and training continues from
    its checkpoint. Queued submissions start dispatching again.
  </Step>
</Steps>

Because a resumed job keeps its run id, its metrics, logs and artifacts stay in one place. From the
console it reads as one run that paused, not two runs that have to be mentally joined.

## Write the note

`FORGE_MAINTENANCE_NOTE` is shown to anyone whose submission is held. A note with an expected end
time answers the question people would otherwise ask you. "Maintenance" on its own generates support
traffic.

## What still works while drained

| Works                                                 | Does not                      |
| ----------------------------------------------------- | ----------------------------- |
| Signing in, browsing jobs, reading logs and artifacts | Dispatching anything new      |
| `GET /api/version` and health probes                  | Starting a Playground session |
| The API, for reads                                    |                               |

Model deployments serving application traffic are a separate concern — draining the training queue
does not stop them. Suspend a deployment explicitly if that is what you want.

## Upgrading with it

The intended sequence for a cluster image change:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf admin maintenance drain --note "New NeMo-RL image, ~30 min"
# wait for the running count to reach zero
sf admin maintenance status
# do the upgrade
sf admin maintenance resume
```

<Warning>
  A job that cannot resume from a checkpoint — because the framework version changed incompatibly —
  fails on resume rather than at drain time. When an upgrade crosses a framework major version, plan
  to let jobs finish rather than draining them.
</Warning>

See [upgrades](/en/ops/upgrades) for the rest of that process.