Maintenance mode

Drain the cluster without losing work, then resume where everything left off.

sf admin maintenance drain --note "Upgrading the Ray cluster image, back by 14:00"
sf admin maintenance status
sf admin maintenance resume

Maintenance mode exists so that upgrading a cluster does not mean throwing away the jobs running on it.

What draining does

1. New submissions queue, but nothing dispatches

   A user can still submit. Their job waits, and the note you wrote is shown to them.

2. Running jobs are paused, keeping their checkpoints

   Each is marked PAUSED with auto-resume set, under its original run id.

3. You do the work

   Replace images, upgrade the operator, patch nodes.

4. Resume

   The queue worker resubmits each paused job under its original run id, and training continues from its checkpoint. Queued submissions start dispatching again.

Because a resumed job keeps its run id, its metrics, logs and artifacts stay in one place. From the console it reads as one run that paused, not two runs that have to be mentally joined.

Write the note

FORGE_MAINTENANCE_NOTE is shown to anyone whose submission is held. A note with an expected end time answers the question people would otherwise ask you. “Maintenance” on its own generates support traffic.

What still works while drained

Works	Does not
Signing in, browsing jobs, reading logs and artifacts	Dispatching anything new
GET /api/version and health probes	Starting a Playground session
The API, for reads

Model deployments serving application traffic are a separate concern — draining the training queue does not stop them. Suspend a deployment explicitly if that is what you want.

Upgrading with it

The intended sequence for a cluster image change:

sf admin maintenance drain --note "New NeMo-RL image, ~30 min"
# wait for the running count to reach zero
sf admin maintenance status
# do the upgrade
sf admin maintenance resume

A job that cannot resume from a checkpoint — because the framework version changed incompatibly — fails on resume rather than at drain time. When an upgrade crosses a framework major version, plan to let jobs finish rather than draining them.

See upgrades for the rest of that process.