Job states

QUEUED through SUCCEEDED, pause/resume, auto-retry, and when GPU time is counted

The ledger on the server is the source of truth for status in the CLI and the console. Backend-specific strings (Kubernetes pod phase, Slurm PENDING) are mapped onto this list. There is no silent “unknown” that still looks like success.

What each state means

State	On the cluster?	Typical cause
QUEUED	No	Admitted, waiting for quota, GPUs, a schedule window, or (after resume) a new launch. job_ref is empty.
SUBMITTED	Launch in flight	Quota is already reserved. The executor has not returned a handle yet.
PENDING	Yes, not training yet	Image pull, node selector, Slurm allocation, Ray cluster coming up.
RUNNING	Yes	Entrypoint is executing.
SUCCEEDED	Terminal	Process exited 0. Optional --then export / --then eval jobs may still start.
FAILED	Terminal (unless retry)	Non-zero exit, ingest hard failure, OOM, preRunning deadline.
STOPPED	Terminal (unless retry)	User or admin stop.
PAUSED	No	Training process stopped; the row goes back through QUEUED on resume. Same path as a new dequeue: quota and capacity gates apply again.

QUEUED is not “the job is on Slurm/K8s and waiting for a GPU.” That situation is PENDING. If the dashboard says queued for a long time, look at quota and the schedule window first, not at kubectl get pods.

Pause, resume, retry

• sf job pause / console pause: stop the running process, keep the ledger row, status PAUSED.
• sf job resume: set status back to QUEUED. The scheduler submits a new launch. It is not a transparent attach to the old container.
• Console retry / sf job retry: a training job that is FAILED or STOPPED can be requeued from either surface, with no second sf submit. It goes back on the queue under the same run_id — same output directory, the framework continues from its latest checkpoint, one ledger row throughout, retry count incremented. Same path as resume, so quota, capacity and schedule-window gates all apply again.
• Auto-retry after FAILED stays inside a budget and a cooldown. The retry count is on the job detail page. Stop the job if you do not want another attempt.

When the retry button is greyed out, its tooltip says why. The usual reason is that the working-directory snapshot is gone: it is the only copy of the code as submitted, it is cleared when the job row is deleted, and after that the job has to be submitted again.

GPU time

Queued time is not billed. Metering starts once the job is actually occupying GPUs after PENDING (container up, start_time set). Occupancy is gpu_seconds (wall time × GPU count), visible on the console usage page.

Watch a job

sf job status          # latest job if you omit the id
sf job logs            # follow; omit id for latest
sf job logs <id> -n 0  # replay history

Console job page: Charts, Logs, Validation, System (GPU util), Events, Diagnostics.