> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Job states

> QUEUED through SUCCEEDED, pause/resume, auto-retry, and when GPU time is counted

The ledger on the server is the source of truth for status in the CLI and the console. Backend-specific strings (Kubernetes pod phase, Slurm `PENDING`) are mapped onto this list. There is no silent "unknown" that still looks like success.

<img src="https://mintcdn.com/starforge/px4T5UDP_V7qWFbL/images/job-states.svg?fit=max&auto=format&n=px4T5UDP_V7qWFbL&q=85&s=c0cf7e3ecf14a8c3212a1fca458828ec" alt="A job's states: QUEUED, SUBMITTED, PENDING, RUNNING, the three terminal states, and PAUSED — with which of them hold GPUs, occupy a job slot, and count GPU-hours." className="block dark:hidden w-full" noZoom width="1400" height="760" data-path="images/job-states.svg" />

<img src="https://mintcdn.com/starforge/px4T5UDP_V7qWFbL/images/job-states-dark.svg?fit=max&auto=format&n=px4T5UDP_V7qWFbL&q=85&s=c6f0bb9195cdbdd07a15bfa2ffea42e3" alt="A job's states: QUEUED, SUBMITTED, PENDING, RUNNING, the three terminal states, and PAUSED — with which of them hold GPUs, occupy a job slot, and count GPU-hours." className="hidden dark:block w-full" noZoom width="1400" height="760" data-path="images/job-states-dark.svg" />

## What each state means

| State       | On the cluster?         | Typical cause                                                                                                                             |
| ----------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `QUEUED`    | No                      | Admitted, waiting for quota, GPUs, a schedule window, or (after resume) a new launch. `job_ref` is empty.                                 |
| `SUBMITTED` | Launch in flight        | Quota is already reserved. The executor has not returned a handle yet.                                                                    |
| `PENDING`   | Yes, not training yet   | Image pull, node selector, Slurm allocation, Ray cluster coming up.                                                                       |
| `RUNNING`   | Yes                     | Entrypoint is executing.                                                                                                                  |
| `SUCCEEDED` | Terminal                | Process exited 0. Optional `--then export` / `--then eval` jobs may still start.                                                          |
| `FAILED`    | Terminal (unless retry) | Non-zero exit, ingest hard failure, OOM, preRunning deadline.                                                                             |
| `STOPPED`   | Terminal (unless retry) | User or admin stop.                                                                                                                       |
| `PAUSED`    | No                      | Training process stopped; the row goes back through `QUEUED` on resume. Same path as a new dequeue: quota and capacity gates apply again. |

`QUEUED` is not "the job is on Slurm/K8s and waiting for a GPU." That situation is `PENDING`. If the dashboard says queued for a long time, look at quota and the schedule window first, not at `kubectl get pods`.

## Pause, resume, retry

* `sf job pause` / console pause: stop the running process, keep the ledger row, status `PAUSED`.
* `sf job resume`: set status back to `QUEUED`. The scheduler submits a new launch. It is not a transparent attach to the old container.
* Console retry / `sf job retry`: a training job that is `FAILED` or `STOPPED` can be requeued from either surface, with no second `sf submit`. It goes back on the queue under the **same run\_id** -- same output directory, the framework continues from its latest checkpoint, one ledger row throughout, retry count incremented. Same path as resume, so quota, capacity and schedule-window gates all apply again.
* Auto-retry after `FAILED` stays inside a budget and a cooldown. The retry count is on the job detail page. Stop the job if you do not want another attempt.

When the retry button is greyed out, its tooltip says why. The usual reason is that the working-directory snapshot is gone: it is the only copy of the code as submitted, it is cleared when the job row is deleted, and after that the job has to be submitted again.

## GPU time

Queued time is not billed. Metering starts once the job is actually occupying GPUs after `PENDING` (container up, `start_time` set). Occupancy is `gpu_seconds` (wall time × GPU count), visible on the console usage page.

## Watch a job

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job status          # latest job if you omit the id
sf job logs            # follow; omit id for latest
sf job logs <id> -n 0  # replay history
```

Console job page: Charts, Logs, Validation, System (GPU util), Events, Diagnostics.