> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# sf job

> Job lifecycle: logs, status, stop, pause/resume, batch operations

## Inspect

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job ls                     # Job list (including submission history)
sf job status <JOB_ID>        # Status details
sf job logs [JOB_ID]          # Follow logs; omit the ID to follow the most recent job
sf job logs <JOB_ID> -n 0     # Replay full history then follow (default replays only the last 2000 lines)
sf job samples <JOB_ID>       # Multi-turn conversation traces from the latest validation
sf job samples <JOB_ID> --vidx 2 -n 10   # 3rd validation round, show 10 samples
```

| `logs` option | Description                                                                          |
| ------------- | ------------------------------------------------------------------------------------ |
| `-n, --tail`  | Replay the last N lines of history then follow; `-n 0` for everything (default 2000) |

| `samples` option | Description                                       |
| ---------------- | ------------------------------------------------- |
| `--vidx`         | Validation round index (default -1 = most recent) |
| `-n, --limit`    | Number of samples to show (default 6)             |

## Control

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job stop <JOB_ID>          # Stop (running → terminated)
sf job pause <JOB_ID>         # Pause: keep the checkpoint, resumable
sf job resume <JOB_ID>        # Automatically resume training from the latest checkpoint
sf job delete <JOB_ID>        # Delete the record of a finished job (running jobs must be stopped first)
```

<Note>
  `pause` / `resume` are maintenance-friendly pause semantics: when an administrator drains the cluster in maintenance mode, jobs go through the same path, and after recovery they resume training automatically without losing progress.
</Note>

## Batch

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job cancel-all             # Stop all of my running / queued jobs
sf job clean                  # Clean up display records of finished jobs
sf job stop-sweep <SWEEP_ID>  # Stop all active jobs of a sweep with one command
```

## `sf job`

Manage jobs

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job <subcommand>
```

| Command             | Description                                   |
| ------------------- | --------------------------------------------- |
| `sf job cancel-all` | Stop every running or queued job of mine      |
| `sf job clean`      | Clean up the display records of finished jobs |
| `sf job delete`     | Delete one finished job record                |
| `sf job logs`       | Follow a job's logs                           |
| `sf job ls`         | List jobs (including the submission history)  |
| `sf job pause`      | Pause a job, keeping its checkpoint           |
| `sf job resume`     | Resume a paused job                           |
| `sf job retry`      | Retry a failed or stopped job                 |
| `sf job samples`    | Show one validation round's traces            |
| `sf job status`     | Show a job's status                           |
| `sf job stop`       | Stop a job (running -> terminated)            |
| `sf job stop-sweep` | Stop every active job of a sweep              |

### `sf job cancel-all`

Stop every running or queued job of mine

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job cancel-all [options]
```

| Option        | Description           |
| ------------- | --------------------- |
| `--yes`, `-y` | Skip the confirmation |

### `sf job clean`

Clean up the display records of finished jobs

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job clean
```

No options beyond `--help`.

### `sf job delete`

Delete one finished job record

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job delete <JOB_ID>
```

| Argument | Description           |
| -------- | --------------------- |
| `JOB_ID` | Job id (**required**) |

No options beyond `--help`.

### `sf job logs`

Follow a job's logs

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job logs [JOB_ID] [options]
```

| Argument | Description                                  |
| -------- | -------------------------------------------- |
| `JOB_ID` | Job id (see sf job ls); omitted = the latest |

| Option                 | Description                                                                                            |
| ---------------------- | ------------------------------------------------------------------------------------------------------ |
| `--tail`, `-n` `<int>` | Replay only the last N log lines before following (default 2000; -n 0 for everything) (default `2000`) |

### `sf job ls`

List jobs (including the submission history)

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job ls [options]
```

| Option                  | Description                                          |
| ----------------------- | ---------------------------------------------------- |
| `--all`                 | Show everything (default: the latest 15)             |
| `--exp`                 | Only this experiment (full name or last segment)     |
| `--limit`, `-n` `<int>` | How many to show (ignored with --all) (default `15`) |

### `sf job pause`

Pause a job, keeping its checkpoint

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job pause <JOB_ID>
```

| Argument | Description           |
| -------- | --------------------- |
| `JOB_ID` | Job id (**required**) |

No options beyond `--help`.

### `sf job resume`

Resume a paused job

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job resume <JOB_ID>
```

| Argument | Description           |
| -------- | --------------------- |
| `JOB_ID` | Job id (**required**) |

No options beyond `--help`.

### `sf job retry`

Retry a failed or stopped job

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job retry <JOB_ID>
```

| Argument | Description           |
| -------- | --------------------- |
| `JOB_ID` | Job id (**required**) |

No options beyond `--help`.

### `sf job samples`

Show one validation round's traces

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job samples <JOB_ID> [options]
```

| Argument | Description                           |
| -------- | ------------------------------------- |
| `JOB_ID` | Job id (see sf job ls) (**required**) |

| Option                  | Description                                                     |
| ----------------------- | --------------------------------------------------------------- |
| `--vidx` `<int>`        | Validation round index (default -1 = the latest) (default `-1`) |
| `--limit`, `-n` `<int>` | How many samples to show (default `6`)                          |

### `sf job status`

Show a job's status

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job status <JOB_ID>
```

| Argument | Description           |
| -------- | --------------------- |
| `JOB_ID` | Job id (**required**) |

No options beyond `--help`.

### `sf job stop`

Stop a job (running -> terminated)

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job stop <JOB_ID>
```

| Argument | Description           |
| -------- | --------------------- |
| `JOB_ID` | Job id (**required**) |

No options beyond `--help`.

### `sf job stop-sweep`

Stop every active job of a sweep

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf job stop-sweep <SWEEP_ID> [options]
```

| Argument   | Description                                                 |
| ---------- | ----------------------------------------------------------- |
| `SWEEP_ID` | The sweep id (printed when sf sweep submits) (**required**) |

| Option        | Description           |
| ------------- | --------------------- |
| `--yes`, `-y` | Skip the confirmation |