> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# slurm Backend

> HPC shape: slurmrestd, runtime profiles, heterogeneous hetjob

For sites that already run Slurm. The control plane talks to **slurmrestd** (REST + JWT) only — no sbatch or SSH side channel. Training runs inside Enroot/Pyxis or Apptainer.

## Prerequisites

1. slurmrestd enabled with JWT authentication configured;
2. Container runtime installed on compute nodes: Enroot + Pyxis (recommended) or Apptainer;
3. A shared filesystem (for job staging directories and artifacts);
4. First-class framework execution artifacts published to the runtime registry as **SQSH / SIF** (custom + slurm does not allow implicit OCI image materialization).

## Configuration

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
FORGE_DEFAULT_FLEET_KIND=slurm
FORGE_SLURM_REST_URL=https://slurmrestd.internal:6820
FORGE_SLURM_REST_API_VERSION=v0.0.43
FORGE_SLURM_REST_USER=starforge
FORGE_SLURM_JWT_FILE=/run/secrets/slurm-jwt
FORGE_SLURM_PARTITION=gpu
FORGE_SLURM_RUNTIME_PROFILE=enroot-pyxis   # or apptainer
FORGE_STORAGE_ROOT=/lustre/starforge        # the one storage root, on the parallel filesystem
# FORGE_SLURM_STORAGE_ROOT=                 # only if the cluster mounts it at another path
FORGE_SLURM_TLS_VERIFY=1
```

## Key semantics

<AccordionGroup>
  <Accordion title="Batch script rendering" icon="file-code">
    LaunchRequest is rendered to a batch script (image, mounts, `srun` topology, ingest credentials). After terminal logs are archived, shared staging and scheduler logs are deleted. Interrupted submits are cleaned before retry.
  </Accordion>

  <Accordion title="Heterogeneous multi-pool = hetjob" icon="git-fork">
    Multi-pool jobs are rendered as Slurm **heterogeneous jobs**: one component per pool, with `--constraint` pinning nodes via the GPU series' `slurm_constraint`. If a series declares no constraint, the assembly layer rejects outright (fail-closed — components never land on arbitrary GPU nodes).
  </Accordion>

  <Accordion title="Status and logs" icon="activity">
    Status is polled via REST and mapped onto the platform state machine; logs are read back from the shared directory (handled asynchronously, so large logs never block the event loop).
  </Accordion>
</AccordionGroup>

## Known boundaries

* **Playground is not supported**: training nodes usually have no inbound network;
* The E2E closed loop does not yet cover slurm (requires containerized slurmrestd + Apptainer, planned) — batch script rendering, the REST client, hetjob, and status mapping have full unit-test coverage. See [E2E Closed-Loop Testing](/en/ops/e2e-testing).