slurm Backend

HPC shape: slurmrestd, runtime profiles, heterogeneous hetjob

For sites that already run Slurm. The control plane talks to slurmrestd (REST + JWT) only — no sbatch or SSH side channel. Training runs inside Enroot/Pyxis or Apptainer.

Prerequisites

1. slurmrestd enabled with JWT authentication configured;
2. Container runtime installed on compute nodes: Enroot + Pyxis (recommended) or Apptainer;
3. A shared filesystem (for job staging directories and artifacts);
4. First-class framework execution artifacts published to the runtime registry as SQSH / SIF (custom + slurm does not allow implicit OCI image materialization).

Configuration

FORGE_DEFAULT_FLEET_KIND=slurm
FORGE_SLURM_REST_URL=https://slurmrestd.internal:6820
FORGE_SLURM_REST_API_VERSION=v0.0.43
FORGE_SLURM_REST_USER=starforge
FORGE_SLURM_JWT_FILE=/run/secrets/slurm-jwt
FORGE_SLURM_PARTITION=gpu
FORGE_SLURM_RUNTIME_PROFILE=enroot-pyxis   # or apptainer
FORGE_STORAGE_ROOT=/lustre/starforge        # the one storage root, on the parallel filesystem
# FORGE_SLURM_STORAGE_ROOT=                 # only if the cluster mounts it at another path
FORGE_SLURM_TLS_VERIFY=1

Key semantics

Batch script rendering

LaunchRequest is rendered to a batch script (image, mounts, srun topology, ingest credentials). After terminal logs are archived, shared staging and scheduler logs are deleted. Interrupted submits are cleaned before retry.

Heterogeneous multi-pool = hetjob

Multi-pool jobs are rendered as Slurm heterogeneous jobs: one component per pool, with --constraint pinning nodes via the GPU series’ slurm_constraint. If a series declares no constraint, the assembly layer rejects outright (fail-closed — components never land on arbitrary GPU nodes).

Status and logs

Status is polled via REST and mapped onto the platform state machine; logs are read back from the shared directory (handled asynchronously, so large logs never block the event loop).

Known boundaries

• Playground is not supported: training nodes usually have no inbound network;
• The E2E closed loop does not yet cover slurm (requires containerized slurmrestd + Apptainer, planned) — batch script rendering, the REST client, hetjob, and status mapping have full unit-test coverage. See E2E Closed-Loop Testing.