local Backend

Single-node Docker executor: GPU allocation, shared directories, reclamation

The simplest shape: Console and training containers on the same machine, launched directly with docker run. Suited to teams with a single node of 1~8 GPUs.

Configuration

FORGE_DEFAULT_FLEET_KIND=local
FORGE_IMAGE_NEMO_RL=registry.internal/starforge/nemo-rl:v0.7.0
FORGE_NEMO_RL_DIR=/opt/nemo-rl
FORGE_LOCAL_GPU_COUNT=8            # 0 = auto-detect via nvidia-smi
FORGE_STORAGE_ROOT=/data/starforge   # the one storage root: caches, runs, state (strongly recommended)
FORGE_LOCAL_NETWORK=host           # simplest way for jobs to reach the console
FORGE_LOCAL_SHM_SIZE=64g           # /dev/shm; if empty, falls back to FORGE_K8S_SHM_SIZE with automatic docker syntax conversion
FORGE_LOCAL_CPU_LIMIT=             # optional --cpus
FORGE_LOCAL_MEMORY_LIMIT=          # optional --memory
FORGE_LOCAL_CHECK_EXTERNAL_GPUS=1  # detect off-platform processes occupying GPUs (counted as blocked)

Key semantics

Atomic GPU allocation

GPU selection and container launch happen under the same lock, and the source of truth for allocation lives in container labels — after a console restart, the usage view is rebuilt from docker ps, so “ghost GPU occupation” cannot happen.

Container reclamation

After reconciliation confirms a terminal state, the platform archives any backend-only tail logs and immediately deletes the container. FORGE_LOCAL_CONTAINER_TTL_S only reaps orphans left by control-plane failures.

Images

Resolution order is per-job --image → framework default → runtime registry → catalog. Tags work; prefer a digest for production reproducibility. User images must satisfy FORGE_ALLOWED_IMAGE_REGISTRIES.

No-NVIDIA environments (testing only)

FORGE_LOCAL_GPU_PASSTHROUGH=0 simulation mode: allocation / accounting / labels work as usual, but containers launch without --gpus. For E2E and dev machines; production must keep the default passthrough.

Playground

Supported: vLLM containers start on the same machine, ports are allocated automatically, and services are reclaimed on TTL expiry.

Quick verification

uv run python scripts/e2e/local_loop.py     # full closed loop, runs even without a GPU