kuberay Backend

K8s + KubeRay: operator, RBAC, job package distribution, GPU scheduling

The recommended production shape: one ephemeral RayCluster (RayJob) per training job, destroyed on reaching a terminal state, with resource reclamation guaranteed by K8s.

Prerequisites

1. A Kubernetes cluster (GPU nodes with the device plugin installed);
2. KubeRay operator (installed via helm, v1.4+);
3. Object storage (required: job packages are distributed into containers via presigned URLs);
4. GPU nodes labeled by series: kubectl label node <n> starforge/gpu-series=h200.

Configuration

FORGE_DEFAULT_FLEET_KIND=kuberay
FORGE_IMAGE_NEMO_RL=registry.internal/starforge/nemo-rl:v0.7.0
FORGE_NEMO_RL_DIR=/opt/nemo-rl
FORGE_K8S_API_SERVER=https://<apiserver>:6443
FORGE_K8S_TOKEN=<ServiceAccount token>        # needs RayJob/Pod CRUD permissions
FORGE_K8S_NAMESPACE=starforge
FORGE_K8S_GPU_NODE_LABEL=starforge/gpu-series  # series label key (same source for capacity stats and nodeSelector)
FORGE_K8S_SHM_SIZE=64Gi                       # Ray object store uses /dev/shm
FORGE_STORAGE_ROOT=/starforge                  # the one storage root
FORGE_K8S_STORAGE_PVC=starforge-storage        # the RWX claim behind it
FORGE_K8S_STORAGE_SHARED=true                  # false if the claim is RWO: multi-node jobs are then rejected
# FORGE_K8S_CORPUS_PVC=starforge-corpus        # optional read-only corpus claim
# FORGE_K8S_STORAGE_MOUNT=/storage             # only if the pod cannot mount at FORGE_STORAGE_ROOT
FORGE_KUBERAY_RAY_VERSION=2.55.1              # Ray version compatible with the cluster operator
FORGE_KUBERAY_TTL_SECONDS=600                 # K8s GC fallback if active cleanup fails
FORGE_KUBERAY_PRERUNNING_DEADLINE_S=1800      # how long before a job that never starts (image/scheduling) is failed
FORGE_S3_ENDPOINT=…                           # object storage (required for job package distribution)
FORGE_S3_BUCKET=starforge
FORGE_S3_ACCESS_KEY=…                         # Secret
FORGE_S3_SECRET_KEY=…                         # Secret

Key semantics

Job Capsule distribution

The console assembles user code, the authoritative JobSpec, manifest, and runner.pex into a content-addressed Capsule and uploads it to object storage. HTTPS archives use Ray’s runtimeEnvYAML.working_dir; for HTTP-only storage, an init container in every Ray Pod verifies the transport SHA-256 and exposes the same file:// path to the runtime-env agent. Training images need no platform bootstrap package.

RayJob shape

head = machine zero of the primary pool; multi-pool jobs get one worker group per pool (nodeSelector by series, pinned via Ray custom resources). shutdownAfterJobFinishes=true + backoffLimit=0: training never retries blindly. The platform deletes the RayJob immediately after archiving terminal logs; TTL is only a failure fallback.

Capacity and scheduling

Cluster capacity is computed live from node nvidia.com/gpu capacity plus series labels; unschedulable nodes count as blocked. The per-series capacity shown in the UI and the nodes the scheduler can actually land on come from the same source.

Troubleshooting jobs stuck in Pending

The real cause only lives in K8s Events (image pull failure / no node satisfying the nodeSelector / not enough GPUs) — the job detail’s event view surfaces them directly. Jobs exceeding the preRunning deadline are failed automatically, so they never occupy the queue indefinitely.

GPU-less clusters (testing only)

FORGE_K8S_GPU_PASSTHROUGH=0: Pods don’t request nvidia.com/gpu, while Ray logical resources work as usual. For running the closed loop on kind/CI; production keeps the default.

Playground

Supported: standalone vLLM Pods (not RayJobs), accessed via the K8s API proxy, reclaimed on TTL expiry.

Quick verification

bash scripts/e2e/setup-kind.sh              # local kind + operator (dev verification)
uv run python scripts/e2e/kuberay_loop.py   # full closed loop