> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# kuberay Backend

> K8s + KubeRay: operator, RBAC, job package distribution, GPU scheduling

The recommended production shape: one ephemeral RayCluster (RayJob) per training job, destroyed on reaching a terminal state, with resource reclamation guaranteed by K8s.

## Prerequisites

1. A Kubernetes cluster (GPU nodes with the device plugin installed);
2. [KubeRay operator](https://github.com/ray-project/kuberay) (installed via helm, v1.4+);
3. Object storage (**required**: job packages are distributed into containers via presigned URLs);
4. GPU nodes labeled by series: `kubectl label node <n> starforge/gpu-series=h200`.

## Configuration

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
FORGE_DEFAULT_FLEET_KIND=kuberay
FORGE_IMAGE_NEMO_RL=registry.internal/starforge/nemo-rl:v0.7.0
FORGE_NEMO_RL_DIR=/opt/nemo-rl
FORGE_K8S_API_SERVER=https://<apiserver>:6443
FORGE_K8S_TOKEN=<ServiceAccount token>        # needs RayJob/Pod CRUD permissions
FORGE_K8S_NAMESPACE=starforge
FORGE_K8S_GPU_NODE_LABEL=starforge/gpu-series  # series label key (same source for capacity stats and nodeSelector)
FORGE_K8S_SHM_SIZE=64Gi                       # Ray object store uses /dev/shm
FORGE_STORAGE_ROOT=/starforge                  # the one storage root
FORGE_K8S_STORAGE_PVC=starforge-storage        # the RWX claim behind it
FORGE_K8S_STORAGE_SHARED=true                  # false if the claim is RWO: multi-node jobs are then rejected
# FORGE_K8S_CORPUS_PVC=starforge-corpus        # optional read-only corpus claim
# FORGE_K8S_STORAGE_MOUNT=/storage             # only if the pod cannot mount at FORGE_STORAGE_ROOT
FORGE_KUBERAY_RAY_VERSION=2.55.1              # Ray version compatible with the cluster operator
FORGE_KUBERAY_TTL_SECONDS=600                 # K8s GC fallback if active cleanup fails
FORGE_KUBERAY_PRERUNNING_DEADLINE_S=1800      # how long before a job that never starts (image/scheduling) is failed
FORGE_S3_ENDPOINT=…                           # object storage (required for job package distribution)
FORGE_S3_BUCKET=starforge
FORGE_S3_ACCESS_KEY=…                         # Secret
FORGE_S3_SECRET_KEY=…                         # Secret
```

## Key semantics

<AccordionGroup>
  <Accordion title="Job Capsule distribution" icon="package">
    The console assembles user code, the authoritative JobSpec, manifest, and `runner.pex` into a content-addressed Capsule and uploads it to object storage. HTTPS archives use Ray's `runtimeEnvYAML.working_dir`; for HTTP-only storage, an init container in every Ray Pod verifies the transport SHA-256 and exposes the same `file://` path to the runtime-env agent. Training images need no platform bootstrap package.
  </Accordion>

  <Accordion title="RayJob shape" icon="boxes">
    head = machine zero of the primary pool; multi-pool jobs get one worker group per pool (nodeSelector by series, pinned via Ray custom resources). `shutdownAfterJobFinishes=true` + `backoffLimit=0`: training never retries blindly. The platform deletes the RayJob immediately after archiving terminal logs; TTL is only a failure fallback.
  </Accordion>

  <Accordion title="Capacity and scheduling" icon="cpu">
    Cluster capacity is computed live from node `nvidia.com/gpu` capacity plus series labels; unschedulable nodes count as blocked. The per-series capacity shown in the UI and the nodes the scheduler can actually land on come from the **same source**.
  </Accordion>

  <Accordion title="Troubleshooting jobs stuck in Pending" icon="search">
    The real cause only lives in K8s Events (image pull failure / no node satisfying the nodeSelector / not enough GPUs) — the job detail's event view surfaces them directly. Jobs exceeding the preRunning deadline are failed automatically, so they never occupy the queue indefinitely.
  </Accordion>

  <Accordion title="GPU-less clusters (testing only)" icon="flask-conical">
    `FORGE_K8S_GPU_PASSTHROUGH=0`: Pods don't request `nvidia.com/gpu`, while Ray logical resources work as usual. For running the closed loop on kind/CI; production keeps the default.
  </Accordion>
</AccordionGroup>

## Playground

Supported: standalone vLLM Pods (not RayJobs), accessed via the K8s API proxy, reclaimed on TTL expiry.

## Quick verification

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
bash scripts/e2e/setup-kind.sh              # local kind + operator (dev verification)
uv run python scripts/e2e/kuberay_loop.py   # full closed loop
```