Install on Kubernetes

The console in-cluster with kustomize, and the RWX claim the KubeRay executor depends on.

kubectl apply -k deploy/k8s/
kubectl -n starforge rollout status deploy/starforge-console

deploy/k8s/ is a kustomize overlay covering the namespace, config, secrets, RBAC, deployment, service, ingress and the two PVCs.

What it creates

Manifest	What it is for
namespace.yaml	starforge
configmap.yaml	Non-secret FORGE_* settings
secret.example.yaml	Secret settings — replace the source before production
rbac.yaml	The RayJob permissions the kuberay executor needs. Without it, dispatch gets a 403
deployment.yaml	One replica, probes on /healthz, 250m–2 CPU, 512Mi–2Gi memory
service.yaml, ingress.yaml	In-cluster address and the external route
pvc.yaml	The storage-root claim, plus an optional one for local SQLite

Point the image at your registry before applying:

deploy/k8s/kustomization.yaml

images:
  - name: registry.company.com/forge/starforge-console
    newTag: "0.3.0"

The storage claim is the part that matters

accessModes: [ReadWriteMany]
resources:
  requests:
    storage: 2Ti
storageClassName: nfs-client     # must actually support RWX

ReadWriteMany is not optional, and RWO fails quietly. An RWO claim does not error — Kubernetes gives each pod its own volume. A multi-node job then splits its checkpoint shards across pods and the run cannot be resumed, with nothing in the logs to say why.

The executor’s health check verifies FORGE_K8S_STORAGE_SHARED against the claim’s real accessModes. Read it after the first deploy rather than assuming.

The console mounts that claim at exactly the path FORGE_STORAGE_ROOT names — it stages job directories into runs/<user>/<exp>/<run_id>/work, measures disk pressure, and reclaims. Training pods mount the same claim, so a path the console writes is the path a job opens.

Size it for weight cache plus dataset cache plus every run directory. 2Ti is a starting point, not a recommendation; the weight cache alone is tens of gigabytes per base model.

Recommended configuration

deploy/k8s/configmap.yaml

FORGE_DEFAULT_FLEET_KIND: "kuberay"
FORGE_STORAGE_ROOT: "/starforge"
FORGE_K8S_STORAGE_PVC: "starforge-storage"
FORGE_K8S_STORAGE_SHARED: "true"
FORGE_PUBLIC_URL: "https://starforge.company.com"
FORGE_INGEST_URL: "http://starforge-console.starforge.svc.cluster.local"
FORGE_KUBERAY_RAY_VERSION: "2.55.1"
FORGE_K8S_SHM_SIZE: "64Gi"
FORGE_KUBERAY_PRERUNNING_DEADLINE_S: "1800"
FORGE_JOB_RUNNER_MODE: "bundled"
FORGE_QUOTA_ENFORCE: "1"

Four of those are worth understanding rather than copying:

FORGE_INGEST_URL uses the in-cluster service name

Training pods report back to it. The external ingress address would leave the cluster and come back in; the service DNS name does not. Never 127.0.0.1 — that is the worker’s own loopback.

FORGE_KUBERAY_RAY_VERSION must match the training image

A mismatch presents as “the cluster came up but workers never registered”, and the only clue is a single version warning easily lost in startup output.

FORGE_K8S_SHM_SIZE, because 64MB is the container default

Ray’s object store lives in /dev/shm. At the default, anything but a tiny batch OOMs immediately. It is RAM-backed, so this plus the memory limit must stay under physical memory.

FORGE_KUBERAY_PRERUNNING_DEADLINE_S bounds a stuck job

Without it, an unpullable image or a nodeSelector nothing satisfies leaves a job Pending forever while holding its queue slot.

Secrets

secret.example.yaml exists to show the shape. Replace it with a real source — kubectl create secret, Sealed Secrets, or the External Secrets Operator — and remove it from kustomization.yaml before production.

At minimum: FORGE_WEB_JWT_SECRET, FORGE_DB_URL, FORGE_REDIS_URL, FORGE_S3_SECRET_KEY, FORGE_SECRET_ENC_KEY, and the OIDC client secret.

Replicas

The manifest ships one replica, and that is the safe default. Before scaling up:

• FORGE_DB_URL must point at Postgres. SQLite is single-writer
• FORGE_WEB_JWT_SECRET must be fixed, or replicas reject each other’s tokens
• FORGE_REDIS_URL must be set, or background roles stop entirely rather than run unsafely across replicas — storage accounting, diagnosis, the watchdog and the daily report all go quiet
• Run the background roles in their own pod. Set FORGE_INPROCESS_WORKERS=0 and apply the worker Deployment — worker.yaml in the kustomize path, worker.enabled=true in the chart. The diagnosis and daily-report ticks call an LLM and hold the event loop for seconds at a time; out of process they stop inflating API tail latency, and the console’s replicas become stateless. One replica is correct: every role elects a leader through Redis, so a second one idles

Confirm it worked

kubectl -n starforge get pods
kubectl -n starforge port-forward svc/starforge-console 8080:80
curl -s localhost:8080/api/version

Then check the executor’s own view — it is what tells you whether the storage claim is really RWX:

curl -s localhost:8080/api/cluster/health | jq .storage