> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Install on Kubernetes

> The console in-cluster with kustomize, and the RWX claim the KubeRay executor depends on.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
kubectl apply -k deploy/k8s/
kubectl -n starforge rollout status deploy/starforge-console
```

`deploy/k8s/` is a kustomize overlay covering the namespace, config, secrets, RBAC, deployment,
service, ingress and the two PVCs.

## What it creates

| Manifest                       | What it is for                                                                       |
| ------------------------------ | ------------------------------------------------------------------------------------ |
| `namespace.yaml`               | `starforge`                                                                          |
| `configmap.yaml`               | Non-secret `FORGE_*` settings                                                        |
| `secret.example.yaml`          | Secret settings — **replace the source before production**                           |
| `rbac.yaml`                    | The RayJob permissions the `kuberay` executor needs. Without it, dispatch gets a 403 |
| `deployment.yaml`              | One replica, probes on `/healthz`, `250m`–`2` CPU, `512Mi`–`2Gi` memory              |
| `service.yaml`, `ingress.yaml` | In-cluster address and the external route                                            |
| `pvc.yaml`                     | The storage-root claim, plus an optional one for local SQLite                        |

Point the image at your registry before applying:

```yaml deploy/k8s/kustomization.yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
images:
  - name: registry.company.com/forge/starforge-console
    newTag: "0.3.0"
```

## The storage claim is the part that matters

```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
accessModes: [ReadWriteMany]
resources:
  requests:
    storage: 2Ti
storageClassName: nfs-client     # must actually support RWX
```

<Warning>
  **ReadWriteMany is not optional, and RWO fails quietly.** An RWO claim does not error — Kubernetes
  gives each pod its own volume. A multi-node job then splits its checkpoint shards across pods and
  the run cannot be resumed, with nothing in the logs to say why.

  The executor's health check verifies `FORGE_K8S_STORAGE_SHARED` against the claim's real
  `accessModes`. Read it after the first deploy rather than assuming.
</Warning>

The console mounts that claim at exactly the path `FORGE_STORAGE_ROOT` names — it stages job
directories into `runs/<user>/<exp>/<run_id>/work`, measures disk pressure, and reclaims. Training
pods mount the same claim, so a path the console writes is the path a job opens.

Size it for weight cache plus dataset cache plus every run directory. 2Ti is a starting point, not a
recommendation; the weight cache alone is tens of gigabytes per base model.

## Recommended configuration

```yaml deploy/k8s/configmap.yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
FORGE_DEFAULT_FLEET_KIND: "kuberay"
FORGE_STORAGE_ROOT: "/starforge"
FORGE_K8S_STORAGE_PVC: "starforge-storage"
FORGE_K8S_STORAGE_SHARED: "true"
FORGE_PUBLIC_URL: "https://starforge.company.com"
FORGE_INGEST_URL: "http://starforge-console.starforge.svc.cluster.local"
FORGE_KUBERAY_RAY_VERSION: "2.55.1"
FORGE_K8S_SHM_SIZE: "64Gi"
FORGE_KUBERAY_PRERUNNING_DEADLINE_S: "1800"
FORGE_JOB_RUNNER_MODE: "bundled"
FORGE_QUOTA_ENFORCE: "1"
```

Four of those are worth understanding rather than copying:

<AccordionGroup>
  <Accordion title="FORGE_INGEST_URL uses the in-cluster service name" icon="network">
    Training pods report back to it. The external ingress address would leave the cluster and come
    back in; the service DNS name does not. Never `127.0.0.1` — that is the worker's own loopback.
  </Accordion>

  <Accordion title="FORGE_KUBERAY_RAY_VERSION must match the training image" icon="triangle-alert">
    A mismatch presents as "the cluster came up but workers never registered", and the only clue is a
    single version warning easily lost in startup output.
  </Accordion>

  <Accordion title="FORGE_K8S_SHM_SIZE, because 64MB is the container default" icon="cpu">
    Ray's object store lives in `/dev/shm`. At the default, anything but a tiny batch OOMs
    immediately. It is RAM-backed, so this plus the memory limit must stay under physical memory.
  </Accordion>

  <Accordion title="FORGE_KUBERAY_PRERUNNING_DEADLINE_S bounds a stuck job" icon="clock">
    Without it, an unpullable image or a nodeSelector nothing satisfies leaves a job Pending forever
    while holding its queue slot.
  </Accordion>
</AccordionGroup>

## Secrets

`secret.example.yaml` exists to show the shape. Replace it with a real source — `kubectl create
secret`, Sealed Secrets, or the External Secrets Operator — and remove it from
`kustomization.yaml` before production.

At minimum: `FORGE_WEB_JWT_SECRET`, `FORGE_DB_URL`, `FORGE_REDIS_URL`, `FORGE_S3_SECRET_KEY`,
`FORGE_SECRET_ENC_KEY`, and the OIDC client secret.

## Replicas

The manifest ships one replica, and that is the safe default. Before scaling up:

* `FORGE_DB_URL` must point at Postgres. SQLite is single-writer
* `FORGE_WEB_JWT_SECRET` must be fixed, or replicas reject each other's tokens
* `FORGE_REDIS_URL` must be set, or **background roles stop entirely** rather than run unsafely
  across replicas — storage accounting, diagnosis, the watchdog and the daily report all go quiet
* Run the background roles in their own pod. Set `FORGE_INPROCESS_WORKERS=0` and apply the worker
  Deployment — `worker.yaml` in the kustomize path, `worker.enabled=true` in the chart. The
  diagnosis and daily-report ticks call an LLM and hold the event loop for seconds at a time; out
  of process they stop inflating API tail latency, and the console's replicas become stateless.
  One replica is correct: every role elects a leader through Redis, so a second one idles

## Confirm it worked

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
kubectl -n starforge get pods
kubectl -n starforge port-forward svc/starforge-console 8080:80
curl -s localhost:8080/api/version
```

Then check the executor's own view — it is what tells you whether the storage claim is really RWX:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
curl -s localhost:8080/api/cluster/health | jq .storage
```