Operational troubleshooting

The failures that look like something other than what they are.

Ordered by how often the symptom points somewhere other than the cause.

Jobs queue forever with free GPUs on the dashboard

Check	Because
The user’s quota	Concurrent GPUs, concurrent jobs, or daily GPU-hours. A queued job was admitted; a refused one was not
Scheduling windows	FORGE_SCHEDULE_ENFORCE with a closed window for that series
Disk watermark	FORGE_DISK_WATERMARK_PCT stops dequeuing entirely to protect running jobs
Starvation	A large job repeatedly denied while small ones behind it take the freed capacity. FORGE_SCHED_RESERVE_AFTER_S is what stops this; if it is 0 the queue starves large jobs indefinitely

The starvation case is the one that looks most like a bug. The console shows “waiting for capacity” forever, and the platform itself generates the stream that causes it — lifecycle jobs and Playground sessions each take one GPU, so a four-GPU job is skipped every pass.

Training starts but is absurdly slow, or hangs in all-reduce

FORGE_K8S_NCCL_SOCKET_IFNAME is unset on a multi-NIC machine, so NCCL picked the wrong interface. There is no error — only bad throughput or a hang. Common values are bond0, eth0, ib0.

The cluster comes up but workers never register

FORGE_KUBERAY_RAY_VERSION does not match the Ray inside the training image. The only clue is one version-mismatch warning line, easily lost in startup output.

Jobs OOM on anything but a tiny batch

/dev/shm is at the container default of 64MB. Ray’s object store lives there. FORGE_K8S_SHM_SIZE or FORGE_LOCAL_SHM_SIZE, and remember it is RAM-backed — it plus the memory limit must stay under physical memory.

A multi-node run cannot be resumed

FORGE_K8S_STORAGE_SHARED is true but the PVC is RWO. An RWO claim does not fail the job — it gives each pod its own volume, so checkpoint shards split across nodes. The executor’s health check verifies the declaration against the live PVC; read it.

Every background task is stopped

No Redis. Background roles need a shared lock so several replicas do not run the same task at once. Single instance: acceptable. Multiple replicas: storage accounting, diagnosis and the daily report are not happening.

Weight downloads stall, or fail on a 400 with only a URL in the body

The mirror, or Xet. See air-gapped deployments.

Logs stream but charts stay empty

Ingest. Check FORGE_INGEST_URL is reachable from a worker node, not from your laptop — 127.0.0.1 there is the worker’s own loopback. For a custom trainer, check that starforge.report is actually called.

Everyone is logged out after a restart

FORGE_WEB_JWT_SECRET is unset, so each process generates its own. Set a fixed strong random value. With more than one replica this also means tokens signed by one are rejected by the others.

GPUs are permanently “unschedulable”

Health checks found pending ECC retirements, uncorrected errors, or hardware throttling — or a process outside the platform is holding the card. The capacity view names which. These faults raise no error anywhere; they produce bad results or a silent slowdown, which is why the check exists.

Storage quotas report “not measured”

FORGE_STORAGE_SCAN_INTERVAL_S is 0, or the scan role is not running. A full scan takes minutes on a large tree, which is why it is a background role rather than computed on request.

An environment is refused at admission

Either it needs a sandbox and none is configured, or it is openenv-image — which this deployment does not host, deliberately. Refusing at admission rather than in the launcher means you find out before an allocation is spent.