> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Operational troubleshooting

> The failures that look like something other than what they are.

Ordered by how often the symptom points somewhere other than the cause.

## Jobs queue forever with free GPUs on the dashboard

| Check              | Because                                                                                                                                                                                  |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The user's quota   | Concurrent GPUs, concurrent jobs, or daily GPU-hours. A queued job was admitted; a refused one was not                                                                                   |
| Scheduling windows | `FORGE_SCHEDULE_ENFORCE` with a closed window for that series                                                                                                                            |
| Disk watermark     | `FORGE_DISK_WATERMARK_PCT` stops dequeuing entirely to protect running jobs                                                                                                              |
| Starvation         | A large job repeatedly denied while small ones behind it take the freed capacity. `FORGE_SCHED_RESERVE_AFTER_S` is what stops this; if it is 0 the queue starves large jobs indefinitely |

The starvation case is the one that looks most like a bug. The console shows "waiting for capacity"
forever, and the platform itself generates the stream that causes it — lifecycle jobs and Playground
sessions each take one GPU, so a four-GPU job is skipped every pass.

## Training starts but is absurdly slow, or hangs in all-reduce

`FORGE_K8S_NCCL_SOCKET_IFNAME` is unset on a multi-NIC machine, so NCCL picked the wrong interface.
There is no error — only bad throughput or a hang. Common values are `bond0`, `eth0`, `ib0`.

## The cluster comes up but workers never register

`FORGE_KUBERAY_RAY_VERSION` does not match the Ray inside the training image. The only clue is one
version-mismatch warning line, easily lost in startup output.

## Jobs OOM on anything but a tiny batch

`/dev/shm` is at the container default of 64MB. Ray's object store lives there.
`FORGE_K8S_SHM_SIZE` or `FORGE_LOCAL_SHM_SIZE`, and remember it is RAM-backed — it plus the memory
limit must stay under physical memory.

## A multi-node run cannot be resumed

`FORGE_K8S_STORAGE_SHARED` is true but the PVC is RWO. An RWO claim does not fail the job — it gives
each pod its own volume, so checkpoint shards split across nodes. The executor's health check
verifies the declaration against the live PVC; read it.

## Every background task is stopped

No Redis. Background roles need a shared lock so several replicas do not run the same task at once.
Single instance: acceptable. Multiple replicas: storage accounting, diagnosis and the daily report
are not happening.

## Weight downloads stall, or fail on a 400 with only a URL in the body

The mirror, or Xet. See [air-gapped deployments](/en/ops/airgapped).

## Logs stream but charts stay empty

Ingest. Check `FORGE_INGEST_URL` is reachable *from a worker node*, not from your laptop —
`127.0.0.1` there is the worker's own loopback. For a custom trainer, check that
[`starforge.report`](/en/api-reference/python-sdk) is actually called.

## Everyone is logged out after a restart

`FORGE_WEB_JWT_SECRET` is unset, so each process generates its own. Set a fixed strong random value.
With more than one replica this also means tokens signed by one are rejected by the others.

## GPUs are permanently "unschedulable"

Health checks found pending ECC retirements, uncorrected errors, or hardware throttling — or a
process outside the platform is holding the card. The capacity view names which. These faults raise
no error anywhere; they produce bad results or a silent slowdown, which is why the check exists.

## Storage quotas report "not measured"

`FORGE_STORAGE_SCAN_INTERVAL_S` is 0, or the scan role is not running. A full scan takes minutes on a
large tree, which is why it is a background role rather than computed on request.

## An environment is refused at admission

Either it needs a sandbox and none is configured, or it is `openenv-image` — which this deployment
does not host, deliberately. Refusing at admission rather than in the launcher means you find out
before an allocation is spent.