> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Diagnosing common issues with login, submission, scheduling, training, and logs

## Login & Authentication

<AccordionGroup>
  <Accordion title="sf login can't open a browser / SSH environments">
    Add `--device-flow` to log in with a device code: the CLI shows a short code; open the prompted URL in any device's browser and enter it. For CI, use `sf login --token <service token>`.
  </Accordion>

  <Accordion title="401 / expired token">
    Run `sf login` again. Login state lives in `~/.forge/`; multiple machines log in independently without affecting each other. A disabled account also shows up as 401 — check your account status first.
  </Accordion>
</AccordionGroup>

## Rejected Submissions

<AccordionGroup>
  <Accordion title="&#x22;Recipe exact-contract mismatch / runtime_id incompatible&#x22;">
    The experiment lock has drifted from the server-side catalog (the platform published a new recipe). Use `sf recipe status <exp>` to see the differences, `sf recipe upgrade <exp>` to upgrade, or add `--upgrade-recipe` when submitting.
  </Accordion>

  <Accordion title="&#x22;Workspace has uncommitted changes&#x22;">
    The platform requires submissions to be traceable to an exact commit. Run `git commit` and retry; if you really need to submit with dirty changes, use `--allow-dirty` (untracked files are listed as warnings — be careful not to include large or sensitive files).
  </Accordion>

  <Accordion title="&#x22;Config validation failed&#x22;">
    Fix each reported error: misspelled keys (struct mode does not allow new keys), out-of-range values, batch-size divisibility. After fixing, confirm locally with `sf validate <exp>` before resubmitting.
  </Accordion>

  <Accordion title="&#x22;HuggingFace resource preflight failed&#x22;">
    The config references a gated model / dataset your HF account is not authorized for, or the dataset id is misspelled (the full `org/name` is required). Request access on the HF website first, then confirm your account is linked on the console's HuggingFace page.
  </Accordion>

  <Accordion title="&#x22;Image registry not on the allowlist&#x22; (custom)">
    A custom `--image` may use a tag; pin `@sha256:…` for production. The registry host must be on `FORGE_ALLOWED_IMAGE_REGISTRIES`. An empty list rejects custom user images entirely. [Custom images](/en/guides/custom-images).
  </Accordion>

  <Accordion title="custom external observability 要求 spec.framework.observability_url">
    Catalog `custom/custom` is `external` observability. Pass `--observability-url` on submit. Console charts still need `starforge.report` (install `starforge-core` in the image). [Custom training](/en/guides/custom-training).
  </Accordion>

  <Accordion title="train.sh still exits 1 / 2">
    The scaffold script prints that you have not filled in a training command. Replace it. Working directory is `FORGE_WORK_DIR`; run `"${FORGE_EXP_DIR}/train.py"`.
  </Accordion>

  <Accordion title="Insufficient quota">
    The submission waits in the queue instead of failing; check current usage with `sf status`. If you're in a hurry, free capacity by stopping your own older jobs with `sf job stop`, or ask an admin to adjust your quota.
  </Accordion>
</AccordionGroup>

## Job Issues

<AccordionGroup>
  <Accordion title="Stuck in QUEUED, never dequeued">
    Check in order: quota usage (`sf status`), idle GPUs on the cluster (console dashboard), whether the schedule window is closed (jobs are held when the remaining window is below the threshold), and whether maintenance mode is on.
  </Accordion>

  <Accordion title="Stuck in PENDING (kuberay)">
    Usually a slow image pull / no node satisfying the nodeSelector / not enough GPUs. The **Events** section of the job details shows the raw K8s Events; the job is automatically marked failed once the preRunning deadline is exceeded.
  </Accordion>

  <Accordion title="RUNNING but no data in the charts">
    Check the **Logs** tab first. Training may still be loading the model. If logs look fine and metrics are missing, `FORGE_INGEST_URL` must be reachable from the training container (not 127.0.0.1). Custom jobs also need `starforge.report` in the training code; see [Custom training](/en/guides/custom-training).
  </Accordion>

  <Accordion title="A new job appears automatically after FAILED">
    This is automatic retry (within budget, with a cooldown). The retry count is visible in the job details; if you don't want retries, manually stop the job after it fails.
  </Accordion>

  <Accordion title="OOM / container killed">
    When the job's failure reason is marked "OOM killer": lower `train_micro_batch_size`, enable activation checkpointing, lower vLLM `gpu_memory_utilization`, or switch to a larger-shape profile. The AI analysis in the **Diagnostics** tab usually suggests the exact changes.
  </Accordion>
</AccordionGroup>

## Logs & Observability

<AccordionGroup>
  <Accordion title="sf job logs stream drops">
    On SSE disconnects the CLI automatically reconnects and resumes; for persistent drops check your reverse proxy config (nginx needs buffering disabled for `/api`). Historical logs can be replayed anytime: `sf job logs <ID> -n 0`.
  </Accordion>

  <Accordion title="Validation samples are empty">
    Confirm the method has a validation phase (`val_period` > 0) and a validation step has been reached; GRPO/PPO samples are generated during validation rounds, so being empty early in training is normal.
  </Accordion>
</AccordionGroup>

Still stuck? Open the job details in the console → **Diagnostics** to run an AI diagnosis, or hand the run id to an admin to check the server-side logs.