Skip to main content

​
Login & Authentication

Add --device-flow to log in with a device code: the CLI shows a short code; open the prompted URL in any device’s browser and enter it. For CI, use sf login --token <service token>.
Run sf login again. Login state lives in ~/.forge/; multiple machines log in independently without affecting each other. A disabled account also shows up as 401 — check your account status first.

​
Rejected Submissions

The experiment lock has drifted from the server-side catalog (the platform published a new recipe). Use sf recipe status <exp> to see the differences, sf recipe upgrade <exp> to upgrade, or add --upgrade-recipe when submitting.
The platform requires submissions to be traceable to an exact commit. Run git commit and retry; if you really need to submit with dirty changes, use --allow-dirty (untracked files are listed as warnings — be careful not to include large or sensitive files).
Fix each reported error: misspelled keys (struct mode does not allow new keys), out-of-range values, batch-size divisibility. After fixing, confirm locally with sf validate <exp> before resubmitting.
The config references a gated model / dataset your HF account is not authorized for, or the dataset id is misspelled (the full org/name is required). Request access on the HF website first, then confirm your account is linked on the console’s HuggingFace page.
A custom --image may use a tag; pin @sha256:… for production. The registry host must be on FORGE_ALLOWED_IMAGE_REGISTRIES. An empty list rejects custom user images entirely. Custom images.
Catalog custom/custom is external observability. Pass --observability-url on submit. Console charts still need starforge.report (install starforge-core in the image). Custom training.
The scaffold script prints that you have not filled in a training command. Replace it. Working directory is FORGE_WORK_DIR; run "${FORGE_EXP_DIR}/train.py".
The submission waits in the queue instead of failing; check current usage with sf status. If you’re in a hurry, free capacity by stopping your own older jobs with sf job stop, or ask an admin to adjust your quota.

​
Job Issues

Check in order: quota usage (sf status), idle GPUs on the cluster (console dashboard), whether the schedule window is closed (jobs are held when the remaining window is below the threshold), and whether maintenance mode is on.
Usually a slow image pull / no node satisfying the nodeSelector / not enough GPUs. The Events section of the job details shows the raw K8s Events; the job is automatically marked failed once the preRunning deadline is exceeded.
Check the Logs tab first. Training may still be loading the model. If logs look fine and metrics are missing, FORGE_INGEST_URL must be reachable from the training container (not 127.0.0.1). Custom jobs also need starforge.report in the training code; see Custom training.
This is automatic retry (within budget, with a cooldown). The retry count is visible in the job details; if you don’t want retries, manually stop the job after it fails.
When the job’s failure reason is marked “OOM killer”: lower train_micro_batch_size, enable activation checkpointing, lower vLLM gpu_memory_utilization, or switch to a larger-shape profile. The AI analysis in the Diagnostics tab usually suggests the exact changes.

​
Logs & Observability

On SSE disconnects the CLI automatically reconnects and resumes; for persistent drops check your reverse proxy config (nginx needs buffering disabled for /api). Historical logs can be replayed anytime: sf job logs <ID> -n 0.
Confirm the method has a validation phase (val_period > 0) and a validation step has been reached; GRPO/PPO samples are generated during validation rounds, so being empty early in training is normal.
Still stuck? Open the job details in the console → Diagnostics to run an AI diagnosis, or hand the run id to an admin to check the server-side logs.