> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Configuration reference

> Every setting a StarForge deployment reads, with its type, default, and what it does.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
export FORGE_DEFAULT_FLEET_KIND=kuberay
export FORGE_STORAGE_ROOT=/mnt/shared/starforge
export FORGE_WEB_JWT_SECRET=$(openssl rand -hex 32)
```

Every setting is an environment variable prefixed `FORGE_`. There is no configuration file — a
deployment is described entirely by its environment, which is what makes the same image work under
Docker Compose, Kubernetes and systemd without a translation layer.

<Info>
  This page is generated from `server.core.config.WebSettings` by `scripts/gen_settings_docs.py`, and
  CI fails if a setting is added without a description. If a knob exists, it is on this page.
</Info>

## The five you cannot skip

Everything else has a working default. These do not.

<ParamField path="FORGE_DEFAULT_FLEET_KIND" type="local | node | kuberay | slurm" default="local">
  The backend of the Fleet seeded when the console has none yet — read once, at first start. After
  that the backend is a property of the Fleet a workload was placed on, and registering a second
  Fleet is how a console gains another. There is no silent fallback between backends, so set this
  explicitly even when you want `local`. See [backends](/en/ops/overview).
</ParamField>

<ParamField path="FORGE_STORAGE_ROOT" type="path">
  The one path a deployment configures. It must be an absolute path, and it must resolve identically
  on the console, on every node, and inside every container. Empty means single-machine development.
</ParamField>

<ParamField path="FORGE_WEB_JWT_SECRET" type="string">
  Generated per process when unset, which logs everyone out on restart and breaks multi-replica
  deployments outright. Set a fixed strong random value for anything but a laptop.
</ParamField>

<ParamField path="FORGE_DB_URL" type="url">
  Points at Postgres for a team deployment. Left empty it uses a local SQLite file, which is correct
  for one machine and wrong for two.
</ParamField>

<ParamField path="FORGE_PUBLIC_URL" type="url">
  The https address users reach this deployment at. The OIDC redirect URI is built from it, so single
  sign-on cannot work without it.
</ParamField>

## Reading the tables

* **Required** means the deployment refuses to start without it.
* **generated** means a value is derived per process when you do not supply one.
* A dash means the default is empty, and empty usually means the feature is off.

Absolute paths are validated at startup: a relative `FORGE_STORAGE_ROOT` is refused rather than
resolved against whatever directory the process happened to start in.

## Core

| Environment variable              | Type    | Default                | Description                                                                                                                                                                                                                                                                                                                                                                    |
| --------------------------------- | ------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `FORGE_CACHE_TTL`                 | `float` | `5.0`                  | How long in-process caches hold a value before re-reading it.                                                                                                                                                                                                                                                                                                                  |
| `FORGE_COOKIE_SECURE`             | `bool`  | —                      | Whether the session cookie carries the Secure flag. Follows whether public\_url is https when unset; set it false to debug over plain http locally.                                                                                                                                                                                                                            |
| `FORGE_DB_PATH`                   | `Path`  | `.forge/web.db`        | Where SQLite lives, relative to the console's working directory. Deliberately not under the storage root: that root is usually a shared filesystem, and SQLite over NFS has well-known locking problems. Unused once db\_url points at Postgres.                                                                                                                               |
| `FORGE_DB_URL`                    | `str`   | —                      | Database URL. Empty falls back to the SQLite file at db\_path. A team deployment points this at Postgres.                                                                                                                                                                                                                                                                      |
| `FORGE_GITHUB_ALLOWED_ORGS`       | `str`   | —                      | Comma-separated organization logins whose members may sign in. Empty means any GitHub account can create an account here. Setting it also makes the consent screen ask for `read:org`; a member whose membership is private and unshared is refused.                                                                                                                           |
| `FORGE_GITHUB_CLIENT_ID`          | `str`   | —                      | OAuth app client ID from GitHub. Setting it and the secret puts a "Sign in with GitHub" button on the login page, with no restart. The callback URL to register is `&lt;public URL&gt;/api/auth/social/github/callback`.                                                                                                                                                       |
| `FORGE_GITHUB_CLIENT_SECRET`      | `str`   | —                      | OAuth app client secret from GitHub. Stored encrypted and never returned by the API.                                                                                                                                                                                                                                                                                           |
| `FORGE_GOOGLE_ALLOWED_DOMAINS`    | `str`   | —                      | Comma-separated email domains allowed to sign in with Google. Empty means any Google account can create an account here, which on a console reachable from the internet is open registration. Both the verified email domain and the Workspace `hd` claim are accepted.                                                                                                        |
| `FORGE_GOOGLE_CLIENT_ID`          | `str`   | —                      | OAuth client ID from the Google Cloud console. Setting it and the secret puts a "Sign in with Google" button on the login page, with no restart. The redirect URI to register is `&lt;public URL&gt;/api/auth/social/google/callback`.                                                                                                                                         |
| `FORGE_GOOGLE_CLIENT_SECRET`      | `str`   | —                      | OAuth client secret from the Google Cloud console. Stored encrypted and never returned by the API.                                                                                                                                                                                                                                                                             |
| `FORGE_GOOGLE_ONE_TAP`            | `bool`  | `false`                | Show Google's One Tap prompt on the login page, which offers to sign in with an account the browser is already signed into. Off by default because the prompt appears without the visitor asking for it. Requires the Google client ID and secret.                                                                                                                             |
| `FORGE_HOST`                      | `str`   | `127.0.0.1`            | Address the console binds to. The CLI's --host overrides it.                                                                                                                                                                                                                                                                                                                   |
| `FORGE_INGEST_URL`                | `str`   | —                      | Where training processes POST their logs and metrics. With a remote cluster, do not use 127.0.0.1 — that is the worker's own loopback. Give the console URL as reachable from inside the cluster network.                                                                                                                                                                      |
| `FORGE_JWT_HOURS`                 | `int`   | `8`                    | How long an access token stays valid.                                                                                                                                                                                                                                                                                                                                          |
| `FORGE_NO_AUTH`                   | `bool`  | `false`                | Treat every request as an admin and skip authentication entirely. Development only — never set this on anything reachable by other people.                                                                                                                                                                                                                                     |
| `FORGE_OIDC_CLIENT_ID`            | `str`   | —                      | Client id registered with the identity provider.                                                                                                                                                                                                                                                                                                                               |
| `FORGE_OIDC_CLIENT_SECRET`        | `str`   | —                      | Client secret registered with the identity provider.                                                                                                                                                                                                                                                                                                                           |
| `FORGE_OIDC_DEFAULT_ROLE`         | `str`   | `operator`             | Role given to a user on their first single sign-on login. Not admin: the first administrator should be somebody a person deliberately promoted.                                                                                                                                                                                                                                |
| `FORGE_OIDC_DISCOVERY_URL`        | `str`   | —                      | Explicit discovery document URL. Empty derives one by appending /.well-known/openid-configuration to the issuer; Casdoor serves a per-application path and needs this set.                                                                                                                                                                                                     |
| `FORGE_OIDC_GROUP_CLAIM`          | `str`   | —                      | The claim holding the user's groups, read as team membership on every sign-in. Empty (the default) means the platform never touches team membership -- which is what a deployment that manages teams by hand needs, since reconciling against an absent claim would empty everybody out. A group the directory names but the platform does not have is skipped, never created. |
| `FORGE_OIDC_GROUP_PREFIX`         | `str`   | —                      | Stripped from the front of each group before it is read as a team key. Directories hand out paths and DNs where the platform wants a key.                                                                                                                                                                                                                                      |
| `FORGE_OIDC_INSECURE_SKIP_VERIFY` | `bool`  | `false`                | Skip TLS verification against the identity provider. For debugging a self-signed certificate on an internal network; never in production.                                                                                                                                                                                                                                      |
| `FORGE_OIDC_ISSUER`               | `str`   | —                      | OIDC issuer URL, for example a Casdoor application. Setting it turns on single sign-on.                                                                                                                                                                                                                                                                                        |
| `FORGE_OIDC_SCOPE`                | `str`   | `openid profile email` | Scopes requested at login.                                                                                                                                                                                                                                                                                                                                                     |
| `FORGE_PORT`                      | `int`   | `8080`                 | Port the console listens on. The CLI's --port overrides it.                                                                                                                                                                                                                                                                                                                    |
| `FORGE_PUBLIC_URL`                | `str`   | —                      | The address users and identity providers reach this deployment at. Used to build the OIDC redirect URI, so a team deployment must set it to the real https domain.                                                                                                                                                                                                             |
| `FORGE_REDIS_PREFIX`              | `str`   | `lab`                  | Key namespace, so several environments can share one Redis without colliding.                                                                                                                                                                                                                                                                                                  |
| `FORGE_REDIS_URL`                 | `str`   | —                      | Redis for shared cache, distributed locks, rate limiting and immediate token revocation. Empty disables all four and the console behaves as a single instance.                                                                                                                                                                                                                 |
| `FORGE_REFRESH_DAYS`              | `int`   | `30`                   | How long a refresh token stays valid. It is delivered only as an httpOnly cookie, so browser JavaScript cannot read it.                                                                                                                                                                                                                                                        |
| `FORGE_REPO_ROOT`                 | `Path`  | *generated*            | Project root the console reads experiment definitions and local bookkeeping from. Defaults to the working directory.                                                                                                                                                                                                                                                           |
| `FORGE_SERVE`                     | `bool`  | `false`                | Bind 0.0.0.0 instead of loopback. Turn it on for a team or container deployment.                                                                                                                                                                                                                                                                                               |
| `FORGE_STATIC_DIR`                | `Path`  | —                      | Directory holding the built console assets (web/dist). The server wheel ships them, so this is normally left alone.                                                                                                                                                                                                                                                            |
| `FORGE_WEB_JWT_SECRET`            | `str`   | *generated*            | Signing key for access tokens. Generated per process when unset, which is fine on one machine and wrong for anything else: two replicas would sign with different keys, so every restart logs everyone out.                                                                                                                                                                    |

## Proxied submission

| Environment variable    | Type  | Default | Description                                                                                                                                                                                     |
| ----------------------- | ----- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FORGE_CLUSTER_PROFILE` | `str` | —       | Default CLUSTER\_PROFILE forwarded to jobs. A profile named by the client is recorded but does not override this.                                                                               |
| `FORGE_NEMO_RL_DIR`     | `str` | —       | Path to NeMo-RL inside the training container, forwarded to the job as NEMO\_RL\_DIR. Required to submit NeMo-RL jobs. This is a path in the image, not on the host and not in the CLI project. |

## Storage

| Environment variable               | Type    | Default        | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| ---------------------------------- | ------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FORGE_AUDIT_RETENTION_DAYS`       | `int`   | `0`            | How long an audit row is kept, in days. 0 keeps them forever, which is the default. Pruning does not break the tamper-evident chain: each run records the range it removed and the hash of the last row it deleted, and the verification endpoint accepts a gap that such a record explains — those records are themselves never deleted. Enforcing a retention period is a commercial feature.                                                                                                     |
| `FORGE_AUDIT_RETENTION_INTERVAL_S` | `float` | `86400.0`      | How often the audit retention task runs.                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `FORGE_CLUSTER_SECRETS_FILE`       | `str`   | —              | Path to a secrets file inside the container. When set, the path is forwarded and the secret values never enter the Ray dashboard.                                                                                                                                                                                                                                                                                                                                                                   |
| `FORGE_DATA_SCAN_ALLOW_TERMS`      | `str`   | —              | Comma-separated terms to suppress as known false positives.                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `FORGE_DATA_SCAN_DENY_TERMS`       | `str`   | —              | Comma-separated terms this deployment treats as sensitive, in addition to the built-in rules. Terms, never patterns: an operator pasting a regex into a settings box is how a config change becomes a backtrack that hangs a worker.                                                                                                                                                                                                                                                                |
| `FORGE_DATA_SCAN_POLICY`           | `str`   | `report`       | What the platform does about sensitive values found in a dataset. `off` does not scan; `report` scans, records and shows the findings while admitting the job anyway; `block` refuses a submission whose data has a high-confidence finding. Blocking keys on severity, never on any finding — the low-confidence rules hit every base64 blob and git SHA in a code corpus.                                                                                                                         |
| `FORGE_INGEST_TOKEN_DAYS`          | `int`   | `30`           | How long a job's ingest token stays valid. It has to outlast the longest training run you expect, or a long job loses the ability to report partway through.                                                                                                                                                                                                                                                                                                                                        |
| `FORGE_JOB_RUNNER_MODE`            | `str`   | `bundled`      | How the platform runtime reaches a job. `bundled` injects a content-addressed PEX, so the training image needs nothing. `image-installed` requires starforge preinstalled in the image and exists only as a migration rollback — there is never an automatic fallback between the two.                                                                                                                                                                                                              |
| `FORGE_JOB_RUNNER_PATH`            | `str`   | —              | Where the bundled runner PEX is read from. Empty means \<repo\_root>/dist/starforge-runner.pex.                                                                                                                                                                                                                                                                                                                                                                                                     |
| `FORGE_JOB_RUNNER_PYTHON`          | `str`   | —              | Deployment-level override for the interpreter the runner uses. Empty uses whatever the runtime artifact declares.                                                                                                                                                                                                                                                                                                                                                                                   |
| `FORGE_JOB_RUNNER_REQUIRES_PYTHON` | `str`   | `>=3.10,<3.14` | Python versions the bundled runner supports. A training image outside this range is refused rather than failing later.                                                                                                                                                                                                                                                                                                                                                                              |
| `FORGE_LICENSE_FILE`               | `str`   | —              | A file holding the licence instead. Useful because a licence is a few hundred bytes nobody wants in a Helm values file or a shell history, and mounting a secret as a file is how Kubernetes prefers to do this. `license_key` wins if both are set.                                                                                                                                                                                                                                                |
| `FORGE_LICENSE_KEY`                | `str`   | —              | The commercial licence, as the signed string the vendor issued. Empty runs the community edition, which is a supported configuration rather than a degraded one -- everything in it works and keeps working. Verified offline: there is no call home, ever. Settable from the admin console, and it takes effect on the next request rather than needing a restart, because entitlement is checked per request rather than at mount time.                                                           |
| `FORGE_MAX_UPLOAD_MB`              | `int`   | `1024`         | Size ceiling for a submitted job package, compressed. The uncompressed ceiling is fifty times this, which is what stops a compression bomb.                                                                                                                                                                                                                                                                                                                                                         |
| `FORGE_PASSTHROUGH_ENV`            | `dict`  | *generated*    | Non-secret environment variables forwarded into every job, such as an HF endpoint mirror.                                                                                                                                                                                                                                                                                                                                                                                                           |
| `FORGE_QUEUE_POLL_INTERVAL`        | `float` | `5.0`          | How often the dequeue loop looks for work.                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `FORGE_QUOTA_ENFORCE`              | `bool`  | `true`         | Whether the soft quota actually blocks admission. Turning it off records usage without refusing anything.                                                                                                                                                                                                                                                                                                                                                                                           |
| `FORGE_SERVER_SECRETS_FILE`        | `str`   | —              | Server-side KEY=VALUE secrets file, injected into jobs when cluster\_secrets\_file is not used.                                                                                                                                                                                                                                                                                                                                                                                                     |
| `FORGE_STORAGE_ROOT`               | `str`   | —              | The one path a deployment configures. Everything the platform writes derives from it: caches, run directories, job packages and control-plane state. The invariant is that this path is identical on the console, on every node, and inside every container — a shared filesystem or a same-path bind mount satisfies it. Empty means single-machine development, falling back to ./.forge/storage.                                                                                                 |
| `FORGE_SUBMIT_DRY_RUN`             | `bool`  | `false`        | Assemble the runtime environment but never actually submit to Ray. Debugging aid.                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `FORGE_VOLUME_ROOT`                | `str`   | —              | Root of the governed Volumes a job may read, distinct from its training data. Must sit outside the storage root, which is checked at startup - that root is mounted writable into job containers, so a volume beneath it would be reachable by a second, writable path and the read-only mount would be decorative. Mount JuiceFS here, with the S3 settings pointed at its gateway, to give one content both an object API and a filesystem. Empty means this deployment does not support volumes. |

## Automatic retry

| Environment variable       | Type    | Default   | Description                                                                                                                                                                                                   |
| -------------------------- | ------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FORGE_AUTO_RETRY_DELAY_S` | `float` | `60.0`    | How long to wait after a failure before resubmitting, so a transient fault has time to clear.                                                                                                                 |
| `FORGE_AUTO_RETRY_ENABLED` | `bool`  | `false`   | Whether a failed job is resubmitted automatically, resuming from its last checkpoint under the same run id. Off by default — turn it on deliberately.                                                         |
| `FORGE_AUTO_RETRY_MAX`     | `int`   | `1`       | How many automatic retries a job gets. A job can override it with its own max\_retries.                                                                                                                       |
| `FORGE_AUTO_RETRY_ON`      | `str`   | `started` | Which failures are retried. `started` retries only jobs that reached RUNNING — a configuration or environment error usually fails before that, and retrying it just fails again. `all` retries every failure. |

## LLM judge

| Environment variable        | Type    | Default | Description                                                                                                                                                                                             |
| --------------------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FORGE_JUDGE_CACHE_ENABLED` | `bool`  | `true`  | Cache deterministic scores (temperature at or below zero) so the same prompt is not paid for twice.                                                                                                     |
| `FORGE_JUDGE_ENABLED`       | `bool`  | `false` | Whether training jobs get a judge endpoint and token injected, so a reward function can score with an LLM against a Rubric. The upstream credential and the audit trail both stay on the platform side. |
| `FORGE_JUDGE_LLM_API_KEY`   | `str`   | —       | API key for the judge's upstream. Empty reuses agent\_llm\_api\_key.                                                                                                                                    |
| `FORGE_JUDGE_LLM_BASE_URL`  | `str`   | —       | OpenAI-compatible upstream the judge calls. Empty reuses the diagnosis agent's LLM configuration.                                                                                                       |
| `FORGE_JUDGE_LLM_MODEL`     | `str`   | —       | Model the judge scores with. Empty reuses agent\_llm\_model.                                                                                                                                            |
| `FORGE_JUDGE_LLM_TIMEOUT`   | `float` | `60.0`  | Request timeout for one judge call.                                                                                                                                                                     |

## Agent sandbox

| Environment variable                  | Type    | Default       | Description                                                                                                                                                                                                                                                                                                                                                         |
| ------------------------------------- | ------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FORGE_ENVIRONMENT_ALLOWED_HOSTS`     | `str`   | —             | Hosts a job may reach for an `openenv-remote` environment, comma separated. Empty means none, deliberately: a training container usually has the run of the internal network, so accepting whatever host a manifest names would turn an environment reference into an outbound channel written by somebody else.                                                    |
| `FORGE_ENVIRONMENT_SELF_REGISTRATION` | `bool`  | `false`       | Whether an environment may authorise its own host by proving one answers there. Off by default, because turning it on moves the egress judgement from "an administrator named this host" to "an OpenEnv server answered here when the version was sealed, and we recorded who asked" — a weaker claim about the host and a much stronger one about the audit trail. |
| `FORGE_REFLOW_ENABLED`                | `bool`  | `true`        | Whether the Reflow background task runs. It is on by default and does nothing until a deployment sets a capture sample rate above zero — capture is enabled per deployment, never globally.                                                                                                                                                                         |
| `FORGE_REFLOW_FLUSH_INTERVAL_S`       | `int`   | `60`          | How often captured traffic is flushed from memory into the buffer.                                                                                                                                                                                                                                                                                                  |
| `FORGE_SANDBOX_CPUS`                  | `str`   | `1`           | CPU limit for one sandbox container.                                                                                                                                                                                                                                                                                                                                |
| `FORGE_SANDBOX_ENDPOINT`              | `str`   | —             | An external E2B-compatible sandbox service to use instead of the built-in container.                                                                                                                                                                                                                                                                                |
| `FORGE_SANDBOX_IMAGE`                 | `str`   | —             | Image for the first-party code sandbox. Off unless configured: running model-generated code is not something to enable by default, and there is no sensible image to guess.                                                                                                                                                                                         |
| `FORGE_SANDBOX_MAX_CONCURRENCY`       | `int`   | `8`           | How many sandboxes may run at once.                                                                                                                                                                                                                                                                                                                                 |
| `FORGE_SANDBOX_MEMORY`                | `str`   | `512m`        | Memory limit for one sandbox container.                                                                                                                                                                                                                                                                                                                             |
| `FORGE_SANDBOX_TIMEOUT_S`             | `float` | `30.0`        | Wall-clock limit for one sandbox execution.                                                                                                                                                                                                                                                                                                                         |
| `FORGE_SANDBOX_TOKEN`                 | `str`   | —             | Bearer token for that external sandbox service.                                                                                                                                                                                                                                                                                                                     |
| `FORGE_SANDBOX_USER`                  | `str`   | `65534:65534` | User the sandbox process runs as. Non-root by default; change it only if the image needs a different unprivileged account.                                                                                                                                                                                                                                          |

## Playground

| Environment variable        | Type    | Default  | Description                                                                                                                                               |
| --------------------------- | ------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FORGE_PLAYGROUND_ENABLED`  | `bool`  | `false`  | Whether users may start Playground sessions to chat with a run's artifacts. The local backend only.                                                       |
| `FORGE_PLAYGROUND_IMAGE`    | `str`   | —        | Inference image for Playground sessions. Must be a digest-pinned OCI reference.                                                                           |
| `FORGE_PLAYGROUND_MAX_GPUS` | `int`   | `4`      | GPU ceiling for a single Playground session.                                                                                                              |
| `FORGE_PLAYGROUND_PORT_MAX` | `int`   | `18099`  | High end of that port range. It must not overlap the deployment range: where they overlap, a throwaway session can take the port a stable endpoint needs. |
| `FORGE_PLAYGROUND_PORT_MIN` | `int`   | `18000`  | Low end of the host port range Playground sessions claim, one port each.                                                                                  |
| `FORGE_PLAYGROUND_TTL_S`    | `float` | `3600.0` | How long an idle session lives before it is stopped and its GPUs released.                                                                                |

## Model deployments

| Environment variable                       | Type    | Default                 | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| ------------------------------------------ | ------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FORGE_ALLOWED_IMAGE_REGISTRIES`           | `str`   | —                       | Registry hosts a user may name with `sf submit --image`, comma separated and matched exactly. Empty leaves first-party framework overrides unrestricted but closes custom images entirely; a production deployment should configure it explicitly.                                                                                                                                                                                                                                                    |
| `FORGE_DEFAULT_FLEET_KIND`                 | `str`   | `local`                 | The backend of the Fleet seeded when a console has none yet. `local` is one container on the console host, `node` one container on a remote machine running the forgelet daemon, `kuberay` an ephemeral RayCluster per job, `slurm` an exclusive allocation through slurmrestd. Read once, at first start — after that the backend is a property of the Fleet a workload was placed on, so registering a second Fleet is how a console gains a backend, and changing this moves no existing workload. |
| `FORGE_DEPLOYMENT_ALLOW_TRUST_REMOTE_CODE` | `bool`  | `false`                 | Whether a deployment may load a model that executes code from its own repository. Off by default, because it means running code the model author wrote.                                                                                                                                                                                                                                                                                                                                               |
| `FORGE_DEPLOYMENT_HEALTH_TIMEOUT_S`        | `float` | `10.0`                  | Timeout for one health probe against a serving revision.                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `FORGE_DEPLOYMENT_MAX_CONCURRENT_REQUESTS` | `int`   | `0`                     | How many inference requests one deployment may have in flight through the console at once, or 0 for no cap. Backpressure rather than a quota - a saturated engine serves no faster for having more sockets open, and a refusal the caller can retry beats a queue that slows every request already in it.                                                                                                                                                                                             |
| `FORGE_DEPLOYMENT_MAX_GPUS`                | `int`   | `8`                     | GPU ceiling for one deployment revision.                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `FORGE_DEPLOYMENT_PORT_MAX`                | `int`   | `18299`                 | High end of that port range.                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `FORGE_DEPLOYMENT_PORT_MIN`                | `int`   | `18100`                 | Low end of the host port range deployments claim.                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `FORGE_DEPLOYMENT_READINESS_TIMEOUT_S`     | `float` | `900.0`                 | How long a new revision may take to become ready before it is judged failed. Large models loading from cold storage need the headroom.                                                                                                                                                                                                                                                                                                                                                                |
| `FORGE_DEPLOYMENT_SGLANG_IMAGE`            | `str`   | —                       | Serving image for SGLang deployments. Must be digest-pinned.                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `FORGE_DEPLOYMENT_SHARED_ROOTS`            | `str`   | —                       | Extra read-only paths a deployment may load model weights from, comma separated.                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `FORGE_DEPLOYMENT_VLLM_IMAGE`              | `str`   | —                       | Serving image for vLLM deployments. Must be digest-pinned.                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `FORGE_FORGELET_INDEX_URL`                 | `str`   | —                       | Package index the generated `install.sh` fetches the forgelet package from. Empty means PyPI; an intranet deployment points this at its own mirror, since the machine running the script may have no route to the public internet.                                                                                                                                                                                                                                                                    |
| `FORGE_IMAGE_NEMO_RL`                      | `str`   | —                       | Default image for NeMo-RL jobs. A tag is enough. A user's `--image` takes precedence.                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `FORGE_IMAGE_OPENRLHF`                     | `str`   | —                       | Default image for OpenRLHF jobs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `FORGE_IMAGE_TRL`                          | `str`   | —                       | Default image for TRL jobs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `FORGE_IMAGE_VERL`                         | `str`   | —                       | Default image for verl jobs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `FORGE_INPROCESS_WORKERS`                  | `bool`  | `true`                  | Whether background roles — dequeue, diagnosis, watchdog, daily report — run inside the web process. Turn it off to run them as separate processes (`python -m server.worker --role &lt;name&gt;`): the slow LLM-calling ticks stop occupying the event loop and inflating API tail latency, and the web replicas become stateless and freely scalable. On by default so a single-container deployment keeps working unchanged.                                                                        |
| `FORGE_MODEL_DEPLOYMENTS_ENABLED`          | `bool`  | `false`                 | Whether managed model deployments — long-lived serving for application traffic — are available at all.                                                                                                                                                                                                                                                                                                                                                                                                |
| `FORGE_NODE_ADDRESSES`                     | `str`   | —                       | Comma-separated node URLs, for example `http://gpu-1:7070,http://gpu-2:7070`. Requires the storage root to be mounted at the same path on the console and on every node.                                                                                                                                                                                                                                                                                                                              |
| `FORGE_NODE_LAUNCH_TIMEOUT_S`              | `float` | `120.0`                 | Timeout for a launch request, covering container start on a node that already has the image. A first-time image pull runs asynchronously and does not count against this.                                                                                                                                                                                                                                                                                                                             |
| `FORGE_NODE_REQUEST_TIMEOUT_S`             | `float` | `15.0`                  | Timeout for an ordinary request to an agent.                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `FORGE_NODE_STATE_DIR`                     | `str`   | `~/.starforge-forgelet` | Where a node keeps the identity and credential it received when it joined a fleet. Not under the storage root on purpose - a node that cannot see the shared mount still has to be able to say who it is.                                                                                                                                                                                                                                                                                             |
| `FORGE_NODE_TOKEN`                         | `str`   | —                       | Bearer token shared between the console and every agent. Both sides must have it; a missing token refuses startup rather than running unauthenticated.                                                                                                                                                                                                                                                                                                                                                |
| `FORGE_RUNTIME_REGISTRY_FILE`              | `str`   | —                       | Deployment-level map from runtime\_id to an execution artifact. Required on Slurm, which needs SIF or SQSH. Optional on local and kuberay, where an entry overrides the catalog's image reference — which is how an internal mirror gets used.                                                                                                                                                                                                                                                        |
| `FORGE_SLURM_RUNTIME_PROFILE`              | `str`   | —                       | Which container runtime Slurm uses. Must be chosen explicitly; there is no auto-detection and no fallback between profiles.                                                                                                                                                                                                                                                                                                                                                                           |

## Slurm executor

| Environment variable                 | Type    | Default    | Description                                                                                                                                         |
| ------------------------------------ | ------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FORGE_SLURM_ACCOUNT`                | `str`   | —          | Accounting account charged for jobs.                                                                                                                |
| `FORGE_SLURM_CLUSTER_WAIT_TIMEOUT_S` | `int`   | `1800`     | How long to wait for every node in an allocation to join the Ray cluster before giving up.                                                          |
| `FORGE_SLURM_CPUS_PER_NODE`          | `int`   | `1`        | CPUs requested per node.                                                                                                                            |
| `FORGE_SLURM_JWT_FILE`               | `str`   | —          | File holding the JWT used for slurmrestd authentication.                                                                                            |
| `FORGE_SLURM_PARTITION`              | `str`   | —          | Partition jobs are submitted to.                                                                                                                    |
| `FORGE_SLURM_QOS`                    | `str`   | —          | QoS requested for jobs.                                                                                                                             |
| `FORGE_SLURM_RAY_PORT`               | `int`   | `1200`     | Port the Ray head listens on inside an allocation.                                                                                                  |
| `FORGE_SLURM_REST_API_VERSION`       | `str`   | `v0.0.43`  | slurmrestd API version this deployment talks.                                                                                                       |
| `FORGE_SLURM_REST_URL`               | `str`   | —          | slurmrestd endpoint. The control plane always goes through REST; there is no sbatch or SSH fallback.                                                |
| `FORGE_SLURM_REST_USER`              | `str`   | —          | Username the control plane authenticates to slurmrestd as.                                                                                          |
| `FORGE_SLURM_STORAGE_ROOT`           | `str`   | —          | Where the storage root appears on the cluster's parallel filesystem. Empty means the same absolute path as storage\_root, which is the normal case. |
| `FORGE_SLURM_TIMEOUT`                | `float` | `30.0`     | Timeout for one slurmrestd request.                                                                                                                 |
| `FORGE_SLURM_TIME_LIMIT`             | `str`   | `24:00:00` | Wall-clock limit requested per allocation.                                                                                                          |
| `FORGE_SLURM_TLS_VERIFY`             | `bool`  | `true`     | Whether to verify slurmrestd's TLS certificate. Leave it on outside debugging.                                                                      |
| `FORGE_SLURM_WORKER_PORT_MAX`        | `int`   | `2999`     | High end of that range.                                                                                                                             |
| `FORGE_SLURM_WORKER_PORT_MIN`        | `int`   | `2000`     | Low end of the port range Ray workers use.                                                                                                          |

## KubeRay executor

| Environment variable                  | Type    | Default                | Description                                                                                                                                                                                                                                                                     |
| ------------------------------------- | ------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FORGE_K8S_API_SERVER`                | `str`   | —                      | Kubernetes API server address. Resolution order is explicit configuration, then the in-cluster service account, then unavailable. The console usually runs inside the cluster, so this is normally left empty.                                                                  |
| `FORGE_K8S_CA_PATH`                   | `str`   | —                      | CA bundle used to verify the Kubernetes API server.                                                                                                                                                                                                                             |
| `FORGE_K8S_CPU_LIMIT`                 | `str`   | —                      | CPU limit for training pods. Empty means unbounded, decided by node capacity.                                                                                                                                                                                                   |
| `FORGE_K8S_GPU_NODE_LABEL`            | `str`   | `starforge/gpu-series` | Node label whose value names the GPU series, so the Kubernetes scheduler places a job on the right hardware natively.                                                                                                                                                           |
| `FORGE_K8S_GPU_PASSTHROUGH`           | `bool`  | `true`                 | Whether pods actually request nvidia.com/gpu. False is simulation mode for a cluster with no GPUs — kind, OrbStack, CI — where Ray's logical num-gpus still works so the platform's own end-to-end tests can run.                                                               |
| `FORGE_K8S_IMAGE_PULL_POLICY`         | `str`   | `IfNotPresent`         | Image pull policy for training pods.                                                                                                                                                                                                                                            |
| `FORGE_K8S_IMAGE_PULL_SECRET`         | `str`   | —                      | Pull credentials for an internal registry.                                                                                                                                                                                                                                      |
| `FORGE_K8S_INSECURE_SKIP_VERIFY`      | `bool`  | `false`                | Skip TLS verification against the Kubernetes API. Debugging a self-signed certificate only.                                                                                                                                                                                     |
| `FORGE_K8S_MEMORY_LIMIT`              | `str`   | —                      | Memory limit for training pods. Empty means unbounded.                                                                                                                                                                                                                          |
| `FORGE_K8S_NAMESPACE`                 | `str`   | —                      | Namespace training resources are created in. Empty uses the service account's own namespace.                                                                                                                                                                                    |
| `FORGE_K8S_NCCL_SOCKET_IFNAME`        | `str`   | —                      | Which network interface NCCL uses. On a multi-NIC machine, leaving it unset lets NCCL pick the wrong one — which presents as "training starts but is absurdly slow" or a hang in all-reduce, and is very hard to trace. Common values are bond0, eth0, ib0.                     |
| `FORGE_K8S_SERVICE_ACCOUNT`           | `str`   | —                      | Service account the training pods run as, when they need cluster access of their own.                                                                                                                                                                                           |
| `FORGE_K8S_SHM_SIZE`                  | `str`   | `64Gi`                 | Size of /dev/shm in a training pod. Ray's object store lives there, and the container default of 64MB makes anything but a tiny batch OOM immediately.                                                                                                                          |
| `FORGE_K8S_STORAGE_MOUNT`             | `str`   | —                      | Where that PVC appears inside the pod. Empty means the same absolute path as storage\_root, so the console and the container agree by default.                                                                                                                                  |
| `FORGE_K8S_STORAGE_PVC`               | `str`   | —                      | PVC holding the storage root. Without it every job re-downloads tens of gigabytes of weights, which on an internal link costs more than the training does.                                                                                                                      |
| `FORGE_K8S_STORAGE_SHARED`            | `bool`  | `true`                 | Whether the claim is readable by every pod of one job at once. An RWO claim does not fail a multi-node job outright — it gives each pod its own volume, so checkpoint shards split across nodes and the run cannot be resumed. Declared here and verified against the live PVC. |
| `FORGE_K8S_TIMEOUT`                   | `float` | `30.0`                 | Timeout for one Kubernetes API request.                                                                                                                                                                                                                                         |
| `FORGE_K8S_TOKEN`                     | `str`   | —                      | Bearer token for the Kubernetes API, when not using the in-cluster service account.                                                                                                                                                                                             |
| `FORGE_K8S_VOLUME_PVC`                | `str`   | —                      | Optional second claim holding the volume root, mounted read-only. Empty means this deployment cannot honour volume references on KubeRay, and a job that requests one is rejected at admission.                                                                                 |
| `FORGE_KUBERAY_ACTIVE_DEADLINE_S`     | `int`   | `0`                    | Wall-clock ceiling for a running job, so a runaway does not hold cards indefinitely. 0 disables it.                                                                                                                                                                             |
| `FORGE_KUBERAY_HEAD_MODE`             | `str`   | `colocated`            | Whether the Ray head is also the first GPU compute node. `colocated` matches the shape NeMo-RL and KubeRay document. A dedicated GPU-less head changes the allocation shape and is not switched to implicitly.                                                                  |
| `FORGE_KUBERAY_PRERUNNING_DEADLINE_S` | `int`   | `1800`                 | How long a job may sit before running before it is judged failed. Without it, an unpullable image or a nodeSelector nothing satisfies leaves the job Pending forever while holding its queue slot. 0 disables the deadline.                                                     |
| `FORGE_KUBERAY_RAY_VERSION`           | `str`   | `2.55.1`               | Ray version declared to KubeRay. It must match the Ray inside the training image. A mismatch presents as "the cluster came up but workers never registered", with one easily-missed warning line as the only clue.                                                              |
| `FORGE_KUBERAY_SUBMISSION_MODE`       | `str`   | `K8sJobMode`           | How the job is submitted. `K8sJobMode` starts a separate submitter pod; `SidecarMode` injects the submitter into the head pod — one pod and one scheduling round fewer, at the cost of not supporting a submitter pod template.                                                 |
| `FORGE_KUBERAY_TTL_SECONDS`           | `int`   | `600`                  | Kubernetes garbage-collection backstop. Normal terminal states are cleaned up by the platform as soon as logs are archived; this TTL only catches what the control plane failed to remove.                                                                                      |

## Local executor

| Environment variable                  | Type    | Default  | Description                                                                                                                                                                                                                                                                                         |
| ------------------------------------- | ------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FORGE_DISK_WATERMARK_PCT`            | `int`   | `90`     | Stop dequeuing for everyone once the storage root passes this usage percentage, to protect the checkpoint writes of jobs already running. 0 turns it off. A different question from the storage quota: a quota asks who is using too much, a watermark asks whether the disk is about to fill.      |
| `FORGE_LOCAL_CHECK_EXTERNAL_GPUS`     | `bool`  | `true`   | Whether to look for processes occupying GPUs from outside the platform — somebody who logged in and started a notebook. Without a device plugin as a backstop this check earns its keep: otherwise the platform hands out a card that is already busy.                                              |
| `FORGE_LOCAL_CHECK_GPU_HEALTH`        | `bool`  | `true`   | Whether to check GPU health with nvidia-smi: pending ECC retirements, uncorrected errors, hardware throttling. A failing card stops being allocated and the capacity view says why. These faults raise no error — they produce bad results or a silent slowdown.                                    |
| `FORGE_LOCAL_CLI_TIMEOUT`             | `float` | `60.0`   | Timeout for one docker or podman CLI call.                                                                                                                                                                                                                                                          |
| `FORGE_LOCAL_CONTAINER_TTL_S`         | `int`   | `600`    | Cleanup backstop for orphaned containers. A normal terminal state archives logs and removes the workload immediately; this only catches what reconciliation missed.                                                                                                                                 |
| `FORGE_LOCAL_CPU_LIMIT`               | `str`   | —        | Passed to `--cpus`. Empty means unbounded.                                                                                                                                                                                                                                                          |
| `FORGE_LOCAL_GPU_COUNT`               | `int`   | `0`      | Number of physical GPUs. 0 detects them with nvidia-smi, which a containerised console should not rely on: the probe needs the host's NVIDIA Container Toolkit injection, and a failed injection reads as zero cards.                                                                               |
| `FORGE_LOCAL_GPU_PASSTHROUGH`         | `bool`  | `true`   | Whether containers actually receive GPUs. False is simulation mode: allocation, accounting and labels all behave normally but the container runs without `--gpus`, which is how the platform's end-to-end tests run on a machine with no NVIDIA runtime. Production must leave this on.             |
| `FORGE_LOCAL_IMAGE_PULL_TIMEOUT_S`    | `float` | `3600.0` | Timeout for the explicit pull before a first launch with a new image. Training images are routinely tens of gigabytes, so this cannot share the sixty-second CLI timeout — that would make "first use of a new image" fail every time.                                                              |
| `FORGE_LOCAL_MEMORY_LIMIT`            | `str`   | —        | Passed to `--memory`. Empty means unbounded.                                                                                                                                                                                                                                                        |
| `FORGE_LOCAL_NETWORK`                 | `str`   | `host`   | Container network mode. Host networking is the simplest workable choice: a job has to reach back to the console to report status and artifacts, and on a single machine there is nothing to isolate it from.                                                                                        |
| `FORGE_LOCAL_RUNTIME`                 | `str`   | `auto`   | Container runtime. `auto` probes docker, then podman, then falls back to a bare process. Process mode isolates nothing and can leave GPU memory behind, so it is for a development machine only — that isolation is the entire reason for containerising in the first place.                        |
| `FORGE_LOCAL_SHM_SIZE`                | `str`   | —        | Size of /dev/shm in a training container. Empty follows k8s\_shm\_size. It is RAM-backed, so this plus the memory limit must stay under physical memory.                                                                                                                                            |
| `FORGE_LOCAL_STOP_TIMEOUT`            | `int`   | `30`     | How many seconds a stopped job gets after SIGTERM to save a checkpoint before SIGKILL.                                                                                                                                                                                                              |
| `FORGE_REGISTRY_RECONCILE_INTERVAL_S` | `float` | `300.0`  | How often the dataset and corpus projection is rebuilt from object storage. It is a backstop now that the write paths upsert directly, but it is also the only thing that repairs an entry changed outside the API. 0 turns it off.                                                                 |
| `FORGE_STORAGE_SCAN_INTERVAL_S`       | `float` | `600.0`  | How often the storage root is walked to produce per-user usage and platform totals. 0 never scans, and storage quotas and the dashboard card then report "not measured". A full scan takes minutes on a large tree, which is why it is a background role rather than something computed on request. |
| `FORGE_TERMINAL_CLEANUP_GRACE_S`      | `float` | `2.0`    | How long to wait after a lifecycle response before cleaning up, so the training process can finish writing its last stderr.                                                                                                                                                                         |
| `FORGE_TERMINAL_LOG_TAIL_LINES`       | `int`   | `5000`   | How many trailing log lines to pull from the container, pod or Slurm archive when the training side did not manage to report them itself.                                                                                                                                                           |

## Object storage

| Environment variable              | Type    | Default         | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| --------------------------------- | ------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `FORGE_ALLOWED_RECIPES`           | `str`   | —               | Which post-training methods this deployment offers, as `&lt;framework&gt;/&lt;method&gt;`, comma separated. Empty enables everything in the catalog. The catalog says what the platform supports; this says what this deployment opens to its users — a cluster without the memory for distillation turns it off here rather than in the catalog.                                                                                                                              |
| `FORGE_HWCONFIG_REFRESH_INTERVAL` | `float` | `15.0`          | How often each replica reloads the hardware profile configuration.                                                                                                                                                                                                                                                                                                                                                                                                             |
| `FORGE_RECIPE_STRICT_PARAMS`      | `bool`  | `true`          | Whether submission rejects hyperparameter keys the recipe never declared. On by default: a misspelled name that is silently ignored leaves the user believing they tuned something.                                                                                                                                                                                                                                                                                            |
| `FORGE_S3_ACCESS_KEY`             | `str`   | —               | Access key for that endpoint. Server-side only.                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `FORGE_S3_BUCKET`                 | `str`   | `starforge`     | Bucket everything is written into.                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `FORGE_S3_ENDPOINT`               | `str`   | —               | S3-compatible endpoint for job packages, artifacts, datasets and archives. Credentials stay on the server: clients and jobs only ever receive presigned URLs, scoped to one method and one key with an expiry. So credentials never reach the Ray dashboard, a container environment, or a job log, and a leaked URL costs one key for one window rather than the whole bucket. Unconfigured means object storage is off and everything falls back to shared-filesystem paths. |
| `FORGE_S3_JOB_PRESIGN_TTL`        | `int`   | `259200`        | Validity of a presigned URL handed to a job — its code package, or an `--init-from` artifact. It has to cover queueing as well as fetching: waiting hours in a queue is normal, and a URL that expires while queued fails the job on its first step.                                                                                                                                                                                                                           |
| `FORGE_S3_PRESIGN_TTL`            | `int`   | `3600`          | Validity of an ordinary presigned URL.                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `FORGE_S3_REGION`                 | `str`   | `us-east-1`     | Region passed to the S3 client.                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `FORGE_S3_SECRET_KEY`             | `str`   | —               | Secret key for that endpoint. Server-side only.                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `FORGE_S3_SIGNATURE_VERSION`      | `str`   | `s3v4`          | Signature version. Leave it at s3v4 unless the endpoint predates it.                                                                                                                                                                                                                                                                                                                                                                                                           |
| `FORGE_TIMEZONE`                  | `str`   | `Asia/Shanghai` | IANA timezone name for the whole service. Scheduling windows and daily GPU-hour boundaries are both computed in it.                                                                                                                                                                                                                                                                                                                                                            |

## Scheduling windows

| Environment variable                      | Type    | Default         | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| ----------------------------------------- | ------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FORGE_ARGILLA_API_KEY`                   | `str`   | —               | API key for that Argilla instance.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `FORGE_ARGILLA_URL`                       | `str`   | —               | Argilla instance used for human preference annotation. Its address is a deployment-level setting at the same trust level as the database and object storage — on an internal deployment it is already on a private network, so the private-network guard that webhooks get does not apply here.                                                                                                                                                                                                                                                                                                                      |
| `FORGE_ARGILLA_WORKSPACE`                 | `str`   | `argilla`       | Argilla workspace annotation datasets are created in.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `FORGE_FAIRSHARE_WINDOW_H`                | `float` | `24.0`          | How far back fair-share looks. Too short and somebody who just finished a large job is immediately at the front again; too long and usage from weeks ago still holds today's work back.                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `FORGE_SCHEDULE_DISPATCH_MIN_REMAINING_S` | `float` | `1800.0`        | Minimum time that must remain in a window for a job to be dispatched into it.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `FORGE_SCHEDULE_ENFORCE`                  | `bool`  | `false`         | Whether scheduling windows actually block work. Off means the configured windows are displayed but nothing is refused.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `FORGE_SCHEDULE_STOP_GRACE_S`             | `float` | `600.0`         | Grace period after a window closes before running jobs are stopped.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `FORGE_SCHED_POLICY`                      | `str`   | `priority-fifo` | Dequeue ordering. `priority-fifo` is priority descending then first-come-first-served. `fair-share` puts whoever has used less recently ahead within one priority band. A single team will not notice the difference; several teams sharing one cluster will, because first-come-first-served lets one person queue twenty jobs inside their quota and fill the queue — which becomes an interpersonal problem immediately, and gets blamed on the platform.                                                                                                                                                         |
| `FORGE_SCHED_PREEMPTION`                  | `bool`  | `false`         | Turns a team quota from a ceiling into a guarantee. A team may run above it while the cluster has room, and that overflow is reclaimed -- paused, with automatic resume from the last checkpoint -- when a team below its own guarantee is waiting. Off by default, in which case a team quota is a hard cap and nothing is ever taken back. Borrowing and reclaim are one switch: lending without reclaim lets one team hold the cluster indefinitely.                                                                                                                                                              |
| `FORGE_SCHED_PREEMPT_MIN_RUNTIME_S`       | `float` | `600.0`         | How long a job is safe from reclaim after it starts. Without a floor a busy queue reclaims one job, admits another, reclaims that one, and the cluster spends its day writing checkpoints.                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `FORGE_SCHED_RESERVE_AFTER_S`             | `float` | `900.0`         | How long a job may be denied on capacity before the scheduler starts holding cards for it. Without reservation the queue starves large jobs, and the platform itself generates the stream that starves them: lifecycle jobs and Playground sessions take one GPU each, so a denied four-GPU job is skipped and the one-GPU jobs behind it claim the capacity that was just freed, every pass, forever. The console shows it as "waiting for capacity", indefinitely. Reserving costs idle cards while the reservation fills, which is why it only applies to jobs already starved this long. 0 disables reservation. |
| `FORGE_SWEEP_EARLY_STOP_ENABLED`          | `bool`  | `true`          | Deployment-wide kill switch for sweep early stopping. Early stopping is configured per sweep by the client, so this exists purely so that a behaviour which actively kills jobs can be turned off globally without editing any sweep.                                                                                                                                                                                                                                                                                                                                                                                |

## Maintenance mode

| Environment variable     | Type   | Default | Description                                                                                                                                                                                                                                                        |
| ------------------------ | ------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `FORGE_MAINTENANCE_MODE` | `bool` | `false` | When on, new submissions queue but are never dispatched, and drained jobs stay PAUSED with auto-resume set. Turning it off resumes automatically: the queue worker resubmits each paused job under its original run id and training continues from its checkpoint. |
| `FORGE_MAINTENANCE_NOTE` | `str`  | —       | Message shown to anyone whose submission is held by maintenance mode.                                                                                                                                                                                              |

## Runtime watchdog

| Environment variable                | Type    | Default  | Description                                                                                                                                                                                                                                                                                                                                                                                                                            |
| ----------------------------------- | ------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FORGE_MCP_ENABLED`                 | `bool`  | `true`   | Whether the MCP server is mounted, exposing job configuration, metrics and logs to AI agents.                                                                                                                                                                                                                                                                                                                                          |
| `FORGE_MCP_TOKEN_DAYS`              | `int`   | `30`     | Validity of an MCP OAuth access token.                                                                                                                                                                                                                                                                                                                                                                                                 |
| `FORGE_ONBOARDING_CARD`             | `bool`  | `true`   | Whether the overview shows the first-run guide that walks a new user through installing the CLI, signing in and submitting a first job. It hides itself once any job exists.                                                                                                                                                                                                                                                           |
| `FORGE_PREMIUM_PUBLIC`              | `bool`  | `false`  | Whether AI diagnosis, Ask Agent and MCP access are open to every signed-in user rather than administrators only.                                                                                                                                                                                                                                                                                                                       |
| `FORGE_WATCHDOG_CLUSTER_RECONCILE`  | `bool`  | `true`   | Whether to reconcile cluster-wide Ray GPU usage against the ledger. This check does not depend on anything the client reports, which makes it the most reliable of the three.                                                                                                                                                                                                                                                          |
| `FORGE_WATCHDOG_CLUSTER_TOLERANCE`  | `int`   | `0`      | Tolerance in GPUs for the cluster-level reconciliation.                                                                                                                                                                                                                                                                                                                                                                                |
| `FORGE_WATCHDOG_ENABLED`            | `bool`  | `true`   | Whether the runtime watchdog runs. Bare-metal Ray has no cgroup or GPU hard isolation, so admission is a soft gate applied before the job starts; the watchdog periodically reconciles what the cluster is actually using against what the ledger says, alerts and audits on a mismatch, and can stop a job that reliably exceeds its allocation. It is defence in depth — a hard quota needs infrastructure like Kubernetes or Kueue. |
| `FORGE_WATCHDOG_ENFORCE`            | `bool`  | `false`  | Whether the watchdog stops a job it reliably judges over-allocated, rather than only alerting.                                                                                                                                                                                                                                                                                                                                         |
| `FORGE_WATCHDOG_FATAL_HANG_S`       | `float` | `90.0`   | How long a job may stay silent after its logs already show a fatal worker crash before it is marked FAILED and stopped. 0 disables it.                                                                                                                                                                                                                                                                                                 |
| `FORGE_WATCHDOG_GPU_MIN_MEM_MIB`    | `float` | `2048.0` | Memory a card must hold before it counts as in use by this job. It excludes idle cards, which report roughly 0.6 GB of leftover context, so a machine's idle GPUs are not miscounted as over-allocation.                                                                                                                                                                                                                               |
| `FORGE_WATCHDOG_GPU_TOLERANCE`      | `int`   | `0`      | How many GPUs a job may exceed its accounted allocation by before it counts as over.                                                                                                                                                                                                                                                                                                                                                   |
| `FORGE_WATCHDOG_INTERVAL`           | `float` | `120.0`  | How often the watchdog reconciles.                                                                                                                                                                                                                                                                                                                                                                                                     |
| `FORGE_WATCHDOG_SILENCE_ALERT_S`    | `float` | `1800.0` | How long an active job with a non-zero allocation may report no telemetry before an alert fires, which usually means collection was disabled or the job is wedged.                                                                                                                                                                                                                                                                     |
| `FORGE_WATCHDOG_STARTUP_GRACE_S`    | `float` | `300.0`  | How long after a job reaches RUNNING before the watchdog will judge it, avoiding the noise of startup and warm-up.                                                                                                                                                                                                                                                                                                                     |
| `FORGE_WATCHDOG_TELEMETRY_WINDOW_S` | `float` | `300.0`  | How recent telemetry has to be to count as current.                                                                                                                                                                                                                                                                                                                                                                                    |

## Diagnosis agent

| Environment variable              | Type    | Default                     | Description                                                                                                                                                                                |
| --------------------------------- | ------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `FORGE_AGENT_COMPARE_MAX_TURNS`   | `int`   | `6`                         | Tool-call rounds the comparison assistant may take.                                                                                                                                        |
| `FORGE_AGENT_DIAGNOSE_ON_FAIL`    | `bool`  | `true`                      | Whether a failure triggers a diagnosis immediately.                                                                                                                                        |
| `FORGE_AGENT_DIAGNOSE_RUNNING`    | `bool`  | `true`                      | Whether jobs are diagnosed while still running, not only after they finish.                                                                                                                |
| `FORGE_AGENT_DIAGNOSIS_COOLDOWN`  | `int`   | `900`                       | Minimum gap between two diagnoses of the same job.                                                                                                                                         |
| `FORGE_AGENT_DIAGNOSIS_INTERVAL`  | `float` | `300.0`                     | How often running jobs are considered for diagnosis.                                                                                                                                       |
| `FORGE_AGENT_DIAGNOSIS_MAX_TURNS` | `int`   | `8`                         | Tool-call rounds the diagnosis agent may take before it must answer.                                                                                                                       |
| `FORGE_AGENT_ENABLED`             | `bool`  | `true`                      | Whether the diagnosis agent runs at all.                                                                                                                                                   |
| `FORGE_AGENT_LLM_API_KEY`         | `str`   | —                           | API key for that endpoint.                                                                                                                                                                 |
| `FORGE_AGENT_LLM_BASE_URL`        | `str`   | `https://api.openai.com/v1` | OpenAI-compatible endpoint the diagnosis agent calls, for example `https://api.openai.com/v1`.                                                                                             |
| `FORGE_AGENT_LLM_MODEL`           | `str`   | `gpt-4o-mini`               | Model the diagnosis agent reasons with. The default is a light model to keep costs down, but multi-step diagnosis is sensitive to model capability — production should use a stronger one. |
| `FORGE_AGENT_LLM_TEMPERATURE`     | `float` | `0.3`                       | Sampling temperature. A negative value omits the parameter entirely, which some reasoning models require — they reject any explicit temperature with a 400.                                |
| `FORGE_AGENT_LLM_TIMEOUT`         | `float` | `60.0`                      | Timeout for one LLM call.                                                                                                                                                                  |

## Secrets and integrations

| Environment variable           | Type   | Default                                       | Description                                                                                                                                                                                                                                                       |
| ------------------------------ | ------ | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FORGE_HF_ENDPOINT`            | `str`  | —                                             | Hub API mirror for training and Playground containers, read by huggingface\_hub as `HF_ENDPOINT`. This is what makes an air-gapped deployment able to pull weights at all — see [air-gapped deployments](/en/ops/airgapped).                                      |
| `FORGE_HF_LOGIN_ENABLED`       | `bool` | `false`                                       | Offer "Sign in with Hugging Face" on the login page. Separate from the Hugging Face account binding below, which lets an already-signed-in user pull gated repositories — same OAuth application, two unrelated decisions. Sign-in asks only for identity scopes. |
| `FORGE_HF_OAUTH_BASE_URL`      | `str`  | `https://huggingface.co`                      | Hugging Face base URL. Change it only for an enterprise Hub.                                                                                                                                                                                                      |
| `FORGE_HF_OAUTH_CLIENT_ID`     | `str`  | —                                             | Client id of the Hugging Face OAuth application.                                                                                                                                                                                                                  |
| `FORGE_HF_OAUTH_CLIENT_SECRET` | `str`  | —                                             | Client secret of that application.                                                                                                                                                                                                                                |
| `FORGE_HF_OAUTH_ENABLED`       | `bool` | `false`                                       | Whether users can link a Hugging Face account, which is what lets a job read a gated model or push an export.                                                                                                                                                     |
| `FORGE_HF_OAUTH_SCOPES`        | `str`  | `openid profile email read-repos gated-repos` | Scopes requested when a user links their account.                                                                                                                                                                                                                 |
| `FORGE_HF_OAUTH_WRITE_ENABLED` | `bool` | `false`                                       | Whether linking may request write scopes. Off by default, so the safe default is that nothing is ever uploaded.                                                                                                                                                   |
| `FORGE_HF_PREFLIGHT_ENABLED`   | `bool` | `true`                                        | Whether submission checks up front that the Hub resources a job names are reachable and authorised. Only takes effect when the Hugging Face integration is on.                                                                                                    |
| `FORGE_SECRET_ENC_KEY`         | `str`  | —                                             | Fernet key used to encrypt stored secrets at rest. Required once the Hugging Face integration is enabled — there is no implicit fallback, because a silently unencrypted secret store is worse than a refusal to start.                                           |

## Webhook channels

| Environment variable                   | Type   | Default         | Description                                                                                                                                                                                                               |
| -------------------------------------- | ------ | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FORGE_DAILY_REPORT_AGENT_MAX_TURNS`   | `int`  | `6`             | Tool-call rounds the source-analysis agent may take.                                                                                                                                                                      |
| `FORGE_DAILY_REPORT_DEEP_ANALYSIS`     | `bool` | `false`         | Whether the agent reads job working directories to identify new methods and techniques. Needs an LLM configured, and costs noticeably more.                                                                               |
| `FORGE_DAILY_REPORT_DEEP_ANALYSIS_MAX` | `int`  | `3`             | How many projects one report may analyse deeply, which is the cost ceiling.                                                                                                                                               |
| `FORGE_DAILY_REPORT_ENABLED`           | `bool` | `false`         | Whether the daily summary of post-training progress is generated and sent.                                                                                                                                                |
| `FORGE_DAILY_REPORT_KEEP_DAYS`         | `int`  | `90`            | How long the web version of a report stays readable before its link returns 404.                                                                                                                                          |
| `FORGE_DAILY_REPORT_PROMPT`            | `str`  | —               | Custom system prompt for the report. Empty uses the built-in default.                                                                                                                                                     |
| `FORGE_DAILY_REPORT_SECRET`            | `str`  | —               | Signing secret for that dedicated webhook. Empty falls back to the DingTalk secret.                                                                                                                                       |
| `FORGE_DAILY_REPORT_TIME`              | `str`  | `09:00`         | Time of day the report is sent, HH:MM in the service timezone.                                                                                                                                                            |
| `FORGE_DAILY_REPORT_WEBHOOK_URL`       | `str`  | —               | Dedicated webhook for the daily report. Empty falls back to the DingTalk webhook.                                                                                                                                         |
| `FORGE_DAILY_REPORT_WEEKDAYS_ONLY`     | `bool` | `true`          | Whether the report is sent on weekdays only.                                                                                                                                                                              |
| `FORGE_DINGTALK_AT_ALL`                | `bool` | `false`         | Whether to @-mention everyone on push.                                                                                                                                                                                    |
| `FORGE_DINGTALK_AT_MOBILES`            | `str`  | —               | Phone numbers to @-mention on push, comma separated.                                                                                                                                                                      |
| `FORGE_DINGTALK_ENABLED`               | `bool` | `false`         | Whether DingTalk push is active. Off silences both mirrored site messages and the daily report.                                                                                                                           |
| `FORGE_DINGTALK_NOTIFY_SITE_MESSAGES`  | `bool` | `false`         | Whether in-app notifications are mirrored into the DingTalk group.                                                                                                                                                        |
| `FORGE_DINGTALK_SECRET`                | `str`  | —               | Signing secret, required when the robot's security setting is set to signing.                                                                                                                                             |
| `FORGE_DINGTALK_SITE_MESSAGE_KINDS`    | `str`  | `error,warning` | Which notification levels are mirrored, comma separated, to keep the group readable.                                                                                                                                      |
| `FORGE_DINGTALK_WEBHOOK_URL`           | `str`  | —               | DingTalk group robot webhook, including its access token.                                                                                                                                                                 |
| `FORGE_WEBHOOK_ALLOWED_HOSTS`          | `str`  | —               | Outbound host allowlist, comma separated. Empty forbids every webhook, deliberately: the console can usually reach the internal network, so allowing an arbitrary outbound target turns it into an internal port scanner. |
| `FORGE_WEBHOOK_DAILY_REPORT_URL`       | `str`  | —               | Separate target for the daily report. Empty falls back to the primary target.                                                                                                                                             |
| `FORGE_WEBHOOK_ENABLED`                | `bool` | `false`         | Whether the generic webhook channel is active. The console sends one templated HTTP request — extensions that would otherwise run inside the console process cross a process boundary instead of becoming plugins.        |
| `FORGE_WEBHOOK_NOTIFY_SITE_MESSAGES`   | `bool` | `false`         | Whether in-app notifications are mirrored to the webhook.                                                                                                                                                                 |
| `FORGE_WEBHOOK_SITE_MESSAGE_KINDS`     | `str`  | —               | Which notification levels are mirrored, comma separated. Empty mirrors all of them.                                                                                                                                       |
| `FORGE_WEBHOOK_TEMPLATE`               | `str`  | —               | JSON body template, with \{\{title}} and \{\{text}} placeholders. Empty uses a generic shape.                                                                                                                             |
| `FORGE_WEBHOOK_URL`                    | `str`  | —               | Primary webhook target, used for mirrored site messages and as the daily-report fallback.                                                                                                                                 |
