Configuration reference

Every setting a StarForge deployment reads, with its type, default, and what it does.

export FORGE_DEFAULT_FLEET_KIND=kuberay
export FORGE_STORAGE_ROOT=/mnt/shared/starforge
export FORGE_WEB_JWT_SECRET=$(openssl rand -hex 32)

Every setting is an environment variable prefixed FORGE_. There is no configuration file — a deployment is described entirely by its environment, which is what makes the same image work under Docker Compose, Kubernetes and systemd without a translation layer.

This page is generated from server.core.config.WebSettings by scripts/gen_settings_docs.py, and CI fails if a setting is added without a description. If a knob exists, it is on this page.

The five you cannot skip

Everything else has a working default. These do not.

FORGE_DEFAULT_FLEET_KIND · local | node | kuberay | slurm · default: "local"

The backend of the Fleet seeded when the console has none yet — read once, at first start. After that the backend is a property of the Fleet a workload was placed on, and registering a second Fleet is how a console gains another. There is no silent fallback between backends, so set this explicitly even when you want local. See backends.

FORGE_STORAGE_ROOT · path

The one path a deployment configures. It must be an absolute path, and it must resolve identically on the console, on every node, and inside every container. Empty means single-machine development.

FORGE_WEB_JWT_SECRET · string

Generated per process when unset, which logs everyone out on restart and breaks multi-replica deployments outright. Set a fixed strong random value for anything but a laptop.

FORGE_DB_URL · url

Points at Postgres for a team deployment. Left empty it uses a local SQLite file, which is correct for one machine and wrong for two.

FORGE_PUBLIC_URL · url

The https address users reach this deployment at. The OIDC redirect URI is built from it, so single sign-on cannot work without it.

Reading the tables

• Required means the deployment refuses to start without it.
• generated means a value is derived per process when you do not supply one.
• A dash means the default is empty, and empty usually means the feature is off.

Absolute paths are validated at startup: a relative FORGE_STORAGE_ROOT is refused rather than resolved against whatever directory the process happened to start in.

Core

Environment variable	Type	Default	Description
FORGE_CACHE_TTL	float	5.0	How long in-process caches hold a value before re-reading it.
FORGE_COOKIE_SECURE	bool	—	Whether the session cookie carries the Secure flag. Follows whether public_url is https when unset; set it false to debug over plain http locally.
FORGE_DB_PATH	Path	.forge/web.db	Where SQLite lives, relative to the console’s working directory. Deliberately not under the storage root: that root is usually a shared filesystem, and SQLite over NFS has well-known locking problems. Unused once db_url points at Postgres.
FORGE_DB_URL	str	—	Database URL. Empty falls back to the SQLite file at db_path. A team deployment points this at Postgres.
FORGE_GITHUB_ALLOWED_ORGS	str	—	Comma-separated organization logins whose members may sign in. Empty means any GitHub account can create an account here. Setting it also makes the consent screen ask for read:org; a member whose membership is private and unshared is refused.
FORGE_GITHUB_CLIENT_ID	str	—	OAuth app client ID from GitHub. Setting it and the secret puts a “Sign in with GitHub” button on the login page, with no restart. The callback URL to register is <public URL>/api/auth/social/github/callback.
FORGE_GITHUB_CLIENT_SECRET	str	—	OAuth app client secret from GitHub. Stored encrypted and never returned by the API.
FORGE_GOOGLE_ALLOWED_DOMAINS	str	—	Comma-separated email domains allowed to sign in with Google. Empty means any Google account can create an account here, which on a console reachable from the internet is open registration. Both the verified email domain and the Workspace hd claim are accepted.
FORGE_GOOGLE_CLIENT_ID	str	—	OAuth client ID from the Google Cloud console. Setting it and the secret puts a “Sign in with Google” button on the login page, with no restart. The redirect URI to register is <public URL>/api/auth/social/google/callback.
FORGE_GOOGLE_CLIENT_SECRET	str	—	OAuth client secret from the Google Cloud console. Stored encrypted and never returned by the API.
FORGE_GOOGLE_ONE_TAP	bool	false	Show Google’s One Tap prompt on the login page, which offers to sign in with an account the browser is already signed into. Off by default because the prompt appears without the visitor asking for it. Requires the Google client ID and secret.
FORGE_HOST	str	127.0.0.1	Address the console binds to. The CLI’s —host overrides it.
FORGE_INGEST_URL	str	—	Where training processes POST their logs and metrics. With a remote cluster, do not use 127.0.0.1 — that is the worker’s own loopback. Give the console URL as reachable from inside the cluster network.
FORGE_JWT_HOURS	int	8	How long an access token stays valid.
FORGE_NO_AUTH	bool	false	Treat every request as an admin and skip authentication entirely. Development only — never set this on anything reachable by other people.
FORGE_OIDC_CLIENT_ID	str	—	Client id registered with the identity provider.
FORGE_OIDC_CLIENT_SECRET	str	—	Client secret registered with the identity provider.
FORGE_OIDC_DEFAULT_ROLE	str	operator	Role given to a user on their first single sign-on login. Not admin: the first administrator should be somebody a person deliberately promoted.
FORGE_OIDC_DISCOVERY_URL	str	—	Explicit discovery document URL. Empty derives one by appending /.well-known/openid-configuration to the issuer; Casdoor serves a per-application path and needs this set.
FORGE_OIDC_GROUP_CLAIM	str	—	The claim holding the user’s groups, read as team membership on every sign-in. Empty (the default) means the platform never touches team membership — which is what a deployment that manages teams by hand needs, since reconciling against an absent claim would empty everybody out. A group the directory names but the platform does not have is skipped, never created.
FORGE_OIDC_GROUP_PREFIX	str	—	Stripped from the front of each group before it is read as a team key. Directories hand out paths and DNs where the platform wants a key.
FORGE_OIDC_INSECURE_SKIP_VERIFY	bool	false	Skip TLS verification against the identity provider. For debugging a self-signed certificate on an internal network; never in production.
FORGE_OIDC_ISSUER	str	—	OIDC issuer URL, for example a Casdoor application. Setting it turns on single sign-on.
FORGE_OIDC_SCOPE	str	openid profile email	Scopes requested at login.
FORGE_PORT	int	8080	Port the console listens on. The CLI’s —port overrides it.
FORGE_PUBLIC_URL	str	—	The address users and identity providers reach this deployment at. Used to build the OIDC redirect URI, so a team deployment must set it to the real https domain.
FORGE_REDIS_PREFIX	str	lab	Key namespace, so several environments can share one Redis without colliding.
FORGE_REDIS_URL	str	—	Redis for shared cache, distributed locks, rate limiting and immediate token revocation. Empty disables all four and the console behaves as a single instance.
FORGE_REFRESH_DAYS	int	30	How long a refresh token stays valid. It is delivered only as an httpOnly cookie, so browser JavaScript cannot read it.
FORGE_REPO_ROOT	Path	generated	Project root the console reads experiment definitions and local bookkeeping from. Defaults to the working directory.
FORGE_SERVE	bool	false	Bind 0.0.0.0 instead of loopback. Turn it on for a team or container deployment.
FORGE_STATIC_DIR	Path	—	Directory holding the built console assets (web/dist). The server wheel ships them, so this is normally left alone.
FORGE_WEB_JWT_SECRET	str	generated	Signing key for access tokens. Generated per process when unset, which is fine on one machine and wrong for anything else: two replicas would sign with different keys, so every restart logs everyone out.

Proxied submission

Environment variable	Type	Default	Description
FORGE_CLUSTER_PROFILE	str	—	Default CLUSTER_PROFILE forwarded to jobs. A profile named by the client is recorded but does not override this.
FORGE_NEMO_RL_DIR	str	—	Path to NeMo-RL inside the training container, forwarded to the job as NEMO_RL_DIR. Required to submit NeMo-RL jobs. This is a path in the image, not on the host and not in the CLI project.

Storage

Environment variable	Type	Default	Description
FORGE_AUDIT_RETENTION_DAYS	int	0	How long an audit row is kept, in days. 0 keeps them forever, which is the default. Pruning does not break the tamper-evident chain: each run records the range it removed and the hash of the last row it deleted, and the verification endpoint accepts a gap that such a record explains — those records are themselves never deleted. Enforcing a retention period is a commercial feature.
FORGE_AUDIT_RETENTION_INTERVAL_S	float	86400.0	How often the audit retention task runs.
FORGE_CLUSTER_SECRETS_FILE	str	—	Path to a secrets file inside the container. When set, the path is forwarded and the secret values never enter the Ray dashboard.
FORGE_DATA_SCAN_ALLOW_TERMS	str	—	Comma-separated terms to suppress as known false positives.
FORGE_DATA_SCAN_DENY_TERMS	str	—	Comma-separated terms this deployment treats as sensitive, in addition to the built-in rules. Terms, never patterns: an operator pasting a regex into a settings box is how a config change becomes a backtrack that hangs a worker.
FORGE_DATA_SCAN_POLICY	str	report	What the platform does about sensitive values found in a dataset. off does not scan; report scans, records and shows the findings while admitting the job anyway; block refuses a submission whose data has a high-confidence finding. Blocking keys on severity, never on any finding — the low-confidence rules hit every base64 blob and git SHA in a code corpus.
FORGE_INGEST_TOKEN_DAYS	int	30	How long a job’s ingest token stays valid. It has to outlast the longest training run you expect, or a long job loses the ability to report partway through.
FORGE_JOB_RUNNER_MODE	str	bundled	How the platform runtime reaches a job. bundled injects a content-addressed PEX, so the training image needs nothing. image-installed requires starforge preinstalled in the image and exists only as a migration rollback — there is never an automatic fallback between the two.
FORGE_JOB_RUNNER_PATH	str	—	Where the bundled runner PEX is read from. Empty means <repo_root>/dist/starforge-runner.pex.
FORGE_JOB_RUNNER_PYTHON	str	—	Deployment-level override for the interpreter the runner uses. Empty uses whatever the runtime artifact declares.
FORGE_JOB_RUNNER_REQUIRES_PYTHON	str	>=3.10,<3.14	Python versions the bundled runner supports. A training image outside this range is refused rather than failing later.
FORGE_LICENSE_FILE	str	—	A file holding the licence instead. Useful because a licence is a few hundred bytes nobody wants in a Helm values file or a shell history, and mounting a secret as a file is how Kubernetes prefers to do this. license_key wins if both are set.
FORGE_LICENSE_KEY	str	—	The commercial licence, as the signed string the vendor issued. Empty runs the community edition, which is a supported configuration rather than a degraded one — everything in it works and keeps working. Verified offline: there is no call home, ever. Settable from the admin console, and it takes effect on the next request rather than needing a restart, because entitlement is checked per request rather than at mount time.
FORGE_MAX_UPLOAD_MB	int	1024	Size ceiling for a submitted job package, compressed. The uncompressed ceiling is fifty times this, which is what stops a compression bomb.
FORGE_PASSTHROUGH_ENV	dict	generated	Non-secret environment variables forwarded into every job, such as an HF endpoint mirror.
FORGE_QUEUE_POLL_INTERVAL	float	5.0	How often the dequeue loop looks for work.
FORGE_QUOTA_ENFORCE	bool	true	Whether the soft quota actually blocks admission. Turning it off records usage without refusing anything.
FORGE_SERVER_SECRETS_FILE	str	—	Server-side KEY=VALUE secrets file, injected into jobs when cluster_secrets_file is not used.
FORGE_STORAGE_ROOT	str	—	The one path a deployment configures. Everything the platform writes derives from it: caches, run directories, job packages and control-plane state. The invariant is that this path is identical on the console, on every node, and inside every container — a shared filesystem or a same-path bind mount satisfies it. Empty means single-machine development, falling back to ./.forge/storage.
FORGE_SUBMIT_DRY_RUN	bool	false	Assemble the runtime environment but never actually submit to Ray. Debugging aid.
FORGE_VOLUME_ROOT	str	—	Root of the governed Volumes a job may read, distinct from its training data. Must sit outside the storage root, which is checked at startup - that root is mounted writable into job containers, so a volume beneath it would be reachable by a second, writable path and the read-only mount would be decorative. Mount JuiceFS here, with the S3 settings pointed at its gateway, to give one content both an object API and a filesystem. Empty means this deployment does not support volumes.

Automatic retry

Environment variable	Type	Default	Description
FORGE_AUTO_RETRY_DELAY_S	float	60.0	How long to wait after a failure before resubmitting, so a transient fault has time to clear.
FORGE_AUTO_RETRY_ENABLED	bool	false	Whether a failed job is resubmitted automatically, resuming from its last checkpoint under the same run id. Off by default — turn it on deliberately.
FORGE_AUTO_RETRY_MAX	int	1	How many automatic retries a job gets. A job can override it with its own max_retries.
FORGE_AUTO_RETRY_ON	str	started	Which failures are retried. started retries only jobs that reached RUNNING — a configuration or environment error usually fails before that, and retrying it just fails again. all retries every failure.

LLM judge

Environment variable	Type	Default	Description
FORGE_JUDGE_CACHE_ENABLED	bool	true	Cache deterministic scores (temperature at or below zero) so the same prompt is not paid for twice.
FORGE_JUDGE_ENABLED	bool	false	Whether training jobs get a judge endpoint and token injected, so a reward function can score with an LLM against a Rubric. The upstream credential and the audit trail both stay on the platform side.
FORGE_JUDGE_LLM_API_KEY	str	—	API key for the judge’s upstream. Empty reuses agent_llm_api_key.
FORGE_JUDGE_LLM_BASE_URL	str	—	OpenAI-compatible upstream the judge calls. Empty reuses the diagnosis agent’s LLM configuration.
FORGE_JUDGE_LLM_MODEL	str	—	Model the judge scores with. Empty reuses agent_llm_model.
FORGE_JUDGE_LLM_TIMEOUT	float	60.0	Request timeout for one judge call.

Agent sandbox

Environment variable	Type	Default	Description
FORGE_ENVIRONMENT_ALLOWED_HOSTS	str	—	Hosts a job may reach for an openenv-remote environment, comma separated. Empty means none, deliberately: a training container usually has the run of the internal network, so accepting whatever host a manifest names would turn an environment reference into an outbound channel written by somebody else.
FORGE_ENVIRONMENT_SELF_REGISTRATION	bool	false	Whether an environment may authorise its own host by proving one answers there. Off by default, because turning it on moves the egress judgement from “an administrator named this host” to “an OpenEnv server answered here when the version was sealed, and we recorded who asked” — a weaker claim about the host and a much stronger one about the audit trail.
FORGE_REFLOW_ENABLED	bool	true	Whether the Reflow background task runs. It is on by default and does nothing until a deployment sets a capture sample rate above zero — capture is enabled per deployment, never globally.
FORGE_REFLOW_FLUSH_INTERVAL_S	int	60	How often captured traffic is flushed from memory into the buffer.
FORGE_SANDBOX_CPUS	str	1	CPU limit for one sandbox container.
FORGE_SANDBOX_ENDPOINT	str	—	An external E2B-compatible sandbox service to use instead of the built-in container.
FORGE_SANDBOX_IMAGE	str	—	Image for the first-party code sandbox. Off unless configured: running model-generated code is not something to enable by default, and there is no sensible image to guess.
FORGE_SANDBOX_MAX_CONCURRENCY	int	8	How many sandboxes may run at once.
FORGE_SANDBOX_MEMORY	str	512m	Memory limit for one sandbox container.
FORGE_SANDBOX_TIMEOUT_S	float	30.0	Wall-clock limit for one sandbox execution.
FORGE_SANDBOX_TOKEN	str	—	Bearer token for that external sandbox service.
FORGE_SANDBOX_USER	str	65534:65534	User the sandbox process runs as. Non-root by default; change it only if the image needs a different unprivileged account.

Playground

Environment variable	Type	Default	Description
FORGE_PLAYGROUND_ENABLED	bool	false	Whether users may start Playground sessions to chat with a run’s artifacts. The local backend only.
FORGE_PLAYGROUND_IMAGE	str	—	Inference image for Playground sessions. Must be a digest-pinned OCI reference.
FORGE_PLAYGROUND_MAX_GPUS	int	4	GPU ceiling for a single Playground session.
FORGE_PLAYGROUND_PORT_MAX	int	18099	High end of that port range. It must not overlap the deployment range: where they overlap, a throwaway session can take the port a stable endpoint needs.
FORGE_PLAYGROUND_PORT_MIN	int	18000	Low end of the host port range Playground sessions claim, one port each.
FORGE_PLAYGROUND_TTL_S	float	3600.0	How long an idle session lives before it is stopped and its GPUs released.

Model deployments

Environment variable	Type	Default	Description
FORGE_ALLOWED_IMAGE_REGISTRIES	str	—	Registry hosts a user may name with sf submit --image, comma separated and matched exactly. Empty leaves first-party framework overrides unrestricted but closes custom images entirely; a production deployment should configure it explicitly.
FORGE_DEFAULT_FLEET_KIND	str	local	The backend of the Fleet seeded when a console has none yet. local is one container on the console host, node one container on a remote machine running the forgelet daemon, kuberay an ephemeral RayCluster per job, slurm an exclusive allocation through slurmrestd. Read once, at first start — after that the backend is a property of the Fleet a workload was placed on, so registering a second Fleet is how a console gains a backend, and changing this moves no existing workload.
FORGE_DEPLOYMENT_ALLOW_TRUST_REMOTE_CODE	bool	false	Whether a deployment may load a model that executes code from its own repository. Off by default, because it means running code the model author wrote.
FORGE_DEPLOYMENT_HEALTH_TIMEOUT_S	float	10.0	Timeout for one health probe against a serving revision.
FORGE_DEPLOYMENT_MAX_CONCURRENT_REQUESTS	int	0	How many inference requests one deployment may have in flight through the console at once, or 0 for no cap. Backpressure rather than a quota - a saturated engine serves no faster for having more sockets open, and a refusal the caller can retry beats a queue that slows every request already in it.
FORGE_DEPLOYMENT_MAX_GPUS	int	8	GPU ceiling for one deployment revision.
FORGE_DEPLOYMENT_PORT_MAX	int	18299	High end of that port range.
FORGE_DEPLOYMENT_PORT_MIN	int	18100	Low end of the host port range deployments claim.
FORGE_DEPLOYMENT_READINESS_TIMEOUT_S	float	900.0	How long a new revision may take to become ready before it is judged failed. Large models loading from cold storage need the headroom.
FORGE_DEPLOYMENT_SGLANG_IMAGE	str	—	Serving image for SGLang deployments. Must be digest-pinned.
FORGE_DEPLOYMENT_SHARED_ROOTS	str	—	Extra read-only paths a deployment may load model weights from, comma separated.
FORGE_DEPLOYMENT_VLLM_IMAGE	str	—	Serving image for vLLM deployments. Must be digest-pinned.
FORGE_FORGELET_INDEX_URL	str	—	Package index the generated install.sh fetches the forgelet package from. Empty means PyPI; an intranet deployment points this at its own mirror, since the machine running the script may have no route to the public internet.
FORGE_IMAGE_NEMO_RL	str	—	Default image for NeMo-RL jobs. A tag is enough. A user’s --image takes precedence.
FORGE_IMAGE_OPENRLHF	str	—	Default image for OpenRLHF jobs.
FORGE_IMAGE_TRL	str	—	Default image for TRL jobs.
FORGE_IMAGE_VERL	str	—	Default image for verl jobs.
FORGE_INPROCESS_WORKERS	bool	true	Whether background roles — dequeue, diagnosis, watchdog, daily report — run inside the web process. Turn it off to run them as separate processes (python -m server.worker --role <name>): the slow LLM-calling ticks stop occupying the event loop and inflating API tail latency, and the web replicas become stateless and freely scalable. On by default so a single-container deployment keeps working unchanged.
FORGE_MODEL_DEPLOYMENTS_ENABLED	bool	false	Whether managed model deployments — long-lived serving for application traffic — are available at all.
FORGE_NODE_ADDRESSES	str	—	Comma-separated node URLs, for example http://gpu-1:7070,http://gpu-2:7070. Requires the storage root to be mounted at the same path on the console and on every node.
FORGE_NODE_LAUNCH_TIMEOUT_S	float	120.0	Timeout for a launch request, covering container start on a node that already has the image. A first-time image pull runs asynchronously and does not count against this.
FORGE_NODE_REQUEST_TIMEOUT_S	float	15.0	Timeout for an ordinary request to an agent.
FORGE_NODE_STATE_DIR	str	~/.starforge-forgelet	Where a node keeps the identity and credential it received when it joined a fleet. Not under the storage root on purpose - a node that cannot see the shared mount still has to be able to say who it is.
FORGE_NODE_TOKEN	str	—	Bearer token shared between the console and every agent. Both sides must have it; a missing token refuses startup rather than running unauthenticated.
FORGE_RUNTIME_REGISTRY_FILE	str	—	Deployment-level map from runtime_id to an execution artifact. Required on Slurm, which needs SIF or SQSH. Optional on local and kuberay, where an entry overrides the catalog’s image reference — which is how an internal mirror gets used.
FORGE_SLURM_RUNTIME_PROFILE	str	—	Which container runtime Slurm uses. Must be chosen explicitly; there is no auto-detection and no fallback between profiles.

Slurm executor

Environment variable	Type	Default	Description
FORGE_SLURM_ACCOUNT	str	—	Accounting account charged for jobs.
FORGE_SLURM_CLUSTER_WAIT_TIMEOUT_S	int	1800	How long to wait for every node in an allocation to join the Ray cluster before giving up.
FORGE_SLURM_CPUS_PER_NODE	int	1	CPUs requested per node.
FORGE_SLURM_JWT_FILE	str	—	File holding the JWT used for slurmrestd authentication.
FORGE_SLURM_PARTITION	str	—	Partition jobs are submitted to.
FORGE_SLURM_QOS	str	—	QoS requested for jobs.
FORGE_SLURM_RAY_PORT	int	1200	Port the Ray head listens on inside an allocation.
FORGE_SLURM_REST_API_VERSION	str	v0.0.43	slurmrestd API version this deployment talks.
FORGE_SLURM_REST_URL	str	—	slurmrestd endpoint. The control plane always goes through REST; there is no sbatch or SSH fallback.
FORGE_SLURM_REST_USER	str	—	Username the control plane authenticates to slurmrestd as.
FORGE_SLURM_STORAGE_ROOT	str	—	Where the storage root appears on the cluster’s parallel filesystem. Empty means the same absolute path as storage_root, which is the normal case.
FORGE_SLURM_TIMEOUT	float	30.0	Timeout for one slurmrestd request.
FORGE_SLURM_TIME_LIMIT	str	24:00:00	Wall-clock limit requested per allocation.
FORGE_SLURM_TLS_VERIFY	bool	true	Whether to verify slurmrestd’s TLS certificate. Leave it on outside debugging.
FORGE_SLURM_WORKER_PORT_MAX	int	2999	High end of that range.
FORGE_SLURM_WORKER_PORT_MIN	int	2000	Low end of the port range Ray workers use.

KubeRay executor

Environment variable	Type	Default	Description
FORGE_K8S_API_SERVER	str	—	Kubernetes API server address. Resolution order is explicit configuration, then the in-cluster service account, then unavailable. The console usually runs inside the cluster, so this is normally left empty.
FORGE_K8S_CA_PATH	str	—	CA bundle used to verify the Kubernetes API server.
FORGE_K8S_CPU_LIMIT	str	—	CPU limit for training pods. Empty means unbounded, decided by node capacity.
FORGE_K8S_GPU_NODE_LABEL	str	starforge/gpu-series	Node label whose value names the GPU series, so the Kubernetes scheduler places a job on the right hardware natively.
FORGE_K8S_GPU_PASSTHROUGH	bool	true	Whether pods actually request nvidia.com/gpu. False is simulation mode for a cluster with no GPUs — kind, OrbStack, CI — where Ray’s logical num-gpus still works so the platform’s own end-to-end tests can run.
FORGE_K8S_IMAGE_PULL_POLICY	str	IfNotPresent	Image pull policy for training pods.
FORGE_K8S_IMAGE_PULL_SECRET	str	—	Pull credentials for an internal registry.
FORGE_K8S_INSECURE_SKIP_VERIFY	bool	false	Skip TLS verification against the Kubernetes API. Debugging a self-signed certificate only.
FORGE_K8S_MEMORY_LIMIT	str	—	Memory limit for training pods. Empty means unbounded.
FORGE_K8S_NAMESPACE	str	—	Namespace training resources are created in. Empty uses the service account’s own namespace.
FORGE_K8S_NCCL_SOCKET_IFNAME	str	—	Which network interface NCCL uses. On a multi-NIC machine, leaving it unset lets NCCL pick the wrong one — which presents as “training starts but is absurdly slow” or a hang in all-reduce, and is very hard to trace. Common values are bond0, eth0, ib0.
FORGE_K8S_SERVICE_ACCOUNT	str	—	Service account the training pods run as, when they need cluster access of their own.
FORGE_K8S_SHM_SIZE	str	64Gi	Size of /dev/shm in a training pod. Ray’s object store lives there, and the container default of 64MB makes anything but a tiny batch OOM immediately.
FORGE_K8S_STORAGE_MOUNT	str	—	Where that PVC appears inside the pod. Empty means the same absolute path as storage_root, so the console and the container agree by default.
FORGE_K8S_STORAGE_PVC	str	—	PVC holding the storage root. Without it every job re-downloads tens of gigabytes of weights, which on an internal link costs more than the training does.
FORGE_K8S_STORAGE_SHARED	bool	true	Whether the claim is readable by every pod of one job at once. An RWO claim does not fail a multi-node job outright — it gives each pod its own volume, so checkpoint shards split across nodes and the run cannot be resumed. Declared here and verified against the live PVC.
FORGE_K8S_TIMEOUT	float	30.0	Timeout for one Kubernetes API request.
FORGE_K8S_TOKEN	str	—	Bearer token for the Kubernetes API, when not using the in-cluster service account.
FORGE_K8S_VOLUME_PVC	str	—	Optional second claim holding the volume root, mounted read-only. Empty means this deployment cannot honour volume references on KubeRay, and a job that requests one is rejected at admission.
FORGE_KUBERAY_ACTIVE_DEADLINE_S	int	0	Wall-clock ceiling for a running job, so a runaway does not hold cards indefinitely. 0 disables it.
FORGE_KUBERAY_HEAD_MODE	str	colocated	Whether the Ray head is also the first GPU compute node. colocated matches the shape NeMo-RL and KubeRay document. A dedicated GPU-less head changes the allocation shape and is not switched to implicitly.
FORGE_KUBERAY_PRERUNNING_DEADLINE_S	int	1800	How long a job may sit before running before it is judged failed. Without it, an unpullable image or a nodeSelector nothing satisfies leaves the job Pending forever while holding its queue slot. 0 disables the deadline.
FORGE_KUBERAY_RAY_VERSION	str	2.55.1	Ray version declared to KubeRay. It must match the Ray inside the training image. A mismatch presents as “the cluster came up but workers never registered”, with one easily-missed warning line as the only clue.
FORGE_KUBERAY_SUBMISSION_MODE	str	K8sJobMode	How the job is submitted. K8sJobMode starts a separate submitter pod; SidecarMode injects the submitter into the head pod — one pod and one scheduling round fewer, at the cost of not supporting a submitter pod template.
FORGE_KUBERAY_TTL_SECONDS	int	600	Kubernetes garbage-collection backstop. Normal terminal states are cleaned up by the platform as soon as logs are archived; this TTL only catches what the control plane failed to remove.

Local executor

Environment variable	Type	Default	Description
FORGE_DISK_WATERMARK_PCT	int	90	Stop dequeuing for everyone once the storage root passes this usage percentage, to protect the checkpoint writes of jobs already running. 0 turns it off. A different question from the storage quota: a quota asks who is using too much, a watermark asks whether the disk is about to fill.
FORGE_LOCAL_CHECK_EXTERNAL_GPUS	bool	true	Whether to look for processes occupying GPUs from outside the platform — somebody who logged in and started a notebook. Without a device plugin as a backstop this check earns its keep: otherwise the platform hands out a card that is already busy.
FORGE_LOCAL_CHECK_GPU_HEALTH	bool	true	Whether to check GPU health with nvidia-smi: pending ECC retirements, uncorrected errors, hardware throttling. A failing card stops being allocated and the capacity view says why. These faults raise no error — they produce bad results or a silent slowdown.
FORGE_LOCAL_CLI_TIMEOUT	float	60.0	Timeout for one docker or podman CLI call.
FORGE_LOCAL_CONTAINER_TTL_S	int	600	Cleanup backstop for orphaned containers. A normal terminal state archives logs and removes the workload immediately; this only catches what reconciliation missed.
FORGE_LOCAL_CPU_LIMIT	str	—	Passed to --cpus. Empty means unbounded.
FORGE_LOCAL_GPU_COUNT	int	0	Number of physical GPUs. 0 detects them with nvidia-smi, which a containerised console should not rely on: the probe needs the host’s NVIDIA Container Toolkit injection, and a failed injection reads as zero cards.
FORGE_LOCAL_GPU_PASSTHROUGH	bool	true	Whether containers actually receive GPUs. False is simulation mode: allocation, accounting and labels all behave normally but the container runs without --gpus, which is how the platform’s end-to-end tests run on a machine with no NVIDIA runtime. Production must leave this on.
FORGE_LOCAL_IMAGE_PULL_TIMEOUT_S	float	3600.0	Timeout for the explicit pull before a first launch with a new image. Training images are routinely tens of gigabytes, so this cannot share the sixty-second CLI timeout — that would make “first use of a new image” fail every time.
FORGE_LOCAL_MEMORY_LIMIT	str	—	Passed to --memory. Empty means unbounded.
FORGE_LOCAL_NETWORK	str	host	Container network mode. Host networking is the simplest workable choice: a job has to reach back to the console to report status and artifacts, and on a single machine there is nothing to isolate it from.
FORGE_LOCAL_RUNTIME	str	auto	Container runtime. auto probes docker, then podman, then falls back to a bare process. Process mode isolates nothing and can leave GPU memory behind, so it is for a development machine only — that isolation is the entire reason for containerising in the first place.
FORGE_LOCAL_SHM_SIZE	str	—	Size of /dev/shm in a training container. Empty follows k8s_shm_size. It is RAM-backed, so this plus the memory limit must stay under physical memory.
FORGE_LOCAL_STOP_TIMEOUT	int	30	How many seconds a stopped job gets after SIGTERM to save a checkpoint before SIGKILL.
FORGE_REGISTRY_RECONCILE_INTERVAL_S	float	300.0	How often the dataset and corpus projection is rebuilt from object storage. It is a backstop now that the write paths upsert directly, but it is also the only thing that repairs an entry changed outside the API. 0 turns it off.
FORGE_STORAGE_SCAN_INTERVAL_S	float	600.0	How often the storage root is walked to produce per-user usage and platform totals. 0 never scans, and storage quotas and the dashboard card then report “not measured”. A full scan takes minutes on a large tree, which is why it is a background role rather than something computed on request.
FORGE_TERMINAL_CLEANUP_GRACE_S	float	2.0	How long to wait after a lifecycle response before cleaning up, so the training process can finish writing its last stderr.
FORGE_TERMINAL_LOG_TAIL_LINES	int	5000	How many trailing log lines to pull from the container, pod or Slurm archive when the training side did not manage to report them itself.

Object storage

Environment variable	Type	Default	Description
FORGE_ALLOWED_RECIPES	str	—	Which post-training methods this deployment offers, as <framework>/<method>, comma separated. Empty enables everything in the catalog. The catalog says what the platform supports; this says what this deployment opens to its users — a cluster without the memory for distillation turns it off here rather than in the catalog.
FORGE_HWCONFIG_REFRESH_INTERVAL	float	15.0	How often each replica reloads the hardware profile configuration.
FORGE_RECIPE_STRICT_PARAMS	bool	true	Whether submission rejects hyperparameter keys the recipe never declared. On by default: a misspelled name that is silently ignored leaves the user believing they tuned something.
FORGE_S3_ACCESS_KEY	str	—	Access key for that endpoint. Server-side only.
FORGE_S3_BUCKET	str	starforge	Bucket everything is written into.
FORGE_S3_ENDPOINT	str	—	S3-compatible endpoint for job packages, artifacts, datasets and archives. Credentials stay on the server: clients and jobs only ever receive presigned URLs, scoped to one method and one key with an expiry. So credentials never reach the Ray dashboard, a container environment, or a job log, and a leaked URL costs one key for one window rather than the whole bucket. Unconfigured means object storage is off and everything falls back to shared-filesystem paths.
FORGE_S3_JOB_PRESIGN_TTL	int	259200	Validity of a presigned URL handed to a job — its code package, or an --init-from artifact. It has to cover queueing as well as fetching: waiting hours in a queue is normal, and a URL that expires while queued fails the job on its first step.
FORGE_S3_PRESIGN_TTL	int	3600	Validity of an ordinary presigned URL.
FORGE_S3_REGION	str	us-east-1	Region passed to the S3 client.
FORGE_S3_SECRET_KEY	str	—	Secret key for that endpoint. Server-side only.
FORGE_S3_SIGNATURE_VERSION	str	s3v4	Signature version. Leave it at s3v4 unless the endpoint predates it.
FORGE_TIMEZONE	str	Asia/Shanghai	IANA timezone name for the whole service. Scheduling windows and daily GPU-hour boundaries are both computed in it.

Scheduling windows

Environment variable	Type	Default	Description
FORGE_ARGILLA_API_KEY	str	—	API key for that Argilla instance.
FORGE_ARGILLA_URL	str	—	Argilla instance used for human preference annotation. Its address is a deployment-level setting at the same trust level as the database and object storage — on an internal deployment it is already on a private network, so the private-network guard that webhooks get does not apply here.
FORGE_ARGILLA_WORKSPACE	str	argilla	Argilla workspace annotation datasets are created in.
FORGE_FAIRSHARE_WINDOW_H	float	24.0	How far back fair-share looks. Too short and somebody who just finished a large job is immediately at the front again; too long and usage from weeks ago still holds today’s work back.
FORGE_SCHEDULE_DISPATCH_MIN_REMAINING_S	float	1800.0	Minimum time that must remain in a window for a job to be dispatched into it.
FORGE_SCHEDULE_ENFORCE	bool	false	Whether scheduling windows actually block work. Off means the configured windows are displayed but nothing is refused.
FORGE_SCHEDULE_STOP_GRACE_S	float	600.0	Grace period after a window closes before running jobs are stopped.
FORGE_SCHED_POLICY	str	priority-fifo	Dequeue ordering. priority-fifo is priority descending then first-come-first-served. fair-share puts whoever has used less recently ahead within one priority band. A single team will not notice the difference; several teams sharing one cluster will, because first-come-first-served lets one person queue twenty jobs inside their quota and fill the queue — which becomes an interpersonal problem immediately, and gets blamed on the platform.
FORGE_SCHED_PREEMPTION	bool	false	Turns a team quota from a ceiling into a guarantee. A team may run above it while the cluster has room, and that overflow is reclaimed — paused, with automatic resume from the last checkpoint — when a team below its own guarantee is waiting. Off by default, in which case a team quota is a hard cap and nothing is ever taken back. Borrowing and reclaim are one switch: lending without reclaim lets one team hold the cluster indefinitely.
FORGE_SCHED_PREEMPT_MIN_RUNTIME_S	float	600.0	How long a job is safe from reclaim after it starts. Without a floor a busy queue reclaims one job, admits another, reclaims that one, and the cluster spends its day writing checkpoints.
FORGE_SCHED_RESERVE_AFTER_S	float	900.0	How long a job may be denied on capacity before the scheduler starts holding cards for it. Without reservation the queue starves large jobs, and the platform itself generates the stream that starves them: lifecycle jobs and Playground sessions take one GPU each, so a denied four-GPU job is skipped and the one-GPU jobs behind it claim the capacity that was just freed, every pass, forever. The console shows it as “waiting for capacity”, indefinitely. Reserving costs idle cards while the reservation fills, which is why it only applies to jobs already starved this long. 0 disables reservation.
FORGE_SWEEP_EARLY_STOP_ENABLED	bool	true	Deployment-wide kill switch for sweep early stopping. Early stopping is configured per sweep by the client, so this exists purely so that a behaviour which actively kills jobs can be turned off globally without editing any sweep.

Maintenance mode

Environment variable	Type	Default	Description
FORGE_MAINTENANCE_MODE	bool	false	When on, new submissions queue but are never dispatched, and drained jobs stay PAUSED with auto-resume set. Turning it off resumes automatically: the queue worker resubmits each paused job under its original run id and training continues from its checkpoint.
FORGE_MAINTENANCE_NOTE	str	—	Message shown to anyone whose submission is held by maintenance mode.

Runtime watchdog

Environment variable	Type	Default	Description
FORGE_MCP_ENABLED	bool	true	Whether the MCP server is mounted, exposing job configuration, metrics and logs to AI agents.
FORGE_MCP_TOKEN_DAYS	int	30	Validity of an MCP OAuth access token.
FORGE_ONBOARDING_CARD	bool	true	Whether the overview shows the first-run guide that walks a new user through installing the CLI, signing in and submitting a first job. It hides itself once any job exists.
FORGE_PREMIUM_PUBLIC	bool	false	Whether AI diagnosis, Ask Agent and MCP access are open to every signed-in user rather than administrators only.
FORGE_WATCHDOG_CLUSTER_RECONCILE	bool	true	Whether to reconcile cluster-wide Ray GPU usage against the ledger. This check does not depend on anything the client reports, which makes it the most reliable of the three.
FORGE_WATCHDOG_CLUSTER_TOLERANCE	int	0	Tolerance in GPUs for the cluster-level reconciliation.
FORGE_WATCHDOG_ENABLED	bool	true	Whether the runtime watchdog runs. Bare-metal Ray has no cgroup or GPU hard isolation, so admission is a soft gate applied before the job starts; the watchdog periodically reconciles what the cluster is actually using against what the ledger says, alerts and audits on a mismatch, and can stop a job that reliably exceeds its allocation. It is defence in depth — a hard quota needs infrastructure like Kubernetes or Kueue.
FORGE_WATCHDOG_ENFORCE	bool	false	Whether the watchdog stops a job it reliably judges over-allocated, rather than only alerting.
FORGE_WATCHDOG_FATAL_HANG_S	float	90.0	How long a job may stay silent after its logs already show a fatal worker crash before it is marked FAILED and stopped. 0 disables it.
FORGE_WATCHDOG_GPU_MIN_MEM_MIB	float	2048.0	Memory a card must hold before it counts as in use by this job. It excludes idle cards, which report roughly 0.6 GB of leftover context, so a machine’s idle GPUs are not miscounted as over-allocation.
FORGE_WATCHDOG_GPU_TOLERANCE	int	0	How many GPUs a job may exceed its accounted allocation by before it counts as over.
FORGE_WATCHDOG_INTERVAL	float	120.0	How often the watchdog reconciles.
FORGE_WATCHDOG_SILENCE_ALERT_S	float	1800.0	How long an active job with a non-zero allocation may report no telemetry before an alert fires, which usually means collection was disabled or the job is wedged.
FORGE_WATCHDOG_STARTUP_GRACE_S	float	300.0	How long after a job reaches RUNNING before the watchdog will judge it, avoiding the noise of startup and warm-up.
FORGE_WATCHDOG_TELEMETRY_WINDOW_S	float	300.0	How recent telemetry has to be to count as current.

Diagnosis agent

Environment variable	Type	Default	Description
FORGE_AGENT_COMPARE_MAX_TURNS	int	6	Tool-call rounds the comparison assistant may take.
FORGE_AGENT_DIAGNOSE_ON_FAIL	bool	true	Whether a failure triggers a diagnosis immediately.
FORGE_AGENT_DIAGNOSE_RUNNING	bool	true	Whether jobs are diagnosed while still running, not only after they finish.
FORGE_AGENT_DIAGNOSIS_COOLDOWN	int	900	Minimum gap between two diagnoses of the same job.
FORGE_AGENT_DIAGNOSIS_INTERVAL	float	300.0	How often running jobs are considered for diagnosis.
FORGE_AGENT_DIAGNOSIS_MAX_TURNS	int	8	Tool-call rounds the diagnosis agent may take before it must answer.
FORGE_AGENT_ENABLED	bool	true	Whether the diagnosis agent runs at all.
FORGE_AGENT_LLM_API_KEY	str	—	API key for that endpoint.
FORGE_AGENT_LLM_BASE_URL	str	https://api.openai.com/v1	OpenAI-compatible endpoint the diagnosis agent calls, for example https://api.openai.com/v1.
FORGE_AGENT_LLM_MODEL	str	gpt-4o-mini	Model the diagnosis agent reasons with. The default is a light model to keep costs down, but multi-step diagnosis is sensitive to model capability — production should use a stronger one.
FORGE_AGENT_LLM_TEMPERATURE	float	0.3	Sampling temperature. A negative value omits the parameter entirely, which some reasoning models require — they reject any explicit temperature with a 400.
FORGE_AGENT_LLM_TIMEOUT	float	60.0	Timeout for one LLM call.

Secrets and integrations

Environment variable	Type	Default	Description
FORGE_HF_ENDPOINT	str	—	Hub API mirror for training and Playground containers, read by huggingface_hub as HF_ENDPOINT. This is what makes an air-gapped deployment able to pull weights at all — see air-gapped deployments.
FORGE_HF_LOGIN_ENABLED	bool	false	Offer “Sign in with Hugging Face” on the login page. Separate from the Hugging Face account binding below, which lets an already-signed-in user pull gated repositories — same OAuth application, two unrelated decisions. Sign-in asks only for identity scopes.
FORGE_HF_OAUTH_BASE_URL	str	https://huggingface.co	Hugging Face base URL. Change it only for an enterprise Hub.
FORGE_HF_OAUTH_CLIENT_ID	str	—	Client id of the Hugging Face OAuth application.
FORGE_HF_OAUTH_CLIENT_SECRET	str	—	Client secret of that application.
FORGE_HF_OAUTH_ENABLED	bool	false	Whether users can link a Hugging Face account, which is what lets a job read a gated model or push an export.
FORGE_HF_OAUTH_SCOPES	str	openid profile email read-repos gated-repos	Scopes requested when a user links their account.
FORGE_HF_OAUTH_WRITE_ENABLED	bool	false	Whether linking may request write scopes. Off by default, so the safe default is that nothing is ever uploaded.
FORGE_HF_PREFLIGHT_ENABLED	bool	true	Whether submission checks up front that the Hub resources a job names are reachable and authorised. Only takes effect when the Hugging Face integration is on.
FORGE_SECRET_ENC_KEY	str	—	Fernet key used to encrypt stored secrets at rest. Required once the Hugging Face integration is enabled — there is no implicit fallback, because a silently unencrypted secret store is worse than a refusal to start.

Webhook channels

Environment variable	Type	Default	Description
FORGE_DAILY_REPORT_AGENT_MAX_TURNS	int	6	Tool-call rounds the source-analysis agent may take.
FORGE_DAILY_REPORT_DEEP_ANALYSIS	bool	false	Whether the agent reads job working directories to identify new methods and techniques. Needs an LLM configured, and costs noticeably more.
FORGE_DAILY_REPORT_DEEP_ANALYSIS_MAX	int	3	How many projects one report may analyse deeply, which is the cost ceiling.
FORGE_DAILY_REPORT_ENABLED	bool	false	Whether the daily summary of post-training progress is generated and sent.
FORGE_DAILY_REPORT_KEEP_DAYS	int	90	How long the web version of a report stays readable before its link returns 404.
FORGE_DAILY_REPORT_PROMPT	str	—	Custom system prompt for the report. Empty uses the built-in default.
FORGE_DAILY_REPORT_SECRET	str	—	Signing secret for that dedicated webhook. Empty falls back to the DingTalk secret.
FORGE_DAILY_REPORT_TIME	str	09:00	Time of day the report is sent, HH:MM in the service timezone.
FORGE_DAILY_REPORT_WEBHOOK_URL	str	—	Dedicated webhook for the daily report. Empty falls back to the DingTalk webhook.
FORGE_DAILY_REPORT_WEEKDAYS_ONLY	bool	true	Whether the report is sent on weekdays only.
FORGE_DINGTALK_AT_ALL	bool	false	Whether to @-mention everyone on push.
FORGE_DINGTALK_AT_MOBILES	str	—	Phone numbers to @-mention on push, comma separated.
FORGE_DINGTALK_ENABLED	bool	false	Whether DingTalk push is active. Off silences both mirrored site messages and the daily report.
FORGE_DINGTALK_NOTIFY_SITE_MESSAGES	bool	false	Whether in-app notifications are mirrored into the DingTalk group.
FORGE_DINGTALK_SECRET	str	—	Signing secret, required when the robot’s security setting is set to signing.
FORGE_DINGTALK_SITE_MESSAGE_KINDS	str	error,warning	Which notification levels are mirrored, comma separated, to keep the group readable.
FORGE_DINGTALK_WEBHOOK_URL	str	—	DingTalk group robot webhook, including its access token.
FORGE_WEBHOOK_ALLOWED_HOSTS	str	—	Outbound host allowlist, comma separated. Empty forbids every webhook, deliberately: the console can usually reach the internal network, so allowing an arbitrary outbound target turns it into an internal port scanner.
FORGE_WEBHOOK_DAILY_REPORT_URL	str	—	Separate target for the daily report. Empty falls back to the primary target.
FORGE_WEBHOOK_ENABLED	bool	false	Whether the generic webhook channel is active. The console sends one templated HTTP request — extensions that would otherwise run inside the console process cross a process boundary instead of becoming plugins.
FORGE_WEBHOOK_NOTIFY_SITE_MESSAGES	bool	false	Whether in-app notifications are mirrored to the webhook.
FORGE_WEBHOOK_SITE_MESSAGE_KINDS	str	—	Which notification levels are mirrored, comma separated. Empty mirrors all of them.
FORGE_WEBHOOK_TEMPLATE	str	—	JSON body template, with {{title}} and {{text}} placeholders. Empty uses a generic shape.
FORGE_WEBHOOK_URL	str	—	Primary webhook target, used for mirrored site messages and as the daily-report fallback.