Air-gapped deployments

Run StarForge on a network with no route to Hugging Face, PyPI or a public registry.

FORGE_HF_ENDPOINT=https://mirrors.your-company.com/repository/huggingface

That one setting is what makes weight downloads work behind a mirror. The rest of this page is the other three things that break, and what to do about each.

Model weights

huggingface_hub reads HF_ENDPOINT, and the platform injects FORGE_HF_ENDPOINT into every training and Playground container as exactly that. Point it at a reverse proxy — Nexus, Artifactory, or anything else that fronts the Hub.

Two adjustments travel with it automatically, and both exist because of a failure that is painful to diagnose:

Timeouts are widened

huggingface_hub defaults to a ten-second timeout. A mirror’s first fetch of a file it has never cached goes all the way to the origin, and routinely takes longer than that. The symptom is a download that fails on a cold file and succeeds on a retry, which reads like a flaky network rather than a configuration problem.

Xet is disabled

Xet needs short-lived tokens that a reverse proxy cannot issue. Left on, weight downloads stall on a 400 whose body contains nothing but a URL — no message, no field, nothing to search for.

If your image genuinely implements Xet, override it explicitly through FORGE_PASSTHROUGH_ENV.

A job whose weights cannot be fetched does not fail fast. It starts, logs the download, and sits there until it times out — holding its GPU allocation the whole time. Confirm the mirror works before opening submission to a team.

Container images

Training images come from the recipe catalog, which names public registries such as nvcr.io. On an isolated network you mirror them and redirect:

Setting	What it does
FORGE_RUNTIME_REGISTRY_FILE	Maps a runtime_id to the artifact your registry actually holds, overriding the catalog’s reference
FORGE_ALLOWED_IMAGE_REGISTRIES	The hosts a user may name with --image. Set it to your internal registry
FORGE_K8S_IMAGE_PULL_SECRET	Pull credentials, on KubeRay

Pin digests rather than tags in production. A tag that moves inside your own mirror is still a tag that moved.

Building the image behind a mirror

The Dockerfile defaults to public upstreams so it builds anywhere. Point it at an internal mirror with build arguments:

docker build \
  --build-arg APT_MIRROR=https://mirrors.your-company.com/repository \
  --build-arg PYPI_INDEX=https://mirrors.your-company.com/repository/pypi-group/simple/ \
  --build-arg NPM_REGISTRY=https://mirrors.your-company.com/repository/npm-group/ \
  --build-arg UV_IMAGE=mirror.your-company.com/astral-sh/uv:python3.13-trixie-slim \
  -t starforge-console:0.3.0 .

Leaving APT_MIRROR empty keeps Debian’s own sources, which is what you want on a machine with normal internet access.

The platform runtime

The job side needs starforge to report metrics, and the training image will not have it.

FORGE_JOB_RUNNER_MODE=bundled, the default, solves this without network access: the server injects a content-addressed PEX into the job, so the training image needs nothing installed. This is one of the few places where the air-gapped case is the easy one — leave the setting alone.

Datasets

Nothing here reaches the internet. Datasets are pushed with sf dataset push into the deployment’s own object storage, and pulled into the shared cache when a job starts. See datasets.

Confirm it worked

Submit the quickstart job and watch the logs. The sequence you want is:

resolving Qwen/Qwen2.5-1.5B via https://mirrors.your-company.com/repository/huggingface
downloading model-00001-of-00002.safetensors
...
Ray runtime started

A stall between the first and second line is the mirror. A 400 with a bare URL in the body is Xet. A timeout on a large file after roughly ten seconds means the widened timeouts are not being applied — check that the platform, not your own script, is setting HF_ENDPOINT.