> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Air-gapped deployments

> Run StarForge on a network with no route to Hugging Face, PyPI or a public registry.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
FORGE_HF_ENDPOINT=https://mirrors.your-company.com/repository/huggingface
```

That one setting is what makes weight downloads work behind a mirror. The rest of this page is the
other three things that break, and what to do about each.

## Model weights

`huggingface_hub` reads `HF_ENDPOINT`, and the platform injects `FORGE_HF_ENDPOINT` into every
training and Playground container as exactly that. Point it at a reverse proxy — Nexus, Artifactory,
or anything else that fronts the Hub.

Two adjustments travel with it automatically, and both exist because of a failure that is painful to
diagnose:

<AccordionGroup>
  <Accordion title="Timeouts are widened" icon="clock">
    `huggingface_hub` defaults to a ten-second timeout. A mirror's first fetch of a file it has never
    cached goes all the way to the origin, and routinely takes longer than that. The symptom is a
    download that fails on a cold file and succeeds on a retry, which reads like a flaky network
    rather than a configuration problem.
  </Accordion>

  <Accordion title="Xet is disabled" icon="ban">
    Xet needs short-lived tokens that a reverse proxy cannot issue. Left on, weight downloads stall
    on a 400 whose body contains nothing but a URL — no message, no field, nothing to search for.

    If your image genuinely implements Xet, override it explicitly through `FORGE_PASSTHROUGH_ENV`.
  </Accordion>
</AccordionGroup>

<Warning>
  A job whose weights cannot be fetched does not fail fast. It starts, logs the download, and sits
  there until it times out — holding its GPU allocation the whole time. Confirm the mirror works
  before opening submission to a team.
</Warning>

## Container images

Training images come from the recipe catalog, which names public registries such as `nvcr.io`. On an
isolated network you mirror them and redirect:

| Setting                          | What it does                                                                                         |
| -------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `FORGE_RUNTIME_REGISTRY_FILE`    | Maps a `runtime_id` to the artifact your registry actually holds, overriding the catalog's reference |
| `FORGE_ALLOWED_IMAGE_REGISTRIES` | The hosts a user may name with `--image`. Set it to your internal registry                           |
| `FORGE_K8S_IMAGE_PULL_SECRET`    | Pull credentials, on KubeRay                                                                         |

Pin digests rather than tags in production. A tag that moves inside your own mirror is still a tag
that moved.

## Building the image behind a mirror

The Dockerfile defaults to public upstreams so it builds anywhere. Point it at an internal
mirror with build arguments:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
docker build \
  --build-arg APT_MIRROR=https://mirrors.your-company.com/repository \
  --build-arg PYPI_INDEX=https://mirrors.your-company.com/repository/pypi-group/simple/ \
  --build-arg NPM_REGISTRY=https://mirrors.your-company.com/repository/npm-group/ \
  --build-arg UV_IMAGE=mirror.your-company.com/astral-sh/uv:python3.13-trixie-slim \
  -t starforge-console:0.3.0 .
```

Leaving `APT_MIRROR` empty keeps Debian's own sources, which is what you want on a machine
with normal internet access.

## The platform runtime

The job side needs `starforge` to report metrics, and the training image will not have it.

`FORGE_JOB_RUNNER_MODE=bundled`, the default, solves this without network access: the server injects
a content-addressed PEX into the job, so the training image needs nothing installed. This is one of
the few places where the air-gapped case is the *easy* one — leave the setting alone.

## Datasets

Nothing here reaches the internet. Datasets are pushed with `sf dataset push` into the deployment's
own object storage, and pulled into the shared cache when a job starts. See
[datasets](/en/guides/datasets).

## Confirm it worked

Submit the quickstart job and watch the logs. The sequence you want is:

```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
resolving Qwen/Qwen2.5-1.5B via https://mirrors.your-company.com/repository/huggingface
downloading model-00001-of-00002.safetensors
...
Ray runtime started
```

A stall between the first and second line is the mirror. A 400 with a bare URL in the body is Xet.
A timeout on a large file after roughly ten seconds means the widened timeouts are not being applied
— check that the platform, not your own script, is setting `HF_ENDPOINT`.