> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Build a custom image

> Dockerfile, what not to bake in, registry allowlist, tags vs digests

```dockerfile theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
FROM nvidia/cuda:12.4.1-devel-ubuntu22.04
RUN pip install torch transformers trl accelerate
COPY . /workspace
```

A training image holds CUDA, Python, and whatever your `train.sh` imports. That is all.

<Warning>
  Do not pip-install the StarForge CLI into the image, and do not copy observability code out of an
  old tutorial repository. The platform runtime is injected at launch as a content-addressed PEX —
  the image needs nothing for reporting to work.
</Warning>

## What belongs in the image

| Put in the image                                                                                           | Leave out                                           |
| ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| CUDA / cuDNN matching your cluster drivers                                                                 | `sf` CLI, `forge-console`                           |
| Python 3.10 to 3.13 (runner `requires_python` is `>=3.10,<3.14`)                                           | `.forge/` tokens, `HF_TOKEN` as a layer             |
| Your trainer (transformers, deepspeed, vLLM, …)                                                            | Datasets (pull at runtime or use platform datasets) |
| `starforge-core` if you want `import starforge` under the catalog custom recipe (`external` observability) | A baked-in console URL                              |

Logs: `print`, `logging`, and framework stdout/stderr go to the console log tab. Do not POST `/api/ingest/logs`.

Curves: not parsed from stdout. Call `starforge.report` from training code.

## A starting Dockerfile

This matches the style of the platform TRL image: CUDA devel base, a venv on `PATH`, no StarForge package required if you will install the wheel in a later `RUN` (recommended for catalog `custom/custom`).

```dockerfile theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# syntax=docker/dockerfile:1
ARG BASE_IMAGE=nvidia/cuda:12.8.0-devel-ubuntu24.04
FROM ${BASE_IMAGE}

ARG PYTHON_VERSION=3.12
ENV DEBIAN_FRONTEND=noninteractive \
    HF_HUB_ENABLE_HF_TRANSFER=1

RUN apt-get update && apt-get install -y --no-install-recommends \
        git curl ca-certificates \
    && rm -rf /var/lib/apt/lists/*

COPY --from=ghcr.io/astral-sh/uv:0.8 /uv /usr/local/bin/uv

RUN uv python install ${PYTHON_VERSION} \
 && uv venv --python ${PYTHON_VERSION} /opt/train-venv
ENV VIRTUAL_ENV=/opt/train-venv PATH=/opt/train-venv/bin:$PATH

# Pin versions. Do not leave this on unpinned extras in production.
RUN uv pip install --no-cache \
        "torch" \
        "transformers" \
        "accelerate" \
        "trl" \
        "hf-transfer" \
        "starforge-core"

# Job Capsule uses the image Python. Keep this venv first on PATH.
CMD ["python", "-c", "import torch; print(torch.__version__)"]
```

Adjust packages to what `train.py` imports. If you need DeepSpeed JIT compile, a devel image (nvcc) is the usual requirement, same as `Dockerfile.trl`.

`local` executor runs the entrypoint with `bash`. `PATH` must still point at that venv so `python` inside `train.sh` is not `/usr/bin/python` without torch.

## Build and push

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
docker build -t myregistry.io/my-train:v1 .
docker push myregistry.io/my-train:v1

# Record the digest for submit:
docker buildx imagetools inspect myregistry.io/my-train:v1
```

Submit with a tag or a digest:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf submit my-custom --profile h200:8 \
  --image myregistry.io/my-train:v1 \
  --observability-url https://wandb.example/my-proj

sf submit my-custom --profile h200:8 \
  --image myregistry.io/my-train@sha256:abcd… \
  --observability-url https://wandb.example/my-proj
```

## Allowlist

`--image` is parsed for a registry host (`ghcr.io`, `localhost:5000`, `docker.io` when the name has no host). That host must appear in the server list `FORGE_ALLOWED_IMAGE_REGISTRIES` (comma-separated hostnames in `.env`).

| Server setting         | Custom `--image`                        | Catalog framework `--image` override                         |
| ---------------------- | --------------------------------------- | ------------------------------------------------------------ |
| List empty             | **Rejected** ("空：禁止用户镜像")               | Allowed (empty list does not restrict first-class overrides) |
| List set, host missing | Rejected with the allowed hosts printed | Rejected the same way                                        |
| Host present           | Allowed                                 | Allowed                                                      |

Operators: set something like `FORGE_ALLOWED_IMAGE_REGISTRIES=registry.example.com,ghcr.io`. Nodes must be able to pull (KubeRay imagePullSecrets, agent Docker login, Slurm enroot/SIF already converted).

## Executor notes

| Executor          | Image form                                                                                                                              |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `local` / `agent` | OCI. Agent nodes need the same pull credentials. Phase-1 agent is one node per job.                                                     |
| `kuberay`         | OCI + pull secret on the RayJob. Slow pulls look like `PENDING`.                                                                        |
| `slurm`           | Runtime registry usually points at SIF/SQSH. A raw Docker tag from a laptop often is not enough; ops convert and register `runtime_id`. |

If pull is slow, the job sits in `PENDING` until the preRunning deadline, then `FAILED`. That is not a training bug.

## Do not

* COPY the StarForge source tree into `/opt` and hope it matches the server's runner. The capsule is content-addressed and injected.
* Bake `HF_TOKEN` into a layer. The server injects `HF_TOKEN` / `HUGGING_FACE_HUB_TOKEN` when configured, or a secrets file path (`CLUSTER_SECRETS_FILE`).
* Point `FORGE_INGEST_URL` at `127.0.0.1` on the console host. Training nodes cannot use your laptop loopback. That variable is server-side; submitters do not set it, but empty charts with healthy stdout usually mean ops got it wrong.