Build a custom image

Dockerfile, what not to bake in, registry allowlist, tags vs digests

FROM nvidia/cuda:12.4.1-devel-ubuntu22.04
RUN pip install torch transformers trl accelerate
COPY . /workspace

A training image holds CUDA, Python, and whatever your train.sh imports. That is all.

Do not pip-install the StarForge CLI into the image, and do not copy observability code out of an old tutorial repository. The platform runtime is injected at launch as a content-addressed PEX — the image needs nothing for reporting to work.

What belongs in the image

Put in the image	Leave out
CUDA / cuDNN matching your cluster drivers	sf CLI, forge-console
Python 3.10 to 3.13 (runner requires_python is >=3.10,<3.14)	.forge/ tokens, HF_TOKEN as a layer
Your trainer (transformers, deepspeed, vLLM, …)	Datasets (pull at runtime or use platform datasets)
starforge-core if you want import starforge under the catalog custom recipe (external observability)	A baked-in console URL

Logs: print, logging, and framework stdout/stderr go to the console log tab. Do not POST /api/ingest/logs.

Curves: not parsed from stdout. Call starforge.report from training code.

A starting Dockerfile

This matches the style of the platform TRL image: CUDA devel base, a venv on PATH, no StarForge package required if you will install the wheel in a later RUN (recommended for catalog custom/custom).

# syntax=docker/dockerfile:1
ARG BASE_IMAGE=nvidia/cuda:12.8.0-devel-ubuntu24.04
FROM ${BASE_IMAGE}

ARG PYTHON_VERSION=3.12
ENV DEBIAN_FRONTEND=noninteractive \
    HF_HUB_ENABLE_HF_TRANSFER=1

RUN apt-get update && apt-get install -y --no-install-recommends \
        git curl ca-certificates \
    && rm -rf /var/lib/apt/lists/*

COPY --from=ghcr.io/astral-sh/uv:0.8 /uv /usr/local/bin/uv

RUN uv python install ${PYTHON_VERSION} \
 && uv venv --python ${PYTHON_VERSION} /opt/train-venv
ENV VIRTUAL_ENV=/opt/train-venv PATH=/opt/train-venv/bin:$PATH

# Pin versions. Do not leave this on unpinned extras in production.
RUN uv pip install --no-cache \
        "torch" \
        "transformers" \
        "accelerate" \
        "trl" \
        "hf-transfer" \
        "starforge-core"

# Job Capsule uses the image Python. Keep this venv first on PATH.
CMD ["python", "-c", "import torch; print(torch.__version__)"]

Adjust packages to what train.py imports. If you need DeepSpeed JIT compile, a devel image (nvcc) is the usual requirement, same as Dockerfile.trl.

local executor runs the entrypoint with bash. PATH must still point at that venv so python inside train.sh is not /usr/bin/python without torch.

Build and push

docker build -t myregistry.io/my-train:v1 .
docker push myregistry.io/my-train:v1

# Record the digest for submit:
docker buildx imagetools inspect myregistry.io/my-train:v1

Submit with a tag or a digest:

sf submit my-custom --profile h200:8 \
  --image myregistry.io/my-train:v1 \
  --observability-url https://wandb.example/my-proj

sf submit my-custom --profile h200:8 \
  --image myregistry.io/my-train@sha256:abcd… \
  --observability-url https://wandb.example/my-proj

Allowlist

--image is parsed for a registry host (ghcr.io, localhost:5000, docker.io when the name has no host). That host must appear in the server list FORGE_ALLOWED_IMAGE_REGISTRIES (comma-separated hostnames in .env).

Server setting	Custom --image	Catalog framework --image override
List empty	Rejected (“空：禁止用户镜像”)	Allowed (empty list does not restrict first-class overrides)
List set, host missing	Rejected with the allowed hosts printed	Rejected the same way
Host present	Allowed	Allowed

Operators: set something like FORGE_ALLOWED_IMAGE_REGISTRIES=registry.example.com,ghcr.io. Nodes must be able to pull (KubeRay imagePullSecrets, agent Docker login, Slurm enroot/SIF already converted).

Executor notes

Executor	Image form
local / agent	OCI. Agent nodes need the same pull credentials. Phase-1 agent is one node per job.
kuberay	OCI + pull secret on the RayJob. Slow pulls look like PENDING.
slurm	Runtime registry usually points at SIF/SQSH. A raw Docker tag from a laptop often is not enough; ops convert and register runtime_id.

If pull is slow, the job sits in PENDING until the preRunning deadline, then FAILED. That is not a training bug.

Do not

• COPY the StarForge source tree into /opt and hope it matches the server’s runner. The capsule is content-addressed and injected.
• Bake HF_TOKEN into a layer. The server injects HF_TOKEN / HUGGING_FACE_HUB_TOKEN when configured, or a secrets file path (CLUSTER_SECRETS_FILE).
• Point FORGE_INGEST_URL at 127.0.0.1 on the console host. Training nodes cannot use your laptop loopback. That variable is server-side; submitters do not set it, but empty charts with healthy stdout usually mean ops got it wrong.