Bring your own trainer

custom/custom end to end: train.sh, env vars, metrics, checkpoints, and what the platform will not guess

sf new my-trainer --method custom/custom
# write experiments/my-trainer/train.sh
sf submit my-trainer --profile h200:8 --image registry.example.com/my-trainer:v1

Use custom/custom when your trainer is not in the catalog: a private fork, a research loop, Axolotl, a one-off script.

The platform runs exactly one file: experiments/<name>/train.sh. It does not read a FRAMEWORK variable, does not look for a run.py, and never falls back to custom because another adapter failed. What that file does is entirely yours.

Do not reach for custom just to change a learning rate. The catalog methods already wire metrics, checkpoints and images; going custom hands all three back to you.

1. Scaffold

cd my-lab
sf new my-custom --method custom/custom

You get experiments/my-custom/ with config.yaml (unused unless your script reads it), recipe.lock.json, README.md, and train.sh. The catalog entrypoint is kind: experiment, value: train.sh. Rename that file and submit will fail with “custom 入口不存在或越界”.

train.sh must be a file inside the experiment directory. The adapter runs:

bash <absolute path to train.sh> [optional action args]

Working directory is the job package root (FORGE_WORK_DIR), not the experiment folder. Point Python at "${FORGE_EXP_DIR}/train.py".

The custom adapter only compiles operation=train. sf export / sf eval against a custom experiment are not supported by this adapter.

2. Environment contract

The template already :? dies if these are missing. Do not rename them.

Variable	Meaning
FORGE_WORK_DIR	Unpacked job package root
FORGE_EXP_DIR	This experiment directory
FORGE_OUT_DIR	Output directory (already created): <storage root>/runs/<user>/<experiment>/<run_id>/out. Resolved by the control plane; do not compose it yourself
FORGE_FRAMEWORK / FORGE_RECIPE	custom / custom (recipe id is custom/custom in the lock)
FORGE_CLUSTER_NUM_NODES	Node count the server billed
FORGE_CLUSTER_GPUS_PER_NODE	GPUs per node the server billed
STARFORGE_ENDPOINT / STARFORGE_RUN_ID / STARFORGE_TOKEN	Ingest credentials. Absent on a laptop python train.py
STARFORGE_ENABLED	1 when ingest is bound, else 0

Also injected for every job (see the env reference): HF_TOKEN when the server has one, CLUSTER_PROFILE, NRL_RUN_ID, recipe digests, optional STARFORGE_JUDGE_* and STARFORGE_SANDBOX_*.

Quota and the watchdog use FORGE_CLUSTER_*. Occupying more GPUs than that number will get the job warned or stopped. Pass the same numbers into accelerate launch --num_processes / torchrun --nproc_per_node.

3. Three things the script must do

1. Write checkpoints, logs, and exports under $FORGE_OUT_DIR. Anything in the scratch work tree is gone when the container exits.
2. Report scalars with starforge.report if you want console charts. Stdout is logs only. The platform does not parse loss= lines.
3. Honor FORGE_CLUSTER_*.

Recipe artifact globs (what the platform expects to find under the output dir):

Glob	Typical use
checkpoints/*	Weights
logs	Text logs you write yourself (optional; stdout already streams)
hf_export	HuggingFace export directory
eval/report.json	If you write an eval report by hand

Paths are realpath-checked and must stay inside FORGE_OUT_DIR.

4. train.sh you can actually run

Replace the scaffold exit 1 with a launch. set -euo pipefail is already there.

Single process:

exec python "${FORGE_EXP_DIR}/train.py" \
  --output-dir "${FORGE_OUT_DIR}" \
  --config "${FORGE_EXP_DIR}/config.yaml"

HuggingFace Accelerate, one node, one process per GPU:

exec accelerate launch \
  --num_processes "${FORGE_CLUSTER_GPUS_PER_NODE}" \
  "${FORGE_EXP_DIR}/train.py" \
  --output_dir "${FORGE_OUT_DIR}" \
  --config "${FORGE_EXP_DIR}/config.yaml"

Do not cd into a random directory and write ./checkpoints. Use the variables.

5. Metrics: starforge.report

PyPI name starforge-core, import starforge. The module does not import transformers or Ray. Reporting never raises into your training loop. No STARFORGE_TOKEN means no network (local runs, unit tests). Set STARFORGE_ENABLED=0 to force that off.

Manual loop

from starforge.report import init, log, finish

init(hparams={"lr": 1e-5, "batch_size": 8})

for step, batch in enumerate(loader):
    loss = train_one(batch)
    log({"train/loss": float(loss)}, step=step)

finish()

init() is idempotent. Nested dicts are flattened. Non-scalars that cannot be reduced to a mean are dropped. prefix= prepends a namespace if the key does not already have it.

Hardware sampling starts in init(monitor_hardware=True) unless another component already set the hardware-bridge env. Interval: STARFORGE_MONITOR_INTERVAL (seconds, default 10).

HuggingFace / TRL callback

StarForgeCallback is duck-typed (it does not subclass TrainerCallback). It calls init on train begin, log on on_log, log(..., prefix="validation") on evaluate, finish on train end.

from starforge.report import StarForgeCallback
from trl import SFTTrainer

trainer = SFTTrainer(..., callbacks=[StarForgeCallback()])
trainer.train()

From a reward or env

from starforge.report import log

log({"env/tool_success_rate": 0.83, "env/avg_turns": 3.2}, step=step)

You do not need init() first. With credentials, log() opens a session without the hardware thread.

Do not POST /api/ingest/logs yourself. Stdout is already forwarded. A second path duplicates lines.

Do not hard-code the console URL. The container already has STARFORGE_ENDPOINT.

6. Observability: external vs platform

The catalog custom/custom recipe sets adapter_options.observability: external. That has two consequences:

• Submit requires --observability-url (any URL your team uses for wandb/swanlab/etc.). Missing it fails compile: custom external observability 要求 spec.framework.observability_url.
• The adapter does not rewrite PYTHONPATH. import starforge works only if the wheel is in the image (or you put it on PYTHONPATH yourself).

sf submit my-custom --profile h200:8 \
  --image myregistry.io/my-train:v1 \
  --observability-url https://wandb.example/my-proj

--observability-url is stored as FORGE_EXTERNAL_OBSERVABILITY_URL in the process env. The platform does not start wandb for you.

If the recipe were observability: platform (a catalog change by the people who ship starforge-core):

• --observability-url is forbidden.
• The runner prepends the capsule / kernel roots to PYTHONPATH, so from starforge.report import log works without installing the wheel in the image.

You cannot flip that flag from the experiment directory. Changing it means a new recipe in the catalog. For a normal user who wants console charts today: install starforge-core in the image and still pass --observability-url because the published recipe is external.

STARFORGE_ENABLED=1 is independent: that is the ingest binding the server always sets when the job has an ingest token. Catalog custom still wants the extra URL field.

7. Submit

git add -A && git commit -m "custom trainer"
sf submit my-custom --profile h200:8 \
  --image myregistry.io/my-train:v1 \
  --observability-url https://wandb.example/my-proj

--image is required for custom. Empty FORGE_ALLOWED_IMAGE_REGISTRIES rejects custom user images (first-class frameworks can still use deployment defaults). Ask an admin to add your registry hostname.

Resolution for first-class frameworks is --image → console default → runtime registry → catalog. Custom has no catalog OCI pin (runtime.default_version is user-managed), so --image is the image.

Tags work. Pin @sha256:… in production so a moving tag cannot change the job after admit.

8. Empty charts, job died immediately, import errors

Symptom	What to check
Job fails in seconds, log says 还没填训练命令 / exit 2	You never replaced the scaffold train.sh.
Logs flow, charts stay flat	No log() / StarForgeCallback. Or import starforge failed. Or still loading weights (wait).
ModuleNotFoundError: starforge	Image has no wheel, and recipe is external so PYTHONPATH was not patched.
import starforge works locally, not in the job	Different Python than the one train.sh execs. Align PATH with the image venv.
Charts empty, logs empty	Container never started. Status PENDING / image pull. Events tab.
Charts empty, ingest errors in server log	FORGE_INGEST_URL is 127.0.0.1 or not reachable from the GPU node.
Checkpoints vanished	Wrote to cwd or /tmp. Use $FORGE_OUT_DIR/checkpoints.
Allowlist error	Registry host of --image not in FORGE_ALLOWED_IMAGE_REGISTRIES.
custom adapter 尚不支持 'export'	Custom has no export/eval compile path.

Inside the container:

python -c "import starforge.report; print('ok')"
echo "$FORGE_OUT_DIR" "$FORGE_CLUSTER_GPUS_PER_NODE"
env | grep STARFORGE

9. Optional config.yaml

Nothing in the custom adapter reads Hydra. If you want sf validate to do something useful, you still only get what the custom recipe params: declares (currently empty). Treat config.yaml as your own file and parse it in train.py.

--set on submit fills spec.hyperparams. Custom does not map those onto argv unless you write that yourself.

--model / --train-data are the verl/TRL bindings. Custom ignores them unless your script looks at the JobSpec (it should not; use env and files you packed).

10. If you maintain the platform

Shipping a first-class method (new framework or a custom variant with observability: platform) is a catalog change: core/starforge/recipes/catalog/<framework>/<recipe>/, a FrameworkAdapter, tests, a digest-pinned image, then CLI/server handshake. Steps are in the repo file docs/framework-adapters.md. Users of a deployed console do not do that from sf new.