> ## Documentation Index > Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Bring your own trainer > custom/custom end to end: train.sh, env vars, metrics, checkpoints, and what the platform will not guess ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf new my-trainer --method custom/custom # write experiments/my-trainer/train.sh sf submit my-trainer --profile h200:8 --image registry.example.com/my-trainer:v1 ``` Use `custom/custom` when your trainer is not in the catalog: a private fork, a research loop, Axolotl, a one-off script. The platform runs exactly **one** file: `experiments//train.sh`. It does not read a `FRAMEWORK` variable, does not look for a `run.py`, and never falls back to custom because another adapter failed. What that file does is entirely yours. Do not reach for custom just to change a learning rate. The catalog methods already wire metrics, checkpoints and images; going custom hands all three back to you. ## 1. Scaffold ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} cd my-lab sf new my-custom --method custom/custom ``` You get `experiments/my-custom/` with `config.yaml` (unused unless your script reads it), `recipe.lock.json`, `README.md`, and `train.sh`. The catalog entrypoint is `kind: experiment`, `value: train.sh`. Rename that file and submit will fail with "custom 入口不存在或越界". `train.sh` must be a file inside the experiment directory. The adapter runs: ```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} bash [optional action args] ``` Working directory is the **job package root** (`FORGE_WORK_DIR`), not the experiment folder. Point Python at `"${FORGE_EXP_DIR}/train.py"`. The custom adapter only compiles `operation=train`. `sf export` / `sf eval` against a custom experiment are not supported by this adapter. ## 2. Environment contract The template already `:?` dies if these are missing. Do not rename them. | Variable | Meaning | | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `FORGE_WORK_DIR` | Unpacked job package root | | `FORGE_EXP_DIR` | This experiment directory | | `FORGE_OUT_DIR` | Output directory (already created): `/runs////out`. Resolved by the control plane; do not compose it yourself | | `FORGE_FRAMEWORK` / `FORGE_RECIPE` | `custom` / `custom` (recipe id is `custom/custom` in the lock) | | `FORGE_CLUSTER_NUM_NODES` | Node count the server billed | | `FORGE_CLUSTER_GPUS_PER_NODE` | GPUs per node the server billed | | `STARFORGE_ENDPOINT` / `STARFORGE_RUN_ID` / `STARFORGE_TOKEN` | Ingest credentials. Absent on a laptop `python train.py` | | `STARFORGE_ENABLED` | `1` when ingest is bound, else `0` | Also injected for every job (see the [env reference](/en/ops/configuration)): `HF_TOKEN` when the server has one, `CLUSTER_PROFILE`, `NRL_RUN_ID`, recipe digests, optional `STARFORGE_JUDGE_*` and `STARFORGE_SANDBOX_*`. Quota and the watchdog use `FORGE_CLUSTER_*`. Occupying more GPUs than that number will get the job warned or stopped. Pass the same numbers into `accelerate launch --num_processes` / `torchrun --nproc_per_node`. ## 3. Three things the script must do 1. Write checkpoints, logs, and exports under `$FORGE_OUT_DIR`. Anything in the scratch work tree is gone when the container exits. 2. Report scalars with `starforge.report` if you want console charts. Stdout is logs only. The platform does not parse `loss=` lines. 3. Honor `FORGE_CLUSTER_*`. Recipe artifact globs (what the platform expects to find under the output dir): | Glob | Typical use | | ------------------ | --------------------------------------------------------------- | | `checkpoints/*` | Weights | | `logs` | Text logs you write yourself (optional; stdout already streams) | | `hf_export` | HuggingFace export directory | | `eval/report.json` | If you write an eval report by hand | Paths are realpath-checked and must stay inside `FORGE_OUT_DIR`. ## 4. `train.sh` you can actually run Replace the scaffold `exit 1` with a launch. `set -euo pipefail` is already there. Single process: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} exec python "${FORGE_EXP_DIR}/train.py" \ --output-dir "${FORGE_OUT_DIR}" \ --config "${FORGE_EXP_DIR}/config.yaml" ``` HuggingFace Accelerate, one node, one process per GPU: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} exec accelerate launch \ --num_processes "${FORGE_CLUSTER_GPUS_PER_NODE}" \ "${FORGE_EXP_DIR}/train.py" \ --output_dir "${FORGE_OUT_DIR}" \ --config "${FORGE_EXP_DIR}/config.yaml" ``` Do not `cd` into a random directory and write `./checkpoints`. Use the variables. ## 5. Metrics: `starforge.report` PyPI name `starforge-core`, import `starforge`. The module does not import transformers or Ray. Reporting never raises into your training loop. No `STARFORGE_TOKEN` means no network (local runs, unit tests). Set `STARFORGE_ENABLED=0` to force that off. ### Manual loop ```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} from starforge.report import init, log, finish init(hparams={"lr": 1e-5, "batch_size": 8}) for step, batch in enumerate(loader): loss = train_one(batch) log({"train/loss": float(loss)}, step=step) finish() ``` `init()` is idempotent. Nested dicts are flattened. Non-scalars that cannot be reduced to a mean are dropped. `prefix=` prepends a namespace if the key does not already have it. Hardware sampling starts in `init(monitor_hardware=True)` unless another component already set the hardware-bridge env. Interval: `STARFORGE_MONITOR_INTERVAL` (seconds, default `10`). ### HuggingFace / TRL callback `StarForgeCallback` is duck-typed (it does not subclass `TrainerCallback`). It calls `init` on train begin, `log` on `on_log`, `log(..., prefix="validation")` on evaluate, `finish` on train end. ```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} from starforge.report import StarForgeCallback from trl import SFTTrainer trainer = SFTTrainer(..., callbacks=[StarForgeCallback()]) trainer.train() ``` ### From a reward or env ```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} from starforge.report import log log({"env/tool_success_rate": 0.83, "env/avg_turns": 3.2}, step=step) ``` You do not need `init()` first. With credentials, `log()` opens a session without the hardware thread. Do not POST `/api/ingest/logs` yourself. Stdout is already forwarded. A second path duplicates lines. Do not hard-code the console URL. The container already has `STARFORGE_ENDPOINT`. ## 6. Observability: `external` vs `platform` The catalog `custom/custom` recipe sets `adapter_options.observability: external`. That has two consequences: * Submit **requires** `--observability-url` (any URL your team uses for wandb/swanlab/etc.). Missing it fails compile: `custom external observability 要求 spec.framework.observability_url`. * The adapter does **not** rewrite `PYTHONPATH`. `import starforge` works only if the wheel is in the image (or you put it on `PYTHONPATH` yourself). ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf submit my-custom --profile h200:8 \ --image myregistry.io/my-train:v1 \ --observability-url https://wandb.example/my-proj ``` `--observability-url` is stored as `FORGE_EXTERNAL_OBSERVABILITY_URL` in the process env. The platform does not start wandb for you. If the recipe were `observability: platform` (a catalog change by the people who ship `starforge-core`): * `--observability-url` is **forbidden**. * The runner prepends the capsule / kernel roots to `PYTHONPATH`, so `from starforge.report import log` works without installing the wheel in the image. You cannot flip that flag from the experiment directory. Changing it means a new recipe in the catalog. For a normal user who wants console charts today: **install `starforge-core` in the image** and still pass `--observability-url` because the published recipe is `external`. `STARFORGE_ENABLED=1` is independent: that is the ingest binding the server always sets when the job has an ingest token. Catalog custom still wants the extra URL field. ## 7. Submit ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} git add -A && git commit -m "custom trainer" sf submit my-custom --profile h200:8 \ --image myregistry.io/my-train:v1 \ --observability-url https://wandb.example/my-proj ``` `--image` is required for custom. Empty `FORGE_ALLOWED_IMAGE_REGISTRIES` **rejects** custom user images (first-class frameworks can still use deployment defaults). Ask an admin to add your registry hostname. Resolution for first-class frameworks is `--image` → console default → runtime registry → catalog. Custom has no catalog OCI pin (`runtime.default_version` is `user-managed`), so `--image` is the image. Tags work. Pin `@sha256:…` in production so a moving tag cannot change the job after admit. ## 8. Empty charts, job died immediately, import errors | Symptom | What to check | | --------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | Job fails in seconds, log says `还没填训练命令` / `exit 2` | You never replaced the scaffold `train.sh`. | | Logs flow, charts stay flat | No `log()` / `StarForgeCallback`. Or `import starforge` failed. Or still loading weights (wait). | | `ModuleNotFoundError: starforge` | Image has no wheel, and recipe is `external` so PYTHONPATH was not patched. | | `import starforge` works locally, not in the job | Different Python than the one `train.sh` execs. Align `PATH` with the image venv. | | Charts empty, logs empty | Container never started. Status `PENDING` / image pull. Events tab. | | Charts empty, ingest errors in server log | `FORGE_INGEST_URL` is `127.0.0.1` or not reachable from the GPU node. | | Checkpoints vanished | Wrote to cwd or `/tmp`. Use `$FORGE_OUT_DIR/checkpoints`. | | Allowlist error | Registry host of `--image` not in `FORGE_ALLOWED_IMAGE_REGISTRIES`. | | `custom adapter 尚不支持 'export'` | Custom has no export/eval compile path. | Inside the container: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} python -c "import starforge.report; print('ok')" echo "$FORGE_OUT_DIR" "$FORGE_CLUSTER_GPUS_PER_NODE" env | grep STARFORGE ``` ## 9. Optional `config.yaml` Nothing in the custom adapter reads Hydra. If you want `sf validate` to do something useful, you still only get what the **custom recipe `params:`** declares (currently empty). Treat `config.yaml` as your own file and parse it in `train.py`. `--set` on submit fills `spec.hyperparams`. Custom does not map those onto argv unless you write that yourself. `--model` / `--train-data` are the verl/TRL bindings. Custom ignores them unless your script looks at the JobSpec (it should not; use env and files you packed). ## 10. If you maintain the platform Shipping a first-class method (new framework or a custom variant with `observability: platform`) is a catalog change: `core/starforge/recipes/catalog///`, a `FrameworkAdapter`, tests, a digest-pinned image, then CLI/server handshake. Steps are in the repo file `docs/framework-adapters.md`. Users of a deployed console do not do that from `sf new`.