Observability and ingest

How metrics, logs, and artifacts get back to the console, and what to watch in production

Training containers get STARFORGE_ENDPOINT / STARFORGE_RUN_ID / STARFORGE_TOKEN. Catalog methods are instrumented by the runner. Custom training uses starforge.report for curves. stdout/stderr is forwarded by the runner as well. Scripts should not post logs themselves.

Endpoints

These are for the runner and the SDK. You usually should not call them by hand.

Endpoint	Data
/api/ingest/metrics	Training scalars (key / step / value, batched)
/api/ingest/logs	Log lines (chunks + eof)
/api/ingest/validation	Validation samples (multi-turn conversations + rewards)
/api/ingest/hardware	GPU utilization / memory time series
/api/ingest/lifecycle	starting / running / succeeded / failed
/api/ingest/artifact	Checkpoint / hf_export / eval report registration
/api/ingest/benchmark	Standard benchmark scores (idempotent)
/api/ingest/environment	Custom environment metrics

Ingest tokens are issued per run, scoped and time-limited (FORGE_INGEST_TOKEN_DAYS must cover the longest training job). The web guard rejects ingest tokens on regular APIs.

Ops notes

FORGE_INGEST_URL must be reachable from training nodes

Empty charts are most often 127.0.0.1 or an address outside the cluster. Containers / Pods / compute nodes need a direct path to that URL.

SSE and reverse proxies

Live logs are SSE: disable proxy_buffering for /api in nginx and relax read timeouts. Clients resume from a cursor after a disconnect.

Metric failures never stop training

SDK reporting is a side channel: failures log, they do not raise. Lifecycle is strict and is retried.

AI diagnostics

Failed jobs trigger diagnostics (LLM optional). Inputs are the log tail, metric trends, validation samples, and a config snapshot. A diagnostics failure does not change job status.

Custom training

Do not hand-roll urllib, and do not copy common/telemetry.py from the tutorial repo — that file is gone.

from starforge.report import init, log, finish

init(hparams={"lr": 1e-5})
log({"loss": 0.42}, step=1)
finish()

Install starforge-core, import starforge. Image layout, train.sh, and empty-chart checks: Custom training.