> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability and ingest

> How metrics, logs, and artifacts get back to the console, and what to watch in production

Training containers get `STARFORGE_ENDPOINT` / `STARFORGE_RUN_ID` / `STARFORGE_TOKEN`. Catalog methods are instrumented by the runner. Custom training uses [`starforge.report`](/en/api-reference/python-sdk) for curves. stdout/stderr is forwarded by the runner as well. Scripts should not post logs themselves.

## Endpoints

These are for the runner and the SDK. You usually should not call them by hand.

| Endpoint                  | Data                                                    |
| ------------------------- | ------------------------------------------------------- |
| `/api/ingest/metrics`     | Training scalars (key / step / value, batched)          |
| `/api/ingest/logs`        | Log lines (chunks + eof)                                |
| `/api/ingest/validation`  | Validation samples (multi-turn conversations + rewards) |
| `/api/ingest/hardware`    | GPU utilization / memory time series                    |
| `/api/ingest/lifecycle`   | starting / running / succeeded / failed                 |
| `/api/ingest/artifact`    | Checkpoint / hf\_export / eval report registration      |
| `/api/ingest/benchmark`   | Standard benchmark scores (idempotent)                  |
| `/api/ingest/environment` | Custom environment metrics                              |

Ingest tokens are issued per run, scoped and time-limited (`FORGE_INGEST_TOKEN_DAYS` must cover the longest training job). The web guard rejects ingest tokens on regular APIs.

## Ops notes

<AccordionGroup>
  <Accordion title="FORGE_INGEST_URL must be reachable from training nodes" icon="network">
    Empty charts are most often `127.0.0.1` or an address outside the cluster. Containers / Pods / compute nodes need a direct path to that URL.
  </Accordion>

  <Accordion title="SSE and reverse proxies" icon="radio">
    Live logs are SSE: disable `proxy_buffering` for `/api` in nginx and relax read timeouts. Clients resume from a cursor after a disconnect.
  </Accordion>

  <Accordion title="Metric failures never stop training" icon="shield-check">
    SDK reporting is a side channel: failures log, they do not raise. Lifecycle is strict and is retried.
  </Accordion>

  <Accordion title="AI diagnostics" icon="stethoscope">
    Failed jobs trigger diagnostics (LLM optional). Inputs are the log tail, metric trends, validation samples, and a config snapshot. A diagnostics failure does not change job status.
  </Accordion>
</AccordionGroup>

## Custom training

Do not hand-roll urllib, and do not copy `common/telemetry.py` from the tutorial repo — that file is gone.

```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
from starforge.report import init, log, finish

init(hparams={"lr": 1e-5})
log({"loss": 0.42}, step=1)
finish()
```

Install `starforge-core`, import `starforge`. Image layout, `train.sh`, and empty-chart checks: [Custom training](/en/guides/custom-training).