> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# node Backend

> Bare-metal machines: forgelet daemon, shared disk, one job per node

For a handful of GPU machines, when you do not want Kubernetes or Slurm. Each machine runs
one `forgelet` daemon; the console places jobs over a narrow HTTP API — launch, observe,
stop, logs, reap.

The container spec is the same one `local` builds. GPU selection and `docker run` happen on
the machine, under a single lock, and allocation truth stays in the container labels.

<Warning>
  A job must fit on **one node**. Multi-node gang scheduling is not implemented, and a
  submission that needs it is rejected at admission rather than half-placed.
</Warning>

## Prerequisites

1. Docker + NVIDIA Container Toolkit on every GPU machine
2. Shared storage mounted at the **same path** on the console host and every machine
   (`FORGE_STORAGE_ROOT`)
3. A `node` [Fleet](/en/ops/fleets) on the console — **Fleets → New fleet**, or seeded at
   first start with `FORGE_DEFAULT_FLEET_KIND=node`

## Registering a machine

Registration is an administrator's job, not a user's. The Fleet page mints a join token and
shows one line to paste on the machine:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
curl -fsSL https://forge.corp/install.sh | FORGE_FLEET=<fleet> FORGE_JOIN_TOKEN=<token> sh
```

It installs the `starforge-forgelet` package, registers the machine, and starts the daemon.
Running it again on a machine that already has forgelet is the upgrade path.

On a machine that already has the package — a re-join, or one your configuration management
installed — the same registration without the installer:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
forgelet join --server https://forge.corp --fleet <fleet> --token <token>
forgelet serve --port 7070
```

The join token is the whole authorization: you need root on the machine, not an account on
the console. What comes back is this node's identity and a credential, written under
`FORGE_NODE_STATE_DIR`; `forgelet serve` heartbeats with them.

Mixed GPU types need nothing: the node reports its own cards at join, and the console maps
the card name onto a hardware series. A card the registry does not know is reported at join
so an administrator can add it — until then, a job that pins a series is not placed there.

## Node configuration

Set on the machine, not on the console:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
FORGE_STORAGE_ROOT=/srv/starforge          # same path as on the console
FORGE_LOCAL_GPU_COUNT=8                    # 0 probes with nvidia-smi
FORGE_NODE_STATE_DIR=~/.starforge-forgelet # identity and credential
```

Run `forgelet serve` as a systemd service. The daemon does not train; it only starts job
containers.

## Console configuration

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
FORGE_STORAGE_ROOT=/srv/starforge         # same path on console and nodes
FORGE_NODE_REQUEST_TIMEOUT_S=15
FORGE_NODE_LAUNCH_TIMEOUT_S=120           # image already present; first pull is async
```

<Note>
  `FORGE_NODE_ADDRESSES` and `FORGE_NODE_TOKEN` name a node set in the console's own
  configuration. They are the pre-registration path, kept for a deployment that has not
  registered its machines yet; a Fleet with registered nodes never reads them.
</Note>

## Semantics

<AccordionGroup>
  <Accordion title="Placement" icon="map-pin">
    Least-loaded fit among nodes whose series matches the request and that have enough free GPUs. If nothing fits, launch raises `NoCapacity` and the job stays `QUEUED`.
  </Accordion>

  <Accordion title="Lost node ≠ dead job" icon="unplug">
    Observe returns no signal for jobs on an unreachable node. Reconciliation skips that round. When the node returns, state converges from the real container. Those GPUs show as `blocked` in capacity while the node is down.
  </Accordion>

  <Accordion title="Image pull" icon="box">
    First pull on a node is async. The job stays `PENDING` with a pull note. `FORGE_NODE_LAUNCH_TIMEOUT_S` covers start when the image is already local.
  </Accordion>

  <Accordion title="Draining" icon="power-off">
    Draining a node stops new placement and leaves running jobs alone. It is how a machine leaves the fleet without killing work.
  </Accordion>
</AccordionGroup>

Longer notes: `docs/ops/executor-backends.md` in this repository.