Skip to main content
For a handful of GPU machines, when you do not want Kubernetes or Slurm. Each machine runs one forgelet daemon; the console places jobs over a narrow HTTP API — launch, observe, stop, logs, reap. The container spec is the same one local builds. GPU selection and docker run happen on the machine, under a single lock, and allocation truth stays in the container labels.
A job must fit on one node. Multi-node gang scheduling is not implemented, and a submission that needs it is rejected at admission rather than half-placed.

​
Prerequisites

  1. Docker + NVIDIA Container Toolkit on every GPU machine
  2. Shared storage mounted at the same path on the console host and every machine (FORGE_STORAGE_ROOT)
  3. A node Fleet on the console — Fleets → New fleet, or seeded at first start with FORGE_DEFAULT_FLEET_KIND=node

​
Registering a machine

Registration is an administrator’s job, not a user’s. The Fleet page mints a join token and shows one line to paste on the machine:
It installs the starforge-forgelet package, registers the machine, and starts the daemon. Running it again on a machine that already has forgelet is the upgrade path. On a machine that already has the package — a re-join, or one your configuration management installed — the same registration without the installer:
The join token is the whole authorization: you need root on the machine, not an account on the console. What comes back is this node’s identity and a credential, written under FORGE_NODE_STATE_DIR; forgelet serve heartbeats with them. Mixed GPU types need nothing: the node reports its own cards at join, and the console maps the card name onto a hardware series. A card the registry does not know is reported at join so an administrator can add it — until then, a job that pins a series is not placed there.

​
Node configuration

Set on the machine, not on the console:
Run forgelet serve as a systemd service. The daemon does not train; it only starts job containers.

​
Console configuration

FORGE_NODE_ADDRESSES and FORGE_NODE_TOKEN name a node set in the console’s own configuration. They are the pre-registration path, kept for a deployment that has not registered its machines yet; a Fleet with registered nodes never reads them.

​
Semantics

Least-loaded fit among nodes whose series matches the request and that have enough free GPUs. If nothing fits, launch raises NoCapacity and the job stays QUEUED.
Observe returns no signal for jobs on an unreachable node. Reconciliation skips that round. When the node returns, state converges from the real container. Those GPUs show as blocked in capacity while the node is down.
First pull on a node is async. The job stays PENDING with a pull note. FORGE_NODE_LAUNCH_TIMEOUT_S covers start when the image is already local.
Draining a node stops new placement and leaves running jobs alone. It is how a machine leaves the fleet without killing work.
Longer notes: docs/ops/executor-backends.md in this repository.