> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Fleets

> Registering machines: what a Fleet is, how a node joins, drain and liveness

A **Fleet** is a named set of machines and the backend that runs work on them. It is a
record you create, not a setting you export: one console carries several Fleets at once,
and every job and deployment records the Fleet it was admitted against.

This replaced a single deployment-wide backend variable. See
[ADR-0013](https://github.com/wccdev/starforge/blob/main/docs/adr/0013-an-execution-backend-is-a-registered-fleet.md)
for why. `FORGE_DEFAULT_FLEET_KIND` survives it, and seeds the first Fleet on a console
that has none — read once, at first start, and nowhere else.

## What a Fleet carries

| Field          | Values                                    | Changeable                                                      |
| -------------- | ----------------------------------------- | --------------------------------------------------------------- |
| `kind`         | `local` \| `node` \| `kuberay` \| `slurm` | **No.** It describes what the machines are                      |
| `delivery`     | `shared-mount` \| `object-pull`           | **No.** Re-pointing it relocates where every running job writes |
| `capabilities` | any of `train`, `serve`, `env`            | Yes                                                             |
| `visibility`   | `public` \| `project`                     | Yes                                                             |
| `state`        | `active` \| `draining` \| `disabled`      | Yes                                                             |
| `config`       | deployment settings this Fleet overrides  | Yes                                                             |

`config` holds the keys that differ between two Fleets of the same kind — the slurm REST
endpoint, the KubeRay namespace. A key no setting reads is logged as a warning rather than
ignored: a misspelled override is a setting an operator believes is in effect and is not.

<Note>
  A capability is a promise, not a preference. A Fleet without `serve` is not a Fleet that
  serves badly — a deployment naming it is refused.
</Note>

## Creating one

**Fleets → New fleet**. Pick the backend and what the Fleet is for; the delivery follows the
backend unless you change it. Or through the API:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
curl -X POST https://forge.corp/api/fleets \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"id": "gpu-lab", "kind": "node", "delivery": "shared-mount",
       "capabilities": ["train", "serve"]}'
```

Administrators only. Registering machines is an operator's job; picking a Fleet to submit
to is a user's.

A console with no Fleet at all seeds one on first start from `FORGE_DEFAULT_FLEET_KIND`,
so a zero-config deployment never has to make this call.

## Registering a machine

Only `node` Fleets have machines to register — `kuberay` and `slurm` get theirs from the
cluster manager, and `local` is the console's own host.

Mint a join token on the Fleet page — **Add a machine**. It is single-use, expires in 24 hours, and is stored
only as a hash. The page shows one line to paste on the machine:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
curl -fsSL https://forge.corp/install.sh | FORGE_FLEET=gpu-lab FORGE_JOIN_TOKEN=<token> sh
```

On a machine that already has the package:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
forgelet join --server https://forge.corp --fleet gpu-lab --token <token>
forgelet serve --port 7070
```

<Warning>
  The join token is the whole authorization. You need root on the machine, not an account
  on the console — which is what makes the one-liner safe to hand to whoever administers
  the box. Treat a token in a chat log as a standing key until it expires.
</Warning>

What comes back is the node's identity and a long-lived credential, written under
`FORGE_NODE_STATE_DIR`. **The credential is returned once.** Losing it means asking for
another token and joining again as a new node, which is correct: a machine that lost its
identity is not the same machine to a control plane reconciling containers against it.

The node reports its own inventory at join — card name, count, driver, container runtime,
and whether it can see the storage root. The console maps the card name onto a hardware
series. A card the registry does not know is named in the join output so an administrator
can add it; until then, jobs that pin a series are not placed there.

### An address is not an identity

A node is identified by the id the console issued, never by `host:port`. A machine that
moves stays the same machine. The consequence is that two records may legitimately answer
at one address, so joining where another live node already answers is a warning rather
than a refusal — printed by `forgelet join`, because the usual cause is a re-join that
left the old record behind, and then the console counts one machine's cards twice.

## Liveness

One `last_seen_at`, read against three thresholds, because the cost of being wrong differs
by what the machine is for:

| Capability | Quiet for | Consequence                                                                     |
| ---------- | --------- | ------------------------------------------------------------------------------- |
| `train`    | 300s      | Stops receiving new work. **Running jobs are untouched**                        |
| `serve`    | 45s       | Revisions on it are marked not-Ready and the deployment reports degraded        |
| `env`      | 45s       | The Fleet stops being a candidate; in-flight episodes fail on their own timeout |

These are three consequences, not three states. A node is declared `gone` on the **most
generous** of its Fleet's capabilities; the tighter rules answer per capability. "Do not
send traffic there now" and "do not place anything there at all" are different claims, and
a Fleet that declares all three would otherwise stop taking training work after 45 seconds
of silence.

Nothing about being `gone` kills anything. An unreachable node is not a dead job: observe
returns no signal for its jobs, reconciliation skips them, and state converges from the
real container when the machine comes back.

A `shared-mount` node that reports losing the storage root is cordoned, and un-cordoned
when the mount returns. An operator's own cordon is never lifted by a heartbeat — that one
is a decision, and a heartbeat is an observation.

## Drain, cordon, remove

|                  | Scope      | New work | Running work |
| ---------------- | ---------- | -------- | ------------ |
| Fleet `draining` | every node | stops    | continues    |
| Fleet `disabled` | every node | stops    | continues    |
| Node `draining`  | one node   | stops    | continues    |
| Node `cordoned`  | one node   | stops    | continues    |

None of them kills anything; that is the point. `cordoned` is for a machine that is coming
back, `draining` for one that is leaving.

Removing an active node is refused. Its identity is what makes its containers
reconcilable, so drain first and let the work finish. `force=true` exists for a machine
that is already gone; it is recorded in the audit log as forced, because what it leaves
behind is a container nothing can reconcile.

A Fleet is deletable only with no nodes and no live deployment pinned to it. A deployment
never changes Fleet, so deleting the Fleet under it would leave a row whose backend
resolves to nothing: unobservable, unstoppable, and still holding cards.

## Choosing a Fleet

| Workload           | How it picks                                                                                                                    |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------- |
| Training job       | Placed automatically among Fleets that can `train`, fit the requested series and card count, and are visible to the submitter   |
| Lifecycle job      | The same placement, on the same gates                                                                                           |
| Model deployment   | **Named by the submitter.** With one serving Fleet the choice is made for them; with more than one, an unnamed Fleet is refused |
| Volume, dataset    | Neither. They are storage, and reach a job through its Fleet's `delivery`                                                       |
| Playground session | Placed among Fleets that can `serve`; `local` and `kuberay` only                                                                |
| Environment        | **Named in its manifest.** The Fleet must declare `env`, and the address is resolved per job from a node answering right now    |

A deployment is the exception because its address outlives the decision and never moves,
and nothing on a Fleet record says which department owns the machines or what data may
touch them. The platform declines to answer a question it cannot answer.

Placement refuses at admission with a reason — capability, visibility, or inventory fit.
A job that no Fleet can hold is told so while the submitter is still looking at the
response, not hours later at the top of a queue.

<Note>
  Inventory fit is a check, not a reservation. The allocator remains authoritative for
  which cards a job actually gets; this only stops a request for hardware the Fleet has
  never had from queueing forever.
</Note>

## Visibility

A `project` Fleet is invisible outside its project, and a job naming one is refused as
**not found** rather than forbidden — a private Fleet's name does not leak through an
error message.