> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# How StarForge fits together

> CLI → control plane → executor → Job Capsule. One JobSpec contract.

StarForge is four layers around one contract (JobSpec + recipe catalog). Adding a method or swapping the cluster backend should not require changes in the other layers.

<img src="https://mintcdn.com/starforge/KiXW_1e-qWb7Hzzw/images/architecture.png?fit=max&auto=format&n=KiXW_1e-qWb7Hzzw&q=85&s=e83de821bd965982f939b83a432211d1" alt="StarForge — four layers around one JobSpec contract: your machine, the control plane, one of four executors, and the training container. Metrics, logs and artifacts flow back to ingest." className="block dark:hidden w-full" noZoom width="1536" height="1024" data-path="images/architecture.png" />

<img src="https://mintcdn.com/starforge/KiXW_1e-qWb7Hzzw/images/architecture-dark.png?fit=max&auto=format&n=KiXW_1e-qWb7Hzzw&q=85&s=eb3c5590a85463d7dafe7e6b12f96aad" alt="StarForge — four layers around one JobSpec contract: your machine, the control plane, one of four executors, and the training container. Metrics, logs and artifacts flow back to ingest." className="hidden dark:block w-full" noZoom width="1536" height="1024" data-path="images/architecture-dark.png" />

## Layers

| Layer            | Where                                           | Responsibility                                                                                                                         |
| ---------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| CLI              | `core/starforge/cli/`, package `starforge-core` | Experiments, local validation, packing, JobSpec. No cluster credentials.                                                               |
| Control plane    | `server/`, package `starforge-console`          | Auth, quotas, catalog handshake, dequeue, executor dispatch, ingest, UI                                                                |
| Executor         | `server/executors/`                             | Turn a `LaunchRequest` into `docker run` / agent HTTP / a RayJob / slurmrestd. Watch status and reclaim.                               |
| Training runtime | Job Capsule injected by the server              | `bootstrap.sh` checks the manifest, then `runner.pex` with the image's Python. Training images do not preinstall the platform package. |

## Submit path

<Steps>
  <Step title="Client builds a JobSpec">
    `sf submit` reads the experiment and `recipe.lock.json`, expands `--profile` against the hardware registry, validates hyperparameters, and packs the workspace (manifest + git provenance).
  </Step>

  <Step title="Server admits it">
    Catalog handshake (method and framework version must be published), quota, image allowlist, HuggingFace preflight. Any failure rejects the submit.
  </Step>

  <Step title="Queue and assembly">
    The job sits in `QUEUED` until the scheduler dequeues it. Assembly turns JobSpec + server config into a `LaunchRequest` and injects `capsule.json`, `bootstrap.sh`, and content-addressed `runner.pex`.
  </Step>

  <Step title="Executor launches">
    `local`: `docker run` with atomic GPU allocation. `agent`: HTTP to `forgelet` on a node. `kuberay`: RayJob CR. `slurm`: slurmrestd allocation, then `srun` + Ray (hetjob for multi-pool).
  </Step>

  <Step title="Container start">
    Entrypoint is `bash .starforge/capsule/bootstrap.sh` → `python runner.pex run`. The runner checks file digests and Python/framework capability, picks the recipe adapter, reports lifecycle events.
  </Step>

  <Step title="Streaming">
    Stdout goes to the log tab. Curves go through `starforge.report` (wired for catalog methods; custom jobs call it themselves). Failures can trigger diagnostics.
  </Step>
</Steps>

## Design choices that show up in day-to-day use

<AccordionGroup>
  <Accordion title="Secrets stay on the server" icon="key">
    Cluster addresses, HF tokens, and object-storage keys live in the control plane. The client gets a personal access token. The training container gets a per-run, scoped ingest token.
  </Accordion>

  <Accordion title="No silent fallback" icon="file-check">
    The platform does not guess the framework or entrypoint. Methods must be published in the catalog. Custom jobs must declare `train.sh` and an image. If the lockfile disagrees with the catalog, submit is rejected — upgrade with `sf recipe upgrade`.
  </Accordion>

  <Accordion title="Swap the executor, not the job" icon="server">
    Assembly produces a backend-agnostic `LaunchRequest`. Executors implement launch / observe / stop / cleanup. Which one runs is the kind of the Fleet the job was placed on. There is no runtime fallback to another backend.
  </Accordion>

  <Accordion title="Traceable submits" icon="git-commit-horizontal">
    Each run records git commit, config snapshot, recipe digest, runner/capsule digest, and image digest. Dirty workspaces are rejected unless you pass `--allow-dirty`.
  </Accordion>
</AccordionGroup>

## Job states

Ledger statuses (not the same as a Docker/Ray/Slurm native state):

```mermaid theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
stateDiagram-v2
    [*] --> QUEUED: submit accepted
    QUEUED --> SUBMITTED: dequeued
    SUBMITTED --> PENDING: launch returned
    PENDING --> RUNNING: container up
    RUNNING --> SUCCEEDED
    RUNNING --> FAILED
    RUNNING --> STOPPED: stop / window closed
    RUNNING --> PAUSED: pause / drain
    PAUSED --> QUEUED: resume
    FAILED --> QUEUED: auto-retry (budget)
    STOPPED --> QUEUED: resume from checkpoint
```

`QUEUED` jobs have not reached the cluster. `PAUSED` has released GPUs and kept the checkpoint. Auto-retry and maintenance drain reuse the pause/resume path.

## Next

<CardGroup cols={2}>
  <Card title="Recipes" icon="book-marked" href="/en/concepts/recipes">
    Catalog, lockfiles, framework version matrix
  </Card>

  <Card title="Resources" icon="cpu" href="/en/concepts/resources">
    Profiles, quotas, windows, multi-pool
  </Card>
</CardGroup>