How StarForge fits together

CLI → control plane → executor → Job Capsule. One JobSpec contract.

StarForge is four layers around one contract (JobSpec + recipe catalog). Adding a method or swapping the cluster backend should not require changes in the other layers.

Layers

Layer	Where	Responsibility
CLI	core/starforge/cli/, package starforge-core	Experiments, local validation, packing, JobSpec. No cluster credentials.
Control plane	server/, package starforge-console	Auth, quotas, catalog handshake, dequeue, executor dispatch, ingest, UI
Executor	server/executors/	Turn a LaunchRequest into docker run / agent HTTP / a RayJob / slurmrestd. Watch status and reclaim.
Training runtime	Job Capsule injected by the server	bootstrap.sh checks the manifest, then runner.pex with the image’s Python. Training images do not preinstall the platform package.

Submit path

1. Client builds a JobSpec

   sf submit reads the experiment and recipe.lock.json, expands --profile against the hardware registry, validates hyperparameters, and packs the workspace (manifest + git provenance).

2. Server admits it

   Catalog handshake (method and framework version must be published), quota, image allowlist, HuggingFace preflight. Any failure rejects the submit.

3. Queue and assembly

   The job sits in QUEUED until the scheduler dequeues it. Assembly turns JobSpec + server config into a LaunchRequest and injects capsule.json, bootstrap.sh, and content-addressed runner.pex.

4. Executor launches

   local: docker run with atomic GPU allocation. agent: HTTP to forgelet on a node. kuberay: RayJob CR. slurm: slurmrestd allocation, then srun + Ray (hetjob for multi-pool).

5. Container start

   Entrypoint is bash .starforge/capsule/bootstrap.sh → python runner.pex run. The runner checks file digests and Python/framework capability, picks the recipe adapter, reports lifecycle events.

6. Streaming

   Stdout goes to the log tab. Curves go through starforge.report (wired for catalog methods; custom jobs call it themselves). Failures can trigger diagnostics.

Design choices that show up in day-to-day use

Secrets stay on the server

Cluster addresses, HF tokens, and object-storage keys live in the control plane. The client gets a personal access token. The training container gets a per-run, scoped ingest token.

No silent fallback

The platform does not guess the framework or entrypoint. Methods must be published in the catalog. Custom jobs must declare train.sh and an image. If the lockfile disagrees with the catalog, submit is rejected — upgrade with sf recipe upgrade.

Swap the executor, not the job

Assembly produces a backend-agnostic LaunchRequest. Executors implement launch / observe / stop / cleanup. Which one runs is the kind of the Fleet the job was placed on. There is no runtime fallback to another backend.

Traceable submits

Each run records git commit, config snapshot, recipe digest, runner/capsule digest, and image digest. Dirty workspaces are rejected unless you pass --allow-dirty.

Job states

Ledger statuses (not the same as a Docker/Ray/Slurm native state):

QUEUED jobs have not reached the cluster. PAUSED has released GPUs and kept the checkpoint. Auto-retry and maintenance drain reuse the pause/resume path.

Next

Recipes

Catalog, lockfiles, framework version matrix

Resources

Profiles, quotas, windows, multi-pool