> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# E2E Closed-Loop Testing

> Run the full "submit → schedule → containerized training → ingest" chain without GPUs

Closed-loop test: real `sf submit` → console assembly and scheduling → a container actually trains → ingest of metrics/logs/artifacts → `SUCCEEDED` → assertions. Physical GPU attach is simulated. The rest is not mocked. Run it before go-live, after upgrades, and after executor changes.

## Running locally (macOS/OrbStack or any Docker machine)

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# local executor: simulated training workload (~10 seconds)
uv run python scripts/e2e/local_loop.py

# local executor: real TRL SFT (actual training of a tiny model on CPU)
uv run python scripts/e2e/local_loop.py --flavor sft

# kuberay executor: kind cluster + operator + MinIO distribution (~40 seconds)
bash scripts/e2e/setup-kind.sh
uv run python scripts/e2e/kuberay_loop.py
```

On failure, container logs / K8s events / platform-side logs are dumped automatically; `--keep` preserves the scene. The harness runs the real `sf` from this repository's `cli/`, creates an independent temporary project with `sf init`, then submits from there.

## GPU simulation switches

| Setting                         | Semantics                                                                                 |
| ------------------------------- | ----------------------------------------------------------------------------------------- |
| `FORGE_LOCAL_GPU_PASSTHROUGH=0` | local: allocation / accounting / labels work as usual, containers launch without `--gpus` |
| `FORGE_K8S_GPU_PASSTHROUGH=0`   | kuberay: Pods don't request `nvidia.com/gpu`, Ray logical resources work as usual         |

Scheduling, quotas, capacity, and the ledger all execute with real semantics. **Production keeps the default (passthrough).**

## Assertion coverage

1. The state machine reaches SUCCEEDED (passing through PENDING / RUNNING);
2. Metrics closed loop: `train/loss` can be read back from the API;
3. Logs closed loop: training log lines are stored and queryable;
4. Artifacts closed loop: checkpoint registration is visible.

## CI

`.github/workflows/e2e.yml`: unit tests → local closed loop (sim + real SFT matrix) → kuberay closed loop (kind + operator), with no cross-repository token.

## Coverage boundaries

The slurm backend is not yet in the closed loop (requires containerized slurmrestd + Apptainer): batch script rendering, the REST client, hetjob, and status mapping are covered by unit tests. Details and roadmap in the repo's `docs/ops/e2e-closed-loop.md`.