E2E Closed-Loop Testing

Run the full “submit → schedule → containerized training → ingest” chain without GPUs

Closed-loop test: real sf submit → console assembly and scheduling → a container actually trains → ingest of metrics/logs/artifacts → SUCCEEDED → assertions. Physical GPU attach is simulated. The rest is not mocked. Run it before go-live, after upgrades, and after executor changes.

Running locally (macOS/OrbStack or any Docker machine)

# local executor: simulated training workload (~10 seconds)
uv run python scripts/e2e/local_loop.py

# local executor: real TRL SFT (actual training of a tiny model on CPU)
uv run python scripts/e2e/local_loop.py --flavor sft

# kuberay executor: kind cluster + operator + MinIO distribution (~40 seconds)
bash scripts/e2e/setup-kind.sh
uv run python scripts/e2e/kuberay_loop.py

On failure, container logs / K8s events / platform-side logs are dumped automatically; --keep preserves the scene. The harness runs the real sf from this repository’s cli/, creates an independent temporary project with sf init, then submits from there.

GPU simulation switches

Setting	Semantics
FORGE_LOCAL_GPU_PASSTHROUGH=0	local: allocation / accounting / labels work as usual, containers launch without --gpus
FORGE_K8S_GPU_PASSTHROUGH=0	kuberay: Pods don’t request nvidia.com/gpu, Ray logical resources work as usual

Scheduling, quotas, capacity, and the ledger all execute with real semantics. Production keeps the default (passthrough).

Assertion coverage

1. The state machine reaches SUCCEEDED (passing through PENDING / RUNNING);
2. Metrics closed loop: train/loss can be read back from the API;
3. Logs closed loop: training log lines are stored and queryable;
4. Artifacts closed loop: checkpoint registration is visible.

CI

.github/workflows/e2e.yml: unit tests → local closed loop (sim + real SFT matrix) → kuberay closed loop (kind + operator), with no cross-repository token.

Coverage boundaries

The slurm backend is not yet in the closed loop (requires containerized slurmrestd + Apptainer): batch script rendering, the REST client, hetjob, and status mapping are covered by unit tests. Details and roadmap in the repo’s docs/ops/e2e-closed-loop.md.