> ## Documentation Index
> Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# How resources are requested

> Hardware profiles and GPU series, quotas, schedule windows, and multiple resource pools

The only resource flag is `--profile`: GPU type and count. Node topology, parallelism, and NCCL env come from the server-side hardware registry. The client does not invent a second resource declaration.

## Profiles and Series

| Concept               | Meaning                                                                                                            | Examples                                    |
| --------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------- |
| **Series (GPU type)** | The spec of one class of accelerator: compute weight, memory, Ray resource pins, Slurm constraint, total GPU count | `h200`, `h100`                              |
| **Profile**           | A GPU type + a default shape + tuning overrides for that shape (parallelism, vLLM memory ratio, etc.)              | `h200` (1×8), `h200-2g` (1×2), `h100` (1×1) |

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf status                        # View your available profiles and cluster headroom
sf submit my-exp --profile h200          # The registry's default shape (1 node, 8 GPUs)
sf submit my-exp --profile h200:4        # 4 GPUs
sf submit my-exp --profile h200:16       # 16 GPUs = 2 full nodes
```

The profile registry is maintained by admins (built-in defaults + `FORGE_HARDWARE_SERIES` / `FORGE_CLUSTER_PROFILES` overrides). Profiles also deliver process environment (NCCL / Ray memory thresholds / PyTorch allocator) and framework overrides (all-ones parallelism, vLLM `gpu_memory_utilization`, etc.) — these are **not** in the experiment config, so switching GPU types requires no experiment changes.

## Multiple Resource Pools (Heterogeneous Training)

The train and rollout stages of RL training can be pinned to different GPU types (e.g. H100 for training + H20 for sampling), by repeating `--profile` with a role prefix:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
sf submit my-exp \
  --profile train=h200:8 \
  --profile rollout=h100:2
```

Assembly maps this per backend: **kuberay** gets one worker group per pool (nodeSelector by GPU-type label, Ray custom resources). **slurm** gets a hetjob, one component per pool, `--constraint` from the series' `slurm_constraint`. All pools need the same `gpus_per_node`. Slurm rejects a series with no constraint. `local` and `agent` (Phase 1, one node per job) do not support multiple pools.

## Quotas

Admins set soft quotas per user, enforced at submission and dequeue time:

| Dimension              | Description                                |
| ---------------------- | ------------------------------------------ |
| **Concurrent GPU cap** | Total GPUs running + reserved in queue     |
| **Concurrent job cap** | Number of simultaneously active jobs       |
| **Daily GPU-hours**    | Daily cumulative usage cap (0 = unlimited) |
| **Allowed profiles**   | Restricts usable GPU types (empty = all)   |
| **Priority**           | Queue-ordering weight                      |

Over-quota submissions wait in the queue (rather than being rejected outright); current usage levels are visible on the console **Usage** page and via `sf status`.

## Schedule Windows

Admins can configure daily run windows per GPU type (e.g. prioritize inference during the day, open training at night). When the window is closed:

* new jobs are not dequeued (jobs are also held back when the remaining window time is below the configured threshold);
* running jobs receive a grace period and are then paused, **preserving checkpoints**;
* when the window reopens, training resumes automatically.

## GPU Ledger

Every job records `gpu_seconds` (actual GPU occupancy time × GPU count), which drives daily reports and the usage page. In simulation mode (GPU-less test environments), the accounting semantics are unchanged — see [E2E Closed-Loop Testing](/en/ops/e2e-testing).