How resources are requested

Hardware profiles and GPU series, quotas, schedule windows, and multiple resource pools

The only resource flag is --profile: GPU type and count. Node topology, parallelism, and NCCL env come from the server-side hardware registry. The client does not invent a second resource declaration.

Profiles and Series

Concept	Meaning	Examples
Series (GPU type)	The spec of one class of accelerator: compute weight, memory, Ray resource pins, Slurm constraint, total GPU count	h200, h100
Profile	A GPU type + a default shape + tuning overrides for that shape (parallelism, vLLM memory ratio, etc.)	h200 (1×8), h200-2g (1×2), h100 (1×1)

sf status                        # View your available profiles and cluster headroom
sf submit my-exp --profile h200          # The registry's default shape (1 node, 8 GPUs)
sf submit my-exp --profile h200:4        # 4 GPUs
sf submit my-exp --profile h200:16       # 16 GPUs = 2 full nodes

The profile registry is maintained by admins (built-in defaults + FORGE_HARDWARE_SERIES / FORGE_CLUSTER_PROFILES overrides). Profiles also deliver process environment (NCCL / Ray memory thresholds / PyTorch allocator) and framework overrides (all-ones parallelism, vLLM gpu_memory_utilization, etc.) — these are not in the experiment config, so switching GPU types requires no experiment changes.

Multiple Resource Pools (Heterogeneous Training)

The train and rollout stages of RL training can be pinned to different GPU types (e.g. H100 for training + H20 for sampling), by repeating --profile with a role prefix:

sf submit my-exp \
  --profile train=h200:8 \
  --profile rollout=h100:2

Assembly maps this per backend: kuberay gets one worker group per pool (nodeSelector by GPU-type label, Ray custom resources). slurm gets a hetjob, one component per pool, --constraint from the series’ slurm_constraint. All pools need the same gpus_per_node. Slurm rejects a series with no constraint. local and agent (Phase 1, one node per job) do not support multiple pools.

Quotas

Admins set soft quotas per user, enforced at submission and dequeue time:

Dimension	Description
Concurrent GPU cap	Total GPUs running + reserved in queue
Concurrent job cap	Number of simultaneously active jobs
Daily GPU-hours	Daily cumulative usage cap (0 = unlimited)
Allowed profiles	Restricts usable GPU types (empty = all)
Priority	Queue-ordering weight

Over-quota submissions wait in the queue (rather than being rejected outright); current usage levels are visible on the console Usage page and via sf status.

Schedule Windows

Admins can configure daily run windows per GPU type (e.g. prioritize inference during the day, open training at night). When the window is closed:

• new jobs are not dequeued (jobs are also held back when the remaining window time is below the configured threshold);
• running jobs receive a grace period and are then paused, preserving checkpoints;
• when the window reopens, training resumes automatically.

GPU Ledger

Every job records gpu_seconds (actual GPU occupancy time × GPU count), which drives daily reports and the usage page. In simulation mode (GPU-less test environments), the accounting semantics are unchanged — see E2E Closed-Loop Testing.