Hardware Registry and Quotas

Ops configuration for GPU series, profiles, user quotas, and schedule windows

For user-facing semantics see the resource model; this page covers how ops configures it.

GPU series

Specs and scheduling parameters for one class of GPU (h200 / h100 built-in defaults, overridable via FORGE_HARDWARE_SERIES JSON):

Field	Purpose
id / label / weight	Identifier, display name, compute equivalence weight (for cross-series statistics)
ray_accel_types	Matches Ray acceleratorType
pin_resource	Ray custom resource key (for multi-pool pinning)
slurm_constraint	Slurm --constraint; required for slurm multi-pool
total_gpus	Once set, the authoritative capacity for quota admission
mem_gib	Per-GPU memory (for display/diagnostics reference)

Profiles

GPU series + default shape + tuning overrides (built-in defaults, overridable via FORGE_CLUSTER_PROFILES):

• name / series / num_nodes / gpus_per_node: the shape is the authoritative topology for quotas and training;
• env: process environment (NCCL, Ray memory thresholds, PyTorch allocator) — for multi-node, must be identical to what ray start sees;
• overrides: framework override entries (parallelism, vLLM memory fraction, micro batch). Only put keys shared by all experiments (CLI overrides use struct mode — changing a non-existent key raises an error).

The console’s Admin → Platform settings → Hardware offers visual editing with hot reload (instances pull on FORGE_HWCONFIG_REFRESH_INTERVAL).

User quotas

sf admin set-quota alice --gpus 16 --jobs 6 --daily-gpu-hours 200 \
  --max-storage-gb 2000 --profiles h200,h100 --priority 10

FORGE_QUOTA_ENFORCE=0 downgrades the whole system to “record only, don’t block” (for trial periods).

Storage quota

--max-storage-gb caps the total held by all of a user’s run directories (working copy + checkpoints + logs). It counts runs/<user>/; the caches (weights, datasets, corpora) are shared platform infrastructure and are not charged to anyone — the weights one job pulls, every later job uses.

Setting	Meaning
FORGE_STORAGE_SCAN_INTERVAL_S	Scan interval in seconds, 600 by default; 0 disables it (quota and dashboard then report “not scanned”)
FORGE_DISK_WATERMARK_PCT	Whole-volume watermark; crossing it pauses dequeuing for everyone, protecting the checkpoint writes of running jobs

They answer different questions: the quota is “who is holding too much”, the watermark is “the disk is about to fill”.

Schedule windows

Restrict daily run windows per GPU series (Admin → Platform settings → Schedules):

Setting	Semantics
schedule_enforce	Master enforcement switch (off = display only, no blocking)
Window definitions	Daily open intervals per series (service timezone FORGE_TIMEZONE)
schedule_dispatch_min_remaining_s	Don’t dequeue if less than this many seconds remain in the window
schedule_stop_grace_s	Grace period before force-stop after the window closes (jobs receive a pause first, checkpoint preserved)

Jobs paused this way resume training automatically when the window reopens.