Quota, GPU-hours over time, and cost attribution by project.
What you see
How the hours are counted
Attribution is counted at job end, and a running job contributes its elapsed portion so far. That is why the numbers move during the day and settle overnight, and why a long run appears to grow rather than landing all at once. The billing clock starts when training actually begins — therunning lifecycle mark — not when the
executor reports the container as up. The gap between the two is building a virtualenv and pulling
weights, which can be several minutes and is not training. Charging for it would systematically
overcount, and would penalise exactly the jobs on the slowest network links.
What you can do
- Switch attribution between by project and by user.
- Change the time range.
- Follow a project through to its project page.
When you are refused despite free GPUs
Three different limits refuse for three different reasons, and the message says which:
None of these is the cluster being full. A cluster-full refusal says so, and the scheduler will
eventually reserve capacity for a job that has been starved too long.