Chat with a checkpoint

Spin up a vLLM service from training artifacts in one step, with web chat and A/B comparison

After training, start a vLLM OpenAI-compatible server from the artifact and chat in the browser. Two services can sit side by side (A/B). Supported on local and kuberay only.

Start a service

CLI

sf serve start run:<RUN_ID> --gpus 1          # serve a training artifact (hf_export)
sf serve start Qwen/Qwen3.5-9B --gpus 2       # any HF model (--gpus is the tensor parallelism degree)
sf serve start /shared/models/my-model        # shared-disk path

sf serve ls                    # my services (with endpoints and remaining TTL)
sf serve extend <SERVE_ID> --ttl-hours 4      # extend
sf serve stop <SERVE_ID>      # stop and release GPUs

Console

Playground page (/playground) → pick a model (training artifact dropdown / HF id input) → choose GPU count → start. The page shows service status, remaining TTL, and an extend button.

Inference services consume real GPU quota. Services have an idle TTL (default set by server configuration) and stop automatically on expiry to release GPUs — this is protection against “started it and forgot to stop it” hogging GPUs, not a fault. If you need it for longer, extend it.

Chat and A/B comparison

• Chat: streaming, temperature / max tokens, system prompt
• A/B: same prompt to two services (SFT vs GRPO artifact, for example)
• Endpoints are OpenAI-compatible; sf serve ls prints URL and token

Execution backend semantics

Backend	Service form
local	Docker container (vLLM), port on the console host
agent	Not supported
kuberay	Standalone vLLM Pod (not a RayJob — training backoff/TTL does not apply). Reached via the K8s API proxy
slurm	Not supported (compute nodes usually have no inbound path from the console)

Permissions

The Playground is a configurable advanced feature: administrators can enable it per role / user. When not enabled, the page shows a guidance card instead of an error.