Model Deployments

Publish training exports or third-party models as recoverable, revisioned internal OpenAI services

Model Deployments (/deployments) provides durable internal serving and is separate from TTL-based Playground sessions. Every deployment has a stable endpoint, immutable revisions, and deployment-scoped tokens. Suspending releases its GPU allocation while retaining that identity.

Managed deployments, their revisions and endpoints.

Create a deployment

The wizard covers identity, model source, runtime settings, and review. Supported sources are:

• an hf_export from a StarForge run; export a regular checkpoint with sf export first;
• a Hugging Face repository, optionally using the current user’s linked HF credential for private models;
• a shared directory allowed by an administrator.

Choose vLLM or SGLang. Common settings cover GPU count, dtype, quantization, context length, concurrency, memory utilization, and the OpenAI model name. Engine-specific fields are progressively disclosed. StarForge accepts a typed allowlist only—there is no arbitrary shell-argument escape hatch.

trust_remote_code is disabled by default and is available only when an administrator permits it, because it executes code from the model repository.

Status and recovery

The normal progression is Deploying, Warming, then Ready. Failures use Degraded or Failed; a paused deployment is Suspended. Ready means the runtime is alive, health checks pass, /v1/models contains the expected alias, and a minimal generation warm-up succeeds.

The controller recreates a lost or repeatedly unhealthy single-replica runtime with bounded backoff. Suspend stops the runtime and releases GPUs; resume recreates it from the current revision.

Call the stable endpoint

The initial token is shown once after creation. Create, revoke, or rotate additional credentials under Settings → Deployment tokens.

curl "$STARFORGE_URL/inference/<deployment-id>/v1/chat/completions" \
  -H "Authorization: Bearer $DEPLOYMENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"model":"<served-model-name>","messages":[{"role":"user","content":"hello"}]}'

The detail page’s Playground tab uses the signed-in web session and never reads or exposes deployment tokens. Client traffic must use a deployment token; a web access token is not a substitute.

Responses API

A Ready deployment guarantees Responses creation and SSE streaming. The stable gateway also accepts GET, POST, and DELETE, so retrieval, cancellation, deletion, input-item listing, and related endpoints pass through when the selected engine implements them. The vLLM baseline guarantees create, retrieve, and cancel; newer SGLang releases expose a broader stateful surface.

curl "$STARFORGE_URL/inference/<deployment-id>/v1/responses" \
  -H "Authorization: Bearer $DEPLOYMENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"model":"<served-model-name>","input":"Hello","stream":true}'

Response and conversation state lives in the single runtime replica. Old response_id and conversation_id values may be lost after recovery, suspension, or a revision switch. Applications that need state across runtime replacement should retain complete input items and use store: false.

OpenAI-hosted built-ins such as web search, file search, and code interpreter are not automatically available on local engines. Function tools, reasoning, and structured outputs depend on the model, engine version, and configured parser.

Updates and rollback

A model or configuration change creates a new immutable revision. With spare GPU capacity, StarForge warms the candidate before moving the stable endpoint. If it fails, the old revision stays active. Without spare capacity, the user must explicitly accept a maintenance window; the platform never introduces silent downtime.

Use Revisions to inspect configuration snapshots and roll back. The detail page also exposes metrics, logs, and settings. Deleting a deployment stops its runtime and immediately revokes all tokens; it cannot be undone.