Agent environments

Author a task suite an agent practises against: the manifest, the tasks, and the four protocols.

sf env push ./my-env --name calculator-tasks --version 1.0.0
sf env ls
sf env show alice/calculator-tasks@1.0.0

An environment is a taskset plus a manifest, owned and versioned like a dataset and referenced as <owner>/<name>@<version>. It is what an agent practises against. The platform stores it, decides who may reference it, and serves it — it never writes what runs inside it.

Layout

my-env/
├── manifest.json     # what the environment declares
└── tasks.jsonl       # one task per line

Everything under the directory is uploaded. Versions are immutable.

tasks.jsonl

One JSON object per line:

{"prompt": "What is 17% of 340?", "reference": "57.8", "split": "train"}
{"prompt": "Convert 45 degrees Celsius to Fahrenheit.", "reference": "113", "split": "eval"}

split · train | eval · default: "train"

Which half of the taskset this row belongs to. Declared per row.

The platform sets which half a job sees from the operation itself, so a training job cannot be handed the held-out tasks and an evaluation cannot ask for the training ones. An evaluation against a split the environment never declared is refused rather than quietly served the training tasks — that is the one failure this field exists to prevent, and it is the one that looks like a good score.

A taskset is never handed out. No route returns a copy, and there is no export. What a business considers hard is often more revealing than its training data, so a job references an environment and receives task content only inside the approved runtime.

manifest.json

{
  "schema": "forge/environment/v1",
  "protocol": "openenv",
  "verifier": { "kind": "rubric", "ref": "alice/arithmetic-correctness" },
  "tools": [
    {
      "name": "calculator",
      "endpoint": "/tools/calculator",
      "schema": {
        "type": "object",
        "properties": { "expression": { "type": "string" } },
        "required": ["expression"]
      }
    }
  ],
  "sandbox": { "required": false },
  "description": "Arithmetic word problems with a calculator tool."
}

schema · string · required

Always forge/environment/v1.

protocol · nemo-gym | openenv | openenv-remote | openenv-image · required

How a trainer reaches this environment. See below.

verifier · object · required

{"kind": "rubric|plugin|endpoint", "ref": "..."} — what decides whether a task was completed. See verifiers.

tools · array

What the agent may call: a name, an endpoint, and a JSON Schema for the arguments.

harness · object

{"plugin": "<owner>/<name>", "version": "...", "entrypoint": "module:ClassName", "digest": "..."}. Required for nemo-gym. The code belongs to a kind: environment plugin; the platform writes at most a two-line shim that imports it.

sandbox.required · bool · default: "false"

True when the harness runs model-generated code. A job that needs one, on a deployment with no sandbox provider configured, is refused — it does not fall back to running the code in the training container.

domain · string · default: "other"

NeMo Gym’s metrics-grouping category. Required and validated for nemo-gym, meaningless elsewhere. One of math, coding, agent, knowledge, instruction_following, long_context, safety, games, translation, e2e, rlhf, other.

image · string

The OCI image for an openenv environment. Resolved as a pinned runtime artifact like every other image; never pulled ad hoc.

endpoint · url

Required for openenv-remote, and forbidden for everything else. Where the service already runs.

The four protocols

Three of them are routes a job can actually take, and they differ in who runs the server.

openenv — the platform serves it

Turn-level. The platform runs the environment server from the taskset it holds, and the trainer drives the episode itself through openenv-core’s client: reset, step, state.

Used by TRL, SkyRL, Unsloth and Axolotl. No harness needed — the trainer already owns the loop.

nemo-gym — the framework runs it

Trajectory-level (/seed_session + /verify). NeMo Gym launches its own servers from a config entrypoint, so there is nothing for the platform to serve. Instead the platform materialises the environment: it writes the authorised version into the job as the files NeMo Gym expects to find.

Somebody has to run the trajectory loop, and the platform does not write it, so a harness.plugin is required — plus harness.entrypoint as module:ClassName, naming the SimpleResourcesServer subclass Gym imports.

openenv-remote — somebody else runs it

A service that already runs elsewhere. The platform never starts it and never proxies it; it decides whether the job may reach that host and hands the URL to the trainer. Costs an egress decision and nothing more.

sf env runner register http://envbox-3.corp:8080 \
  --name shared-sim --version 1.0.0 \
  --rubric alice/task-completion

That verifies the running service and registers it without needing an administrator.

openenv-image — not runnable here

Accepted by the manifest parser, but refused at admission. Hosting somebody else’s container alongside the job is the executor’s work and this deployment does not do it. The refusal is at submission rather than in the launcher, so you find out before an allocation is spent.

The harness

The code that drives a model through an environment, supplied as a kind: environment plugin.

Needed for the materialised route, where the framework imports and runs it. Not needed for the served turn-level route, where the trainer drives the episode itself.

plugin.yaml

schema: forge/plugin/v1
name: calculator-harness
version: 1.0.0
kind: environment
entrypoint: harness:CalculatorResources

harness.digest in the manifest pins the harness package the environment was authored against. Leave it empty when a team ships the harness alongside the environment; set it when the two are versioned apart and a mismatched harness would score differently. It is verified at submission, never at rollout.

Push and reference

sf env push ./my-env --name calculator-tasks --version 1.0.0 --public

--name · <owner>/<name> or bare name · required

A bare name goes into your own namespace.

--version · string · required

Immutable once pushed.

--public · flag

Only applies when the environment is first created.

The same parsing code runs on your laptop and in the control plane, from one module — so a manifest that pushes is a manifest that runs. An unknown protocol or verifier kind is refused at push time, not discovered at rollout, because an environment that validates and then cannot run has already cost somebody a GPU allocation.

Confirm it worked

sf env show alice/calculator-tasks@1.0.0

Prints the protocol, the verifier, the tools, and whether a sandbox is required. The console’s Environments page shows the same, plus which runs have referenced it.

Next

Verifiers

The three ways a task’s completion is decided, and their exact contracts.

Rubrics

Write the standard a judge scores against.