Upstream Version Adoption SOP

How the platform keeps up when NeMo-RL / verl / TRL ship new versions

The platform uses an exact-pinning strategy for upstream frameworks: every published version = one variant in the recipe version matrix + one execution artifact in the runtime registry. Adopting a new version is a declarative process; in most cases no platform code changes.

Standard steps

1. Build the execution artifact

   Build the training image for the new version (K8s/Docker: OCI digest-pinned) or SIF/SQSH (Slurm), publish it to the runtime registry, and assign a runtime_id.

2. Declare the version variant

   Add a new version entry under runtime.versions in the recipe YAML: runtime_id, dependencies, and the version-bound differences — entrypoint override (entrypoint), parameter path overrides (path_overrides), and certified observer patch versions (observer_versions).

3. Run adaptation tests

   SDK unit tests cover the version matrix (entrypoint / parameter paths / observers asserted per version); full regression via tests/.

4. Canary release

   Don’t make the new version default_version at first: early adopters opt in explicitly with sf new --framework-version X / sf recipe upgrade --framework-version X. Switch the default once it’s stable.

5. User upgrade

   After the catalog is published, existing experiment locks show drift hints; users upgrade explicitly with sf recipe upgrade — there is no silent switching.

Framework-specific notes

Framework	Common differences
NeMo-RL	Official YAML base updates (sync configs/base/), Megatron/DTensor config key migrations
verl	Training entrypoint consolidation (e.g. the unified main_ppo in 0.9), Ray version requirements
TRL	Trainer API signature changes, Accelerate config default changes

What not to do

• ❌ Write if version >= X branches in adapter code — version differences belong in recipe declarations;
• ❌ Use latest / branch names as versions — only exact semver or full commits are accepted;
• ❌ Reuse an old runtime_id to point at a new image — artifacts are immutable; new image, new id.

The full evaluation of this mechanism (why pinning is not over-coupling) is in the repo’s docs/ops/upstream-version-adoption.md.