Score hardware and RTL work

VerilogEval v2, RTLLM v2, RTL-Repo, CVDP — compile the Verilog and run the testbench

sf bench run my-bench -m run:run-4f2a91 --suites verilogeval-v2

RTL evaluation is a different kind of thing from a text benchmark. GSM8K judges whether the answer is right; VerilogEval judges whether the generated circuit compiles and passes its testbench. That needs a simulator and pass@k over repeated sampling — a different execution chain entirely.

The platform has a third runner for it, rtl, alongside lm-eval and evalscope. Scores land in the same dashboard, so an RTL score and a GSM8K score sit on one row for the same model.

Available benchmarks

Benchmark	What it measures	Primary metric
verilogeval-v2	Natural-language spec → Verilog module (spec-to-RTL)	pass@1
verilogeval-v2-completion	Complete an implementation from the module header	pass@1
rtllm-v2	Design-level tasks (ALU, FIFO, pipeline units), synthesizable	func_pass
rtl-repo	Completion inside a real repository, testing long context and project conventions	exact_match
cvdp	NVIDIA’s comprehensive design problems: write, debug, testbench, spec Q&A (non-agentic track)	pass@1

sf bench run my-bench -m run:<RUN_ID> --suites verilogeval-v2,rtllm-v2

One evaluation uses one runner, so RTL benchmarks and GSM8K-style ones go in separate jobs.

rtl-repo’s metric is not comparable to the others. It runs no simulation (there is no testbench) and reports text similarity; ranking its exact_match next to VerilogEval’s pass@1 is wrong — they do not measure the same thing.

Admins: what the image needs

Two things.

The simulators are already in deploy/docker/Dockerfile.evalkit (iverilog + verilator).

The harness is yours to install. VerilogEval, RTLLM and CVDP each ship their own problem sets and testbenches in their own repositories. The platform deliberately does not copy those problems — when upstream revises a problem, a copy silently becomes a different benchmark while still carrying the original name. So the platform only defines the contract:

$FORGE_RTL_HARNESS --benchmark <name> --model <path> --output-dir <dir> \
                   [--samples N] [--limit N]

The harness writes <dir>/rtl_report.json:

{"results": {"verilogeval-v2-spec": {"pass@1": 0.42, "syntax_pass": 0.90}}}

The Dockerfile carries a commented installation example. Pin the upstream commit — evaluation results have to stay comparable across months, and one revision of the problem set invalidates every earlier score.

Without a harness, the job fails with an explicit error (“no rtl_report.json produced; check that the image has a harness”) rather than producing an empty report.

CVDP’s agentic track: possible, but not through this path

CVDP has two tracks. The non-agentic track is the one above: generate once, judge functionality, via the rtl runner.

The agentic track has the model use tools repeatedly while solving (read files, run simulation, inspect waveforms, edit and retry). Two properties change how it must be wired:

1. It launches containers. The official harness pulls a Docker container per problem as a sandbox. The platform’s evaluation job already runs inside a container — Docker-in-Docker on KubeRay or Slurm needs a privileged container or sysbox, which most internal clusters do not grant, and should not.
2. It needs a callable model endpoint, not a weights path.

The platform already has the second: model deployments provide a stable internal address and a revocable token. So the right shape for the agentic track is:

1. Deploy first

   Deploy the version under evaluation as a Model Deployment; take its endpoint and token.

2. Run the harness on a machine that has Docker

   Point the CVDP harness at that endpoint. That machine needs to launch containers — typically a dedicated evaluation host, not a training cluster node.

3. Report scores back

   The harness emits a report following the rtl_report.json contract above; it lands in the same dashboard.

In other words: agentic evaluation should not be a training job. It is an external process plus a score ingestion. Forcing it into the job scheduler buys an evaluation path that requires privileged containers — a much larger cost than the orchestration it saves.

Concretely

# 1. On the platform: open an external evaluation, take the run id and a scores-only token
sf bench external create --model alice/qa@v3 --harness cvdp --train-run <TRAIN_RUN_ID>

# 2. On the evaluation host (which has Docker): forward the platform endpoint into the agent
export OPENAI_BASE_URL=https://<console>/api/inference/<deployment-id>/v1
export OPENAI_API_KEY=<deployment token>
export CVDP_AGENT_ENV=OPENAI_BASE_URL,OPENAI_API_KEY
./run_benchmark.py -f <dataset> -l -g <your agent image> -p work/

# 3. Report the scores back (or POST to /api/ingest/benchmark with the token from the host)
sf bench external submit <RUN_ID> -f scores.json --harness cvdp

--train-run is not decoration: with it, the external scores are attributed to that training run by the evaluation gate and the model registry. Without it they are an isolated number.

The entry point accepts structured scores only; it does not parse the harness’s report text. CVDP emits report.txt and composite_report.txt — text, not JSON. Keep that conversion in your own script: when upstream changes a line of formatting you change the script, and the platform entry point stays untouched. The other way round, every upstream formatting tweak breaks ingestion.

The token carries one permission: report scores (scope=benchmark). It gets pasted into an evaluation host’s environment, so it must not also be able to write logs, change job state, or upload artifacts.