> ## Documentation Index > Fetch the complete documentation index at: https://starforge.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Score hardware and RTL work > VerilogEval v2, RTLLM v2, RTL-Repo, CVDP — compile the Verilog and run the testbench ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench run my-bench -m run:run-4f2a91 --suites verilogeval-v2 ``` RTL evaluation is a different kind of thing from a text benchmark. GSM8K judges whether the answer is right; VerilogEval judges whether **the generated circuit compiles and passes its testbench**. That needs a simulator and pass\@k over repeated sampling — a different execution chain entirely. The platform has a third runner for it, `rtl`, alongside lm-eval and evalscope. Scores land in the same dashboard, so an RTL score and a GSM8K score sit on one row for the same model. ## Available benchmarks | Benchmark | What it measures | Primary metric | | --------------------------- | ---------------------------------------------------------------------------------------------- | -------------- | | `verilogeval-v2` | Natural-language spec → Verilog module (spec-to-RTL) | `pass@1` | | `verilogeval-v2-completion` | Complete an implementation from the module header | `pass@1` | | `rtllm-v2` | Design-level tasks (ALU, FIFO, pipeline units), synthesizable | `func_pass` | | `rtl-repo` | Completion inside a real repository, testing long context and project conventions | `exact_match` | | `cvdp` | NVIDIA's comprehensive design problems: write, debug, testbench, spec Q\&A (non-agentic track) | `pass@1` | ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} sf bench run my-bench -m run: --suites verilogeval-v2,rtllm-v2 ``` One evaluation uses one runner, so RTL benchmarks and GSM8K-style ones go in separate jobs. **`rtl-repo`'s metric is not comparable to the others.** It runs no simulation (there is no testbench) and reports text similarity; ranking its `exact_match` next to VerilogEval's `pass@1` is wrong — they do not measure the same thing. ## Admins: what the image needs Two things. **The simulators** are already in `deploy/docker/Dockerfile.evalkit` (`iverilog` + `verilator`). **The harness is yours to install.** VerilogEval, RTLLM and CVDP each ship their own problem sets and testbenches in their own repositories. The platform deliberately does **not** copy those problems — when upstream revises a problem, a copy silently becomes a different benchmark while still carrying the original name. So the platform only defines the contract: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} $FORGE_RTL_HARNESS --benchmark --model --output-dir \ [--samples N] [--limit N] ``` The harness writes `/rtl_report.json`: ```json theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} {"results": {"verilogeval-v2-spec": {"pass@1": 0.42, "syntax_pass": 0.90}}} ``` The Dockerfile carries a commented installation example. **Pin the upstream commit** — evaluation results have to stay comparable across months, and one revision of the problem set invalidates every earlier score. Without a harness, the job fails with an explicit error ("no rtl\_report.json produced; check that the image has a harness") rather than producing an empty report. ## CVDP's agentic track: possible, but not through this path CVDP has two tracks. The **non-agentic** track is the one above: generate once, judge functionality, via the `rtl` runner. The **agentic** track has the model use tools repeatedly while solving (read files, run simulation, inspect waveforms, edit and retry). Two properties change how it must be wired: 1. **It launches containers.** The official harness pulls a Docker container per problem as a sandbox. The platform's evaluation job **already runs inside a container** — Docker-in-Docker on KubeRay or Slurm needs a privileged container or sysbox, which most internal clusters do not grant, and should not. 2. **It needs a callable model endpoint**, not a weights path. The platform already has the second: [model deployments](/en/guides/model-registry) provide a stable internal address and a revocable token. So the right shape for the agentic track is: Deploy the version under evaluation as a Model Deployment; take its endpoint and token. Point the CVDP harness at that endpoint. That machine needs to launch containers — typically a dedicated evaluation host, not a training cluster node. The harness emits a report following the `rtl_report.json` contract above; it lands in the same dashboard. In other words: **agentic evaluation should not be a training job.** It is an external process plus a score ingestion. Forcing it into the job scheduler buys an evaluation path that requires privileged containers — a much larger cost than the orchestration it saves. ### Concretely ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} # 1. On the platform: open an external evaluation, take the run id and a scores-only token sf bench external create --model alice/qa@v3 --harness cvdp --train-run # 2. On the evaluation host (which has Docker): forward the platform endpoint into the agent export OPENAI_BASE_URL=https:///api/inference//v1 export OPENAI_API_KEY= export CVDP_AGENT_ENV=OPENAI_BASE_URL,OPENAI_API_KEY ./run_benchmark.py -f -l -g -p work/ # 3. Report the scores back (or POST to /api/ingest/benchmark with the token from the host) sf bench external submit -f scores.json --harness cvdp ``` `--train-run` is not decoration: with it, the external scores are attributed to that training run by the evaluation gate and the model registry. Without it they are an isolated number. **The entry point accepts structured scores only; it does not parse the harness's report text.** CVDP emits `report.txt` and `composite_report.txt` — text, not JSON. Keep that conversion in your own script: when upstream changes a line of formatting you change the script, and the platform entry point stays untouched. The other way round, every upstream formatting tweak breaks ingestion. The token carries one permission: report scores (scope=benchmark). It gets pasted into an evaluation host's environment, so it must not also be able to write logs, change job state, or upload artifacts.