Source note
Structural Verification for Reliable EDA Code Generation without Tool-in-the-Loop Debugging
Summary
This paper argues that many LLM failures in OpenROAD script generation come from broken object and API dependencies, not syntax. It introduces pre-execution structural verification so the system can catch and repair those failures before running the EDA tool, which raises pass rate and cuts tool calls.
Problem
- The paper targets natural-language-to-EDA code generation for OpenROAD, where generated scripts often fail because they follow invalid design-object paths, skip required intermediate objects, or call APIs on the wrong types.
- This matters because tool-in-the-loop debugging fixes errors by repeated execution and repair, which adds latency, increases tool usage, and scales poorly on multi-step workflows where early errors propagate.
- In EDA, execution depends on stateful design hierarchies and action preconditions, so syntax checks alone miss many real failures.
Approach
- The method converts each task into a structural dependency graph whose nodes are typed design objects, conditions, and actions, and whose edges encode valid acquisition and dependency relations. The graph acts as an execution contract.
- It builds and validates that graph from the prompt against an OpenROAD API schema, filtering hallucinated nodes, invalid type transitions, and missing prerequisites before code synthesis.
- Retrieval and code generation are conditioned on the graph, so the model fetches API examples for specific object transitions and generates code that follows the required dependency path.
- A four-layer verifier checks syntax, causal flow, API alignment, and task-level semantics, then applies diagnosis-driven localized repair instead of full reruns in the external tool.
- For multi-step tasks, it runs the same process per subtask and adds trajectory-level reflection to diagnose cross-step failures; an uncertainty filter scores verifier-passed programs to reduce false accepts before execution.
Results
- On single-step OpenROAD tasks, the full system reaches 82.5% pass rate, above 73.0% for LLM+RAG and 76.0% for LLM + tool-in-loop with GPT-4.1-mini.
- On the same single-step setting, the method uses 1.00 tool calls per task versus 1.77 for tool-in-loop and 3.54 for OpenROAD-Agent, with 120 total calls versus 248 and 496. The abstract states this is more than 2× fewer tool calls than tool-in-loop debugging.
- Single-step latency in Table 1 is 34.8 s for the full pipeline, compared with 53.0 s for tool-in-loop and 70.0 s for OpenROAD-Agent; Table 2 reports 37.6 s for GPT-4.1-mini tool-in-loop and 34.8 s for the proposed method.
- On multi-step tasks, the abstract reports pass rate improving from 30.0% to 70.0%, and then to 84.0% when adding trajectory-level reflection.
- Uncertainty-aware filtering cuts verifier false positives from 20.0% to 6.7% and raises precision from 80.0% to 93.3%.
- Across generators, the full method reports 83.0% pass with GPT-4o and 82.5% with GPT-4.1-mini; a verifier added to OpenROAD-Agent raises it from 16.4% to 31.8% while keeping 1.00 tool call per task.