NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Jiarui Ma, Jianghan Wang, Yuheng Ma, Ziyi Zhuang, Xiaoguang Liu
• Venue: arXiv: eess.SY
• Published: 2026-08-12
• Collected: 2026-08-13
• URL: http://arxiv.org/abs/2608.12197v1
• PDF: https://arxiv.org/pdf/2608.12197v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers present NetlistBench, a new benchmark for evaluating the reliability of Large Language Models (LLMs) in SPICE netlist recognition and manipulation tasks. The study finds that while LLMs perform well on simple local edits, their accuracy drops significantly for complex operations such as device addition and equivalence judgment. The researchers suggest that netlist reliability is a critical issue for trustworthy LLM-based circuit design automation.

What the Document Says

• NetlistBench is presented as a structure-verified benchmark for SPICE netlist recognition and manipulation.
• It contains 2,342 cases across 24 task families covering various operations such as parameter and connectivity recognition, hierarchical operations, equivalence judgment, and long-horizon compound editing.
• The study involves six non-thinking LLMs.
• Accuracy varies with operation-level structural complexity. Simple local edits have high accuracy (96%-100%), while device addition drops to 41%-83% and equivalence judgment to 49%-90%.
• Enabling reasoning improves weaker models but does not eliminate structure-preservation failures.
• Performance degrades sharply as the edit horizon increases.
• The research suggests that netlist reliability is a bottleneck for trustworthy LLM-based circuit design automation.

Key Concepts Mentioned

• Large Language Models (LLMs)
• NetlistBench
• SPICE netlist recognition and manipulation
• structure-verified benchmark
• circuit design workflows
• task families
• parameter and connectivity recognition
• hierarchical operations
• equivalence judgment
• long-horizon compound editing
• deterministic structure-aware oracle
• operation-level structural complexity
• device addition
• reasoning
• edit horizon

Claims Made by the Document

• NetlistBench is a useful benchmark for evaluating LLMs in SPICE netlist recognition and manipulation.
• LLM performance varies with operation-level structural complexity.
• Simple local edits have high accuracy.
• Device addition drops in accuracy compared to simple local edits.
• Equivalence judgment accuracy also decreases.
• Enabling reasoning improves weaker models but does not eliminate structure-preservation failures.
• Performance degrades sharply as the edit horizon increases.

Evidence Used

• The study presents NetlistBench and its various task families.
• Results are presented showing the accuracy of LLMs for different operations.
• The text discusses the variation in accuracy with operation-level structural complexity.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large Language Models (LLMs) are increasingly used in circuit design workflows, yet their reliability on simulator-facing SPICE netlist recognition and manipulation remains poorly understood and is rarely separated from high-level design reasoning. Although netlists are textual, they encode structured circuit objects through topology and parameters. We present \textbf{NetlistBench}, a structure-verified benchmark for SPICE netlist recognition and manipulation. NetlistBench contains 2,342 cases across 24 task families, covering parameter and connectivity recognition and edits, hierarchical operations, equivalence judgment, and long-horizon compound editing. Model outputs are evaluated by a deterministic structure-aware oracle. Across six non-thinking LLMs, performance varies substantially with operation-level structural complexity. Simple local edits reach $96%$--$100%$ accuracy, while device addition drops to $41%$--$83%$ and equivalence judgment to $49%$--$90%$. Enabling reasoning substantially improves weaker models but does not eliminate structure-preservation failures, with performance still degrading sharply as the edit horizon increases. NetlistBench identifies netlist reliability as a distinct bottleneck for trustworthy LLM-based circuit design automation.