Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models

Source

arXiv

arxiv.org/abs/2606.27047v1

Read the source

Source

Plain-Language Brief

Researchers have created a new benchmark called NuclearQAv2 to evaluate the performance of large language models (LLMs) in the domain of nuclear engineering. The benchmark consists of approximately 1,240 question-answer pairs divided into three categories: boolean, numeric, and verbal. It was built using a hybrid approach that involves expert-authored questions, existing datasets, and LLM-assisted generation from technical corpora. NuclearQAv2 uses structured prompting for both automated question generation and response evaluation, allowing for scalable benchmark construction and evaluation. The study finds significant performance differences across task types, with models performing well on factual questions but struggling with quantitative reasoning and conceptual understanding.

What the Document Says

  • NuclearQAv2 is a benchmark for evaluating LLMs on nuclear engineering knowledge.
  • The benchmark includes about 1,240 question-answer pairs categorized as boolean, numeric, and verbal.
  • NuclearQAv2 was constructed using a hybrid pipeline that combines expert-authored questions, existing datasets, and LLM-assisted generation from domain-specific technical corpora.
  • Structured prompting is used for both automated question generation and response evaluation in NuclearQAv2, allowing for scalable benchmark construction and evaluation.
  • The study finds substantial performance differences across task types in various LLMs.
  • LLMs generally perform well on factual questions but struggle with quantitative reasoning and conceptual understanding.

Key Concepts Mentioned

  • large language models
  • nuclear engineering
  • NuclearQAv2
  • question-answer pairs
  • hybrid pipeline
  • structured prompting

Claims Made by the Document

  • NuclearQAv2 is a useful benchmark for evaluating LLMs on nuclear engineering knowledge.
  • Performance differences exist across task types in various LLMs when using NuclearQAv2.

Evidence Used

  • The text describes the creation and purpose of NuclearQAv2.
  • The text provides details about the question-answer pairs and construction methodology of NuclearQAv2.
  • The text discusses the performance differences observed across task types in various LLMs when using NuclearQAv2.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language models (LLMs) have demonstrated strong performance across a wide range of tasks, but ensuring their reliability in highly technical domains remains a significant challenge. In nuclear engineering, problem solving often requires not only factual knowledge but also quantitative reasoning and conceptual understanding. To address the need for systematic evaluation in this domain, we introduce NuclearQAv2, a benchmark for assessing LLMs on nuclear engineering knowledge. The benchmark comprises approximately 1,240 question-answer pairs spanning three categories: boolean, numeric, and verbal. NuclearQAv2 is constructed using a hybrid pipeline that combines expert-authored questions, existing datasets, and LLM-assisted generation from domain-specific technical corpora. By leveraging structured prompting for both automated question generation and response evaluation, the proposed framework enables scalable benchmark construction and evaluation. We evaluate a diverse set of LLMs using NuclearQAv2 and observe substantial performance differences across task types. While the models generally perform well on factual questions, quantitative reasoning and conceptual understanding remain considerably more challenging. These results highlight the importance of multi-faceted evaluation frameworks and establish NuclearQAv2 as a scalable benchmark for assessing LLM capabilities in technical domains.