Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models

Source

arXiv

arxiv.org/abs/2606.27047v1

Read the source

Source

Plain-Language Brief

Researchers have created a new benchmark called NuclearQAv2 to evaluate the performance of large language models (LLMs) in the domain of nuclear engineering. This benchmark consists of about 1,240 question-answer pairs divided into three categories: boolean, numeric, and verbal. The authors used a hybrid pipeline that combines expert-authored questions, existing datasets, and LLM-assisted generation to construct this benchmark. They tested various LLMs using NuclearQAv2 and found significant performance differences across task types, with models performing well on factual questions but struggling with quantitative reasoning and conceptual understanding.

What the Document Says

  • NuclearQAv2 is a benchmark for evaluating large language models (LLMs) in the nuclear engineering domain.
  • The benchmark consists of approximately 1,240 question-answer pairs categorized as boolean, numeric, and verbal.
  • NuclearQAv2 was constructed using a hybrid pipeline that includes expert-authored questions, existing datasets, and LLM-assisted generation from domain-specific technical corpora.
  • The proposed framework uses structured prompting for automated question generation and response evaluation to enable scalable benchmark construction and evaluation.
  • The study tested various LLMs using NuclearQAv2 and observed substantial performance differences across task types.
  • Models generally performed well on factual questions but struggled with quantitative reasoning and conceptual understanding.

Key Concepts Mentioned

  • NuclearQAv2
  • Large Language Models (LLMs)
  • Nuclear Engineering
  • Question-Answer Pairs
  • Boolean
  • Numeric
  • Verbal
  • Hybrid pipeline
  • Expert-authored questions
  • Domain-specific technical corpora
  • Structured prompting

Claims Made by the Document

  • NuclearQAv2 is a useful benchmark for evaluating LLMs on nuclear engineering knowledge.
  • Substantial performance differences exist across task types when testing various LLMs with NuclearQAv2.
  • Models generally perform well on factual questions but struggle with quantitative reasoning and conceptual understanding.

Evidence Used

  • The introduction of NuclearQAv2
  • Description of the benchmark structure
  • Explanation of the hybrid pipeline used to construct the benchmark
  • Results from testing various LLMs using NuclearQAv2

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language models (LLMs) have demonstrated strong performance across a wide range of tasks, but ensuring their reliability in highly technical domains remains a significant challenge. In nuclear engineering, problem solving often requires not only factual knowledge but also quantitative reasoning and conceptual understanding. To address the need for systematic evaluation in this domain, we introduce NuclearQAv2, a benchmark for assessing LLMs on nuclear engineering knowledge. The benchmark comprises approximately 1,240 question-answer pairs spanning three categories: boolean, numeric, and verbal. NuclearQAv2 is constructed using a hybrid pipeline that combines expert-authored questions, existing datasets, and LLM-assisted generation from domain-specific technical corpora. By leveraging structured prompting for both automated question generation and response evaluation, the proposed framework enables scalable benchmark construction and evaluation. We evaluate a diverse set of LLMs using NuclearQAv2 and observe substantial performance differences across task types. While the models generally perform well on factual questions, quantitative reasoning and conceptual understanding remain considerably more challenging. These results highlight the importance of multi-faceted evaluation frameworks and establish NuclearQAv2 as a scalable benchmark for assessing LLM capabilities in technical domains.