Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

Source

arXiv

arxiv.org/abs/2606.24834v1

Read the source

Source

Plain-Language Brief

Researchers examined the accuracy of Large Language Model (LLM)-based dialogue assistants in handling Non-Functional Requirements (NFRs) related to Health Insurance Portability and Accountability Act (HIPAA) regulatory compliance. They found that while developers often agree with the LLM assessments, the overall accuracy is low compared to expert ground truth. Longer system responses and fewer proactive interactions tend to negatively impact user satisfaction.

What the Document Says

  • The research focuses on the evaluation of LLM-based dialogue systems for handling NFRs in software development.
  • The study was conducted in the domain of HIPAA regulatory compliance with 49 programmers and 148 HIPAA-derived NFRs against the iTrust codebase.
  • Assessments were made across three dimensions: requirement satisfaction level, reasoning, and code localization.
  • Developers tend to agree with LLM assessments but accuracy against expert ground truth is low.
  • Longer system responses and more information-providing turns negatively affect user satisfaction.
  • Proactive interactions positively impact user satisfaction.

Key Concepts Mentioned

  • LLM-based dialogue assistants
  • Non-Functional Requirements (NFRs)
  • Health Insurance Portability and Accountability Act (HIPAA)
  • iTrust codebase
  • user satisfaction

Claims Made by the Document

  • LLM-based dialogue systems have low accuracy in handling NFRs.
  • Longer system responses negatively impact user satisfaction.
  • Proactive interactions positively affect user satisfaction.

Evidence Used

  • The findings provide insights for designing LLM-based dialogue systems that support NFR assessment.

Methods / Approach

  • Hired 49 programmers to interact with GitHub Copilot and assess HIPAA-derived NFRs against the iTrust codebase

Limitations Stated by the Authors

Not stated in source

Limitations Noted by the Agent

Agent interpretation — not from the source.

The study only focused on HIPAA regulatory compliance, and results might vary for other domains.

Open Questions

  • What is the accuracy of LLM-based dialogue systems in handling NFRs outside of the HIPAA domain?
  • How can the accuracy of these systems be improved to match expert ground truth?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The study does not provide information on how the proactive interactions were defined or measured.

Raw Abstract / Excerpt

LLM-based dialogue assistants have become mainstream tools for software developers, yet current evaluation benchmarks focus exclusively on functional correctness. This leaves a critical gap in assessing the quality and accuracy of these conversations when handling Non-Functional Requirements (NFRs), which are inherently vague, context-dependent, and involve many parts of a program. Evaluating how well these systems support collaborative reasoning about NFRs requires methods that go beyond single-turn accuracy to capture both the correctness of the system's outputs and the quality of the multi-turn interaction. In this paper, we investigate the accuracy and quality of multi-turn conversations between developers and an LLM-based agent in the domain of Health Insurance Portability and Accountability Act (HIPAA) regulatory compliance. We hired 49 programmers to interact with GitHub Copilot to assess 148 HIPAA-derived NFRs against the iTrust codebase, a system designed to comply with HIPAA regulations, across three dimensions: requirement satisfaction level, reasoning, and code localization. We find that developers tend to agree with LLM assessments, but accuracy against expert ground truth is low. We model user satisfaction and find that longer system responses and more information-providing turns negatively affect user satisfaction, whereas proactive interactions positively affect it. Our findings provide insights for designing LLM-based dialogue systems that support NFR assessment.