Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

Source

arXiv

arxiv.org/abs/2606.24834v1

Read the source

Source

Plain-Language Brief

Researchers evaluate the accuracy and quality of conversations between developers and a Language Model (LLM)-based agent in the context of Health Insurance Portability and Accountability Act (HIPAA) regulatory compliance. They find that while developers generally agree with the LLM assessments, the system's accuracy against expert ground truth is low. Longer responses and more informational turns negatively affect user satisfaction, but proactive interactions positively impact it.

What the Document Says

  • The paper investigates the accuracy and quality of multi-turn conversations between developers and an LLM-based agent in HIPAA regulatory compliance.
  • 49 programmers were hired to interact with GitHub Copilot to assess 148 HIPAA-derived NFRs against the iTrust codebase across three dimensions: requirement satisfaction level, reasoning, and code localization.
  • Developers tend to agree with LLM assessments, but accuracy against expert ground truth is low.
  • User satisfaction was modeled and found to be negatively affected by longer system responses and more informational turns, but positively affected by proactive interactions.

Key Concepts Mentioned

  • LLM-based dialogue assistants
  • Non-Functional Requirements (NFRs)
  • Health Insurance Portability and Accountability Act (HIPAA)
  • GitHub Copilot
  • iTrust codebase
  • requirement satisfaction level
  • reasoning
  • code localization
  • user satisfaction

Claims Made by the Document

  • LLM assessments are generally agreed upon by developers but have low accuracy against expert ground truth.
  • Longer system responses and more informational turns negatively affect user satisfaction, while proactive interactions positively affect it.

Evidence Used

  • The researchers conducted an experiment with 49 programmers interacting with GitHub Copilot to assess HIPAA-derived NFRs.
  • The findings from the study were presented regarding developer agreement and user satisfaction.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

LLM-based dialogue assistants have become mainstream tools for software developers, yet current evaluation benchmarks focus exclusively on functional correctness. This leaves a critical gap in assessing the quality and accuracy of these conversations when handling Non-Functional Requirements (NFRs), which are inherently vague, context-dependent, and involve many parts of a program. Evaluating how well these systems support collaborative reasoning about NFRs requires methods that go beyond single-turn accuracy to capture both the correctness of the system's outputs and the quality of the multi-turn interaction. In this paper, we investigate the accuracy and quality of multi-turn conversations between developers and an LLM-based agent in the domain of Health Insurance Portability and Accountability Act (HIPAA) regulatory compliance. We hired 49 programmers to interact with GitHub Copilot to assess 148 HIPAA-derived NFRs against the iTrust codebase, a system designed to comply with HIPAA regulations, across three dimensions: requirement satisfaction level, reasoning, and code localization. We find that developers tend to agree with LLM assessments, but accuracy against expert ground truth is low. We model user satisfaction and find that longer system responses and more information-providing turns negatively affect user satisfaction, whereas proactive interactions positively affect it. Our findings provide insights for designing LLM-based dialogue systems that support NFR assessment.