Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

When CQs Go Wrong: Challenges in CQ Verification with OE-Assist

Source

arXiv

arxiv.org/abs/2606.24619v1

Read the source

Source

Plain-Language Brief

This research paper discusses the challenges faced during Competency Questions (CQ) verification, a crucial step in ensuring that an ontology accurately represents its intended purpose. The authors found that CQs can be ambiguous and complex, leading to errors and inconsistencies during this process. They also developed an LLM assistant to aid in ontology evaluation but found the need for refining CQs before their publication.

What the Document Says

  • CQ-verification is a process that evaluates an ontology against natural language questions to determine if it accurately represents its intended purpose.
  • The process can be time-consuming and error-prone due to the need for careful interpretation of linguistic nuances and precise alignment with formal ontology constructs.
  • Ambiguities and complexity in CQs can further complicate the process, leading to inconsistent modelling decisions and verification outcomes.
  • The authors investigated factors that make a CQ challenging and potential solutions to enhance user performance in the CQ-verification process.
  • The study involved 19 participants performing CQ-verification tasks using an LLM assistant.
  • The results indicated a need for a tool to refine CQs before publication to avoid ambiguity or excessive complexity in later phases of ontology engineering.

Key Concepts Mentioned

  • Competency Questions (CQs)
  • Ontology
  • ontology verification
  • LLM assistant

Claims Made by the Document

  • CQ-verification can be error-prone and time-consuming.
  • Ambiguities and complexity in CQs can lead to inconsistent modelling decisions and verification outcomes.

Evidence Used

  • Participants performed CQ-verification tasks using an LLM assistant

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Competency Questions (CQs) are the central component of CQ-verification, an established process in which an ontology is evaluated against a set of natural language questions to determine whether the intended purpose of the ontology has been properly modelled. However, CQ-verification is often time-consuming and error-prone, as it requires careful interpretation of linguistic nuances and precise alignment with formal ontology constructs. Ambiguities and complexity in CQs can further complicate this process, leading to inconsistent modelling decisions and verification outcomes. In this paper, we investigate what makes a CQ challenging and possible solutions to enhance the users' performance in the CQ-verification process. We experimented with the data of 19 participants who performed CQ-verification on 20 tasks using an LLM assistant to support ontology evaluation. The results show the necessity of a tool to refine CQs before publishing them to avoid ambiguity or excessive complexity in later phases of the ontology engineering process.