Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Natural Language Access to Domain-Specific Metadata: A Reusable Framework for LLM Query Generation

Source

arXiv

arxiv.org/abs/2607.18029v1

Read the source

Source

Plain-Language Brief

Researchers have developed a system called Natural Language Knowledge Graph Query (NLKGQ) that allows researchers to ask ad-hoc questions about the contents of domain-specific archives using natural language. The system uses Large Language Models (LLMs) and a well-designed ontology to generate structured queries without requiring fine-tuning or additional tools. The system was demonstrated on metadata derived from a large-scale neuroimaging research archive and achieved 100% accuracy.

What the Document Says

  • Researchers developed a system called NLKGQ for natural language access to domain-specific metadata archives.
  • The system uses LLMs, SPARQL, and an ontology defined by the domain.
  • The development process involves capturing domain vocabulary and semantics in an OWL ontology, extracting metadata from archive sources, and importing it into a knowledge graph.
  • The system includes a web interface for posing natural language questions which are translated to SPARQL queries by a domain-agnostic harness.
  • The system was evaluated on a competence and regression question set developed with domain experts and achieved 100% accuracy.
  • An ablation study revealed that readable entity names and semantic annotations are the dominant factors in accuracy.
  • SPARQL was compared to an auto-generated SQL database as query backends, showing a substantial advantage for LLM-driven query generation using OWL.

Key Concepts Mentioned

  • Researchers
  • domain-specific archives
  • Large Language Models (LLMs)
  • Web Ontology Language (OWL) ontology
  • Natural Language Knowledge Graph Query (NLKGQ)
  • SPARQL
  • metadata

Claims Made by the Document

  • The NLKGQ system can generate accurate structured queries for domain-specific metadata archives.
  • Readable entity names and semantic annotations are the dominant factors in accuracy for the NLKGQ system.

Evidence Used

  • The evaluation results of the NLKGQ system on a competence and regression question set.
  • The findings from the ablation study conducted on eight ontology representations.

Methods / Approach

  • Using LLMs, SPARQL, and an OWL ontology to generate structured queries.
  • Capturing domain vocabulary and semantics in an OWL ontology, extracting metadata from archive sources, and importing it into a knowledge graph.

Limitations Stated by the Authors

  • The system requires local LLMs on modest institutional hardware due to privacy concerns for human subject data.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The study did not discuss potential limitations related to the generalizability of the results across different domains or types of metadata.
  • The study did not address potential issues with the quality or completeness of the metadata in the demonstration domain.

Open Questions

  • How does the NLKGQ system perform on other types of domain-specific metadata archives?
  • What are the potential limitations and challenges for scaling up the NLKGQ system to handle large-scale metadata archives?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The study focuses specifically on a large-scale neuroimaging research archive. It would be interesting to see how the system performs on other types of domain-specific archives.
  • The study did not discuss potential challenges or limitations for scaling up the NLKGQ system to handle larger metadata archives.

Raw Abstract / Excerpt

Researchers need to answer ad-hoc questions about the contents of domain-specific archives but often lack the expertise to write structured queries on the metadata. We show that when domain vocabulary and semantics are captured in a well-designed Web Ontology Language (OWL) ontology, Large Language Models (LLMs) can generate accurate structured queries zero-shot, without fine-tuning, retrieval augmentation, or multi-agent orchestration. We present the Natural Language Knowledge Graph Query (NLKGQ) system, a framework and development process that enables natural language access to metadata in such archives. The framework includes a web interface that helps researchers pose natural language questions, which a domain-agnostic harness translates to SPARQL via an LLM and executes against a knowledge graph. The development process begins with capturing domain vocabulary and semantics in a formal OWL ontology. Domain-specific code then extracts metadata from archive sources and imports it into a knowledge graph defined by the ontology. Both are designed for reuse across domains. We demonstrate the system on metadata derived from a large-scale neuroimaging research archive, evaluating multiple LLMs and ontology representations. The best configurations achieve 100% accuracy on a competence and regression question set developed with domain experts. An ablation study across eight ontology representations reveals that readable entity names and semantic annotations are the dominant factors in accuracy, more significant than model choice or prompt engineering. We also compare SPARQL to an auto-generated SQL database as query backends, showing that OWL's structural features provide a substantial advantage over SQL DDL for LLM-driven query generation. Our demonstration domain also requires local LLMs on modest institutional hardware to address privacy concerns for human subject data.