Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Natural Language Access to Domain-Specific Metadata: A Reusable Framework for LLM Query Generation

Source

arXiv

arxiv.org/abs/2607.18029v1

Read the source

Source

Plain-Language Brief

Researchers have developed a system called Natural Language Knowledge Graph Query (NLKGQ) that allows them to ask ad-hoc questions about the contents of domain-specific archives using natural language, without requiring expertise in structured queries on metadata. The system uses Large Language Models (LLMs) and Web Ontology Language (OWL) ontologies for zero-shot query generation and can achieve 100% accuracy on competence and regression question sets with domain experts.

What the Document Says

  • Researchers present NLKGQ system for natural language access to metadata in domain-specific archives.
  • System requires well-designed OWL ontology to capture domain vocabulary and semantics.
  • An interface helps researchers pose natural language questions, which the system translates to SPARQL via an LLM and executes against a knowledge graph.
  • The development process involves capturing domain vocabulary and semantics in an OWL ontology, extracting metadata from archive sources, and importing it into a knowledge graph defined by the ontology.
  • Demonstrated on a large-scale neuroimaging research archive with multiple LLMs and ontology representations.
  • Achieved 100% accuracy on a competence and regression question set developed with domain experts.
  • An ablation study shows that readable entity names and semantic annotations are the dominant factors in accuracy.
  • Comparisons show OWL's structural features provide a substantial advantage over SQL DDL for LLM-driven query generation.
  • Demonstration domain requires local LLMs on modest institutional hardware to address privacy concerns for human subject data.

Key Concepts Mentioned

  • Domain-specific archives
  • Ad-hoc questions
  • Structured queries
  • Web Ontology Language (OWL) ontologies
  • Large Language Models (LLMs)
  • Natural Language Knowledge Graph Query (NLKGQ) system
  • Metadata
  • Competence and regression question sets
  • Ablation study
  • SQL databases

Claims Made by the Document

  • NLKGQ can generate accurate structured queries for domain-specific archives.
  • Readable entity names and semantic annotations are the dominant factors in accuracy.
  • OWL's structural features provide a substantial advantage over SQL DDL for LLM-driven query generation.

Evidence Used

  • Description of NLKGQ system and its development process.
  • Results from evaluations on the neuroimaging research archive.
  • Results from the ablation study across eight ontology representations.
  • Comparison between SPARQL and SQL databases.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Researchers need to answer ad-hoc questions about the contents of domain-specific archives but often lack the expertise to write structured queries on the metadata. We show that when domain vocabulary and semantics are captured in a well-designed Web Ontology Language (OWL) ontology, Large Language Models (LLMs) can generate accurate structured queries zero-shot, without fine-tuning, retrieval augmentation, or multi-agent orchestration. We present the Natural Language Knowledge Graph Query (NLKGQ) system, a framework and development process that enables natural language access to metadata in such archives. The framework includes a web interface that helps researchers pose natural language questions, which a domain-agnostic harness translates to SPARQL via an LLM and executes against a knowledge graph. The development process begins with capturing domain vocabulary and semantics in a formal OWL ontology. Domain-specific code then extracts metadata from archive sources and imports it into a knowledge graph defined by the ontology. Both are designed for reuse across domains. We demonstrate the system on metadata derived from a large-scale neuroimaging research archive, evaluating multiple LLMs and ontology representations. The best configurations achieve 100% accuracy on a competence and regression question set developed with domain experts. An ablation study across eight ontology representations reveals that readable entity names and semantic annotations are the dominant factors in accuracy, more significant than model choice or prompt engineering. We also compare SPARQL to an auto-generated SQL database as query backends, showing that OWL's structural features provide a substantial advantage over SQL DDL for LLM-driven query generation. Our demonstration domain also requires local LLMs on modest institutional hardware to address privacy concerns for human subject data.