Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI

Source

arXiv

arxiv.org/abs/2608.06167v1

Read the source

Source

Plain-Language Brief

Researchers have developed a framework using generative AI to extract complex, structured information from unstructured text documents. This framework also includes an automated semantic evaluation step to compare the extracted information with a gold standard. The schema serves as an information model that encodes domain knowledge and provides a consistent method for extraction of hierarchical, nested information. The researchers tested their method on documents published by the health technology assessment organisation NICE and found that it significantly reduced the time needed for extraction compared to human experts.

What the Document Says

  • The paper presents a schema-based framework using generative AI for extracting complex, structured information from unstructured text documents.
  • The framework includes an automated semantic evaluation step for comparing the extracted information with a gold standard.
  • The schema serves as an information model encoding domain knowledge and provides a consistent method for extraction of hierarchical, nested information.
  • The researchers tested their method on documents published by the health technology assessment organisation NICE.
  • They found that the time needed to extract attributes from a document was approximately 30 times lower than that of human experts.

Key Concepts Mentioned

  • generative AI
  • information extraction
  • semantic evaluation
  • schema
  • domain knowledge
  • health technology assessment organization
  • NICE

Claims Made by the Document

  • The framework can extract complex, structured information from unstructured text documents.
  • The framework includes an automated semantic evaluation step.
  • The schema provides a consistent method for extraction of hierarchical, nested information.
  • The researchers were able to reduce the time needed for attribute extraction by approximately 30 times compared to human experts.

Evidence Used

  • Results from testing on documents published by NICE demonstrate reduced time for attribute extraction.

Methods / Approach

  • Schema-based framework using generative AI
  • Automated semantic evaluation
  • Path-based semantic matching algorithm
  • Generative AI for semantic comparison
  • Rubric for classifying the result of the comparison

Limitations Stated by the Authors

  • UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The paper does not discuss potential limitations or challenges in implementing the framework, such as possible errors or biases introduced by the generative AI model.

Open Questions

  • What are the potential errors or biases that might be introduced by the generative AI model?
  • How scalable is this framework to large datasets?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper focuses on demonstrating the effectiveness of the framework in reducing the time needed for attribute extraction compared to human experts. However, it does not discuss potential limitations or challenges in implementing the framework.

Raw Abstract / Excerpt

We present a schema-based framework for extracting complex, structured information from unstructured text documents using generative AI, followed by automated semantic evaluation of the extracted information against a gold standard. The schema, serving as an information model encoding domain knowledge, provides a unified, systematic, and consistent framework for extraction of hierarchical, nested information, with attributes of variable cardinality, and subsequent evaluation of the results. Information extraction from a document is performed in a single call to the model, in zero-shot mode. In the evaluation step, we introduce a path-based semantic matching algorithm to align the nested, variable-cardinality attributes in the extracted results with those in the gold standard. We use generative AI for semantic comparison of the extracted and gold standard values of an attribute, and introduce a rubric to classify the result of the comparison, according to domain-specific considerations, as an exact, semantic, useful, or non-match. We were able to extract 12 out of 14 attributes with an F1 score of $>$90% from documents published by the health technology assessment organisation NICE, using the generative AI model Claude Opus 3. The time needed to extract the attributes from a document was $\sim$30 times lower than the time taken by a human domain expert. We further demonstrate generalisability of this framework across different generative AI models and transferability across different HTA organisations and languages.