Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI

Source

arXiv

arxiv.org/abs/2608.06167v1

Read the source

Source

Plain-Language Brief

Researchers have developed a new method for extracting complex, structured information from unstructured text documents using generative AI. The method involves creating a schema that serves as an information model encoding domain knowledge, which provides a consistent framework for extraction of hierarchical, nested information with attributes of variable cardinality. The information is extracted in one call to the model and is then evaluated against a gold standard using a path-based semantic matching algorithm. The researchers tested their method on documents from the health technology assessment organization NICE and found that it was able to extract 12 out of 14 attributes with an F1 score over 90% and was significantly faster than a human domain expert.

What the Document Says

  • The paper presents a schema-based framework for complex information extraction from unstructured text documents using generative AI, followed by automated semantic evaluation against a gold standard.
  • The schema serves as an information model encoding domain knowledge and provides a consistent framework for hierarchical, nested information extraction with attributes of variable cardinality.
  • Information is extracted in one call to the model in zero-shot mode.
  • A path-based semantic matching algorithm is used for evaluating the results against the gold standard.
  • Generative AI is used for semantic comparison of the extracted and gold standard values of an attribute.
  • A rubric is introduced to classify the result of the comparison as exact, semantic, useful or non-match based on domain-specific considerations.
  • The method was tested on documents from NICE using the generative AI model Claude Opus 3, with 12 out of 14 attributes extracted with an F1 score over 90%.

Key Concepts Mentioned

  • information extraction
  • generative AI
  • schema
  • hierarchical information
  • nested information
  • attributes
  • variable cardinality
  • semantic matching algorithm
  • gold standard
  • rubric

Claims Made by the Document

  • The proposed method is effective for complex information extraction from unstructured text documents.
  • The schema provides a consistent framework for hierarchical, nested information extraction with attributes of variable cardinality.

Evidence Used

  • The F1 score over 90% obtained in testing the method on NICE documents

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

We present a schema-based framework for extracting complex, structured information from unstructured text documents using generative AI, followed by automated semantic evaluation of the extracted information against a gold standard. The schema, serving as an information model encoding domain knowledge, provides a unified, systematic, and consistent framework for extraction of hierarchical, nested information, with attributes of variable cardinality, and subsequent evaluation of the results. Information extraction from a document is performed in a single call to the model, in zero-shot mode. In the evaluation step, we introduce a path-based semantic matching algorithm to align the nested, variable-cardinality attributes in the extracted results with those in the gold standard. We use generative AI for semantic comparison of the extracted and gold standard values of an attribute, and introduce a rubric to classify the result of the comparison, according to domain-specific considerations, as an exact, semantic, useful, or non-match. We were able to extract 12 out of 14 attributes with an F1 score of $>$90% from documents published by the health technology assessment organisation NICE, using the generative AI model Claude Opus 3. The time needed to extract the attributes from a document was $\sim$30 times lower than the time taken by a human domain expert. We further demonstrate generalisability of this framework across different generative AI models and transferability across different HTA organisations and languages.