Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Automated Semantic Fault Localization in SysML v2: A Human-in-the-Loop Framework Using Knowledge-Graph Augmented LLMs

Source

arXiv

arxiv.org/abs/2606.23395v1

Read the source

Source

Plain-Language Brief

This research presents a human-in-the-loop framework to identify and repair semantic errors in SysML v2, a systems modeling language. The framework combines a fine-tuned language model with a domain knowledge graph and two SLMs (Qwen2.5-Coder-1.5B and DeepSeek-Coder-6.7B) to localize faults and suggest repairs for engineers. This approach improved semantic fault repair from less than 3% to over 91%, with a significant reduction in token length.

What the Document Says

  • The paper discusses a framework for identifying and repairing semantic errors in SysML v2 using a human-in-the-loop approach.
  • The framework combines a fine-tuned Small Language Model (SLM) with a domain knowledge graph encoding physical compatibility rules between system elements.
  • The knowledge graph guides the generation of synthetic training data and augments the model at inference time to ground repair suggestions in valid engineering constraints.
  • Two SLMs, Qwen2.5-Coder-1.5B and DeepSeek-Coder-6.7B, are fine-tuned to output unified diff patches that localize faults and present candidate repairs for engineer review.
  • The evaluation of 1,184 test samples shows that fine-tuning improves semantic fault repair from less than 3% to more than 91%, with patch-based output reducing token length by over 60%.

Key Concepts Mentioned

  • SysML v2
  • Semantic errors
  • Human-in-the-loop framework
  • Small Language Model (SLM)
  • Domain knowledge graph
  • Physical compatibility rules
  • Engineering constraints
  • Qwen2.5-Coder-1.5B
  • DeepSeek-Coder-6.7B
  • Unified diff patches
  • Fault localization
  • Candidate repairs

Claims Made by the Document

  • The framework improves semantic fault repair significantly.
  • The patch-based output reduces token length significantly.

Evidence Used

  • Evaluation of 1,184 test samples shows that fine-tuning improves semantic fault repair from less than 3% to more than 91%, with patch-based output reducing token length by over 60%.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

SysML v2's textual syntax enables compiler-based validation of model structure and language conformance. However, semantic mistakes that preserve syntactic validity but violate domain rules cannot be detected through compilers. These errors can propagate through the design process and surface late as costly integration failures. This paper presents a human-in-the-loop framework for identifying and repairing such errors automatically. It combines a fine-tuned Small Language Model (SLM) with a domain knowledge graph encoding physical compatibility rules between system elements. The knowledge graph also guides the generation of synthetic training data by systematically introducing plausible domain violations, and augments the model at inference time to ground repair suggestions in valid engineering constraints. We demonstrate the framework using the vehicle systems domain, where the knowledge graph captures the relationships between the mechanical, electrical, fluid, and signal interfaces. Two SLMs, Qwen2.5-Coder-1.5B and DeepSeek-Coder-6.7B, are fine-tuned to output unified diff patches that localize faults and present candidate repairs for engineer review, preserving human judgment in the design process. Evaluation of 1,184 test samples shows that fine-tuning improves semantic fault repair from less than 3% to more than 91%, with patch-based output reducing token length by over 60%. The framework offers a practical path toward AI-assisted model verification that complements existing MBSE tools.