Source note
Automated Semantic Fault Localization in SysML v2: A Human-in-the-Loop Framework Using Knowledge-Graph Augmented LLMs
Summary
The paper proposes a KG-guided small language model method for finding and repairing SysML v2 syntax and semantic faults. Its main claim is that fine-tuning turns semantic repair accuracy from under 3% to over 91% while producing shorter diff patches for engineer review.
Problem
- SysML v2 compilers catch grammar errors, but they can miss models that compile while violating engineering rules, such as connecting incompatible mechanical, electrical, fluid, or signal interfaces.
- These semantic faults matter because they can move through MBSE workflows and appear later as integration failures.
- SysML v2 training data is scarce, which limits general-purpose LLMs on valid syntax and domain-correct repair.
Approach
- The method builds a vehicle-domain knowledge graph with physical interface compatibility rules and a physical-quantity graph that maps quantity kinds to valid units.
- It starts with 256 valid SysML v2 files and creates 8,301 examples: 5,497 syntax faults, 1,402 semantic faults, and 1,402 correct samples.
- Semantic faults are generated by mutating otherwise valid models so they still compile while breaking KG rules, such as port-domain mismatches or invalid unit assignments.
- Qwen2.5 Coder 1.5B Instruct and DeepSeek Coder 6.7B Instruct are fine-tuned with LoRA to classify code as correct or incorrect and output either repaired code or unified diff patches.
- At inference time, compiler messages and KG-derived rules are added to the prompt, and the generated patch stays under human review.
Results
- On the reported 1,184-sample evaluation set, Qwen2.5 Coder 1.5B semantic repair rose from 0.62% baseline accuracy to 95.7% with fine-tuned full-code output and 91.9% with fine-tuned patch output.
- DeepSeek Coder 6.7B semantic repair rose from 2.47% baseline accuracy to 91.9% with fine-tuned full-code output and 91.4% with fine-tuned patch output.
- Qwen overall accuracy improved from 21.8% baseline to 92.8% with fine-tuned full-code output; its patch output reached 73.4% overall accuracy.
- DeepSeek overall accuracy improved from 30.0% baseline to 90.7% with fine-tuned full-code output; its patch output reached 70.2% overall accuracy.
- Patch outputs were shorter: Qwen used 84 output tokens with patches versus 173 with full-code repair, and DeepSeek used 111 versus 271.
- Correct-code classification reached 98.1% for Qwen with both fine-tuned full-code and patch outputs, and 94.7% to 97.6% for DeepSeek depending on output format.