---
source: arxiv
url: https://arxiv.org/abs/2606.23395v1
published_at: '2026-06-22T14:20:11'
authors:
- Haitham Al-Shami
- Rohail Malik
- Riku Ala-Laurinaho
- "Jari Veps\xE4l\xE4inen"
- Raine Viitala
topics:
- sysml-v2
- semantic-fault-localization
- knowledge-graphs
- code-repair
- mbse
- human-ai-interaction
relevance_score: 0.78
run_id: materialize-outputs
language_code: en
---

# Automated Semantic Fault Localization in SysML v2: A Human-in-the-Loop Framework Using Knowledge-Graph Augmented LLMs

## Summary
The paper proposes a KG-guided small language model method for finding and repairing SysML v2 syntax and semantic faults. Its main claim is that fine-tuning turns semantic repair accuracy from under 3% to over 91% while producing shorter diff patches for engineer review.

## Problem
- SysML v2 compilers catch grammar errors, but they can miss models that compile while violating engineering rules, such as connecting incompatible mechanical, electrical, fluid, or signal interfaces.
- These semantic faults matter because they can move through MBSE workflows and appear later as integration failures.
- SysML v2 training data is scarce, which limits general-purpose LLMs on valid syntax and domain-correct repair.

## Approach
- The method builds a vehicle-domain knowledge graph with physical interface compatibility rules and a physical-quantity graph that maps quantity kinds to valid units.
- It starts with 256 valid SysML v2 files and creates 8,301 examples: 5,497 syntax faults, 1,402 semantic faults, and 1,402 correct samples.
- Semantic faults are generated by mutating otherwise valid models so they still compile while breaking KG rules, such as port-domain mismatches or invalid unit assignments.
- Qwen2.5 Coder 1.5B Instruct and DeepSeek Coder 6.7B Instruct are fine-tuned with LoRA to classify code as correct or incorrect and output either repaired code or unified diff patches.
- At inference time, compiler messages and KG-derived rules are added to the prompt, and the generated patch stays under human review.

## Results
- On the reported 1,184-sample evaluation set, Qwen2.5 Coder 1.5B semantic repair rose from 0.62% baseline accuracy to 95.7% with fine-tuned full-code output and 91.9% with fine-tuned patch output.
- DeepSeek Coder 6.7B semantic repair rose from 2.47% baseline accuracy to 91.9% with fine-tuned full-code output and 91.4% with fine-tuned patch output.
- Qwen overall accuracy improved from 21.8% baseline to 92.8% with fine-tuned full-code output; its patch output reached 73.4% overall accuracy.
- DeepSeek overall accuracy improved from 30.0% baseline to 90.7% with fine-tuned full-code output; its patch output reached 70.2% overall accuracy.
- Patch outputs were shorter: Qwen used 84 output tokens with patches versus 173 with full-code repair, and DeepSeek used 111 versus 271.
- Correct-code classification reached 98.1% for Qwen with both fine-tuned full-code and patch outputs, and 94.7% to 97.6% for DeepSeek depending on output format.

## Link
- [https://arxiv.org/abs/2606.23395v1](https://arxiv.org/abs/2606.23395v1)