Source note
LLM4MTLs: Automated Generation and Empirical Evaluation of Model Transformation Languages
LLM Code GenerationModel TransformationDomain Specific LanguagesPrompt EngineeringSoftware Engineering Benchmark
Summary
LLM4MTLs automates prompt construction and evaluation for LLM-generated model transformation language code. It targets low-resource DSLs where LLMs often produce code that parses poorly or fails transformation tests.
Problem
- Model transformation languages such as ATL, ETL, QVTo, and Reactions require language syntax knowledge plus source and target metamodel knowledge.
- LLMs have limited training data for these DSLs, so direct prompting often returns code with syntax errors or wrong transformation behavior.
- Prior work lacks a reproducible benchmark with executable reference transformations and tests across multiple MTLs.
Approach
- The workflow builds task prompts from existing reference transformations, then uses those prompts to ask LLMs to regenerate MTL code.
- It tests prompt variants that combine few-shot examples, grammar text, and language-specific helper methods.
- It evaluates generated code with syntactic similarity, syntactic correctness, and semantic correctness against reference scripts and test suites.
- The authors implement the workflow in n8n, containerize the components, and release a replication package.
Results
- The evaluation suite contains 47 transformation examples across 4 MTLs: ATL, ETL, QVTo, and Reactions.
- The empirical study compares 3 LLMs across the 4 MTLs and multiple prompt configurations.
- Few-shot prompting improves syntactic quality across all 4 MTLs, but semantic correctness gains vary by language.
- For ATL, Pass@1 stays unchanged across all tested strategies and models, which means syntax gains did not translate into better first-try semantic success for that language.
- Grammar prompting helps when paired with few-shot examples, but used alone it can hurt or fail for some model-language pairs.
- The excerpt does not provide exact numeric Pass@1, syntax, semantic, or ChrF scores beyond the 47-example, 4-language, and 3-model study scale.