Recoleta
GitHub
Source note

Empirical Study for Structured Output Control in LLMs for Software Engineering

Authors
Yewei Song, Prateek Rajput, Tiezhu Sun, Saad Ezzini, Tegawendé F. Bissyandé, Jacques Klein
Source
arXiv
Published
Collection
Software Intelligence
Structured OutputCode IntelligenceLLM DecodingSoftware EngineeringFunction Calling
Open arXivMarkdown source

Summary

This paper studies why LLMs fail at structured software-engineering outputs such as JSON, SQL, code, and function calls. Its main claim is that syntax control helps, but it does not solve schema-level and value-level failures.

Problem

  • LLM outputs often feed parsers, APIs, deployment tools, and data pipelines, so a correct idea can still fail if the output violates JSON syntax, SQL grammar, a function signature, or a local schema.
  • The paper separates failures into syntax errors, structural errors, and value errors, because each failure type needs a different fix.
  • The problem matters for automated software workflows: malformed API calls, configs, or queries can break pipelines or pass wrong data into later steps.

Approach

  • The study evaluates structured-output behavior across 4 software-engineering task types: text-to-code on BigCodeBench, text-to-SQL on Spider, text-to-JSON on CallNavi, and function calling on BFCL.
  • It compares 9 LLMs: 7 general-purpose models and 2 code-focused models, with sizes listed for several models from 7B to 70B.
  • It uses pass@1 for Python and SQL tasks, and AST-based evaluation for JSON and function-call tasks.
  • It tests decoder-side controls: grammar-constrained decoding, regex-based validation, and Template Token Match Generation, or TTMG.
  • TTMG fixes fixed-format parts by matching template tokens, then lets the model generate only the variable content, using mode switches between template copying and free generation.

Results

  • The provided excerpt does not include final pass@1 scores, AST accuracy scores, or full error-rate tables, so it does not support a precise numerical claim about overall task accuracy.
  • The strongest claim is that TTMG nearly eliminates syntax errors, while structural errors and value errors still remain.
  • The evaluation spans 4 task families and 4 named benchmarks: BigCodeBench, Spider, CallNavi, and BFCL.
  • The model set contains 9 models: LLaMA-3.1-8B, Qwen-2.5-7B-it, Gemma2-9B-it, Qwen3-30B-A3B-IT, Mixtral-8x7B-Instruct-v0.1, LLaMA-3.1-70B GGUF, GPT-4.1-mini, Qwen2.5-Coder-7B-it, and Seed-Coder-8B-it.
  • The paper reports that prompt tuning uses 5% of each dataset’s samples before the main benchmark.
  • The background survey reports 8 structured-output tools and notes specific system costs or claims, such as LLGuidance at about 50 µs CPU per token for a 128k vocabulary and XGrammar claiming up to 5× faster token generation under load.

Link