Source note
Single-Language Evidence Is Insufficient for Automated Logging: A Multilingual Benchmark and Empirical Study with LLMs
Summary
This paper argues that automated logging cannot be evaluated reliably with Java-only snapshot datasets. It introduces MultiLogBench, a multilingual benchmark across six languages and tests seven LLMs on both snapshot and revision-history logging tasks.
Problem
- Automated logging asks a model to decide where to insert a log, which API/framework to call, what severity level to use, what message to write, and which variables to include.
- Prior evidence is mostly based on single-language, Java-heavy repository snapshots, so it does not show whether findings transfer across language ecosystems such as Python, Go, C++, JavaScript, and C#.
- Snapshot-only evaluation also misses the maintenance setting where developers add logs during real code changes, which matters if model behavior shifts on revision-history data.
Approach
- The authors build MultiLogBench, a benchmark covering six languages: Java, Python, Go, C++, JavaScript, and C#.
- The benchmark has three branches: 63,965 repository-snapshot instances, 744 revision-history cases mined from real log-introducing commits, and a paired transformed revision-history branch for robustness checks.
- They evaluate seven contemporary LLMs with one unified protocol over several subproblems: logging-site localization, framework-anchor matching, severity prediction, message generation, variable recovery, and cascaded end-to-end quality.
- They also analyze performance by structural context inside functions, with attention to hard positions such as loops and nested callables.
Results
- MultiLogBench contains 63,965 snapshot instances and 744 revision-history cases across 6 language ecosystems and 7 LLMs; this is the main benchmark contribution.
- The paper reports clear cross-language variation in automated logging performance. It states that framework-anchor matching is the most language-sensitive component.
- Structural context affects difficulty: loop sites and nested-callable sites are reported as the hardest cases.
- Model rankings are stable only at the top tier. Lower-ranked and mid-tier models reorder across languages, so a leaderboard from one language can mislead model selection.
- On revision-history data, the paper says absolute performance declines and ranking stability weakens, but the main cross-language patterns still remain.
- The excerpt does not provide task metrics or exact score values for the seven models, so no numeric comparison against specific baselines is available here.