Recoleta
GitHub
Source note

TeleSWEBench: A Commit-Driven Benchmark for Evaluating LLM-Powered Software Engineering in Telecommunications

Authors
Pranshav Gajjar, Ali Mamaghani, Dinesh Bharadia, Vijay K Shah
Source
arXiv
Published
Collection
Software Intelligence
Code IntelligenceSoftware Engineering AgentsTelecom SoftwareBenchmarkingLLM As Judge
Open arXivMarkdown source

Summary

TeleSWEBench is a 734-task benchmark for testing LLM software-engineering agents on real srsRAN 5G commits. It shows current agents still struggle with telecom code: the strongest tools produce up to 25% ship-ready changes.

Problem

  • Telecom networks now depend on large C++ wireless stacks; code changes can affect 5G protocol behavior, timing, and state machines.
  • General coding benchmarks such as SWE-bench, HumanEval, and MBPP do not test repository-scale 3GPP logic in systems like srsRAN 5G.
  • Existing telecom LLM evaluations focus on question answering or code understanding, leaving no direct test for multi-file code generation in telecom repositories.

Approach

  • The benchmark mines real developer commits from the srsRAN 5G repository across 2023-2025, starting from more than 15,000 commits.
  • It keeps commits with repository-native tests and turns them into 734 coding tasks: 142 Easy, 279 Medium, and 313 Difficult.
  • Difficulty controls how much location detail the prompt gives: Easy includes exact files and edits, Medium gives affected areas and facts, and Difficult gives only a high-level goal.
  • The evaluation has two stages: file localization first, then functional correctness only for exact-file-match patches.
  • The paper also proposes TeleJudge, a hierarchical LLM judge that scores file-level diffs and combines them into a patch verdict alongside executable unit tests.

Results

  • TeleSWEBench contains 734 questions with executable tests, built from srsRAN 5G commits; task scope ranges from 1 to 300 files.
  • In Stage 1 localization, QwenCoder-2.5 has the best Easy exact-match rate at 37.8%, while Qwen3 has the best cumulative exact-match rate at 14.0% among the listed models.
  • Localization drops on harder tasks: QwenCoder-2.5 falls from 37.8% exact match on Easy to 5.3% on Difficult.
  • Many larger models often make no code changes: GLM-4.7 has a 92.2% cumulative No Changes rate, and Gemma4 has 86.8%.
  • The abstract reports that the strongest evaluated ASE tools reach up to 25% ship-ready changes.
  • The evaluated model set includes Qwen3.5 at 397B parameters with a 1M-token context, Kimi K2.5 at 1T parameters with a 262K-token context, GPT-OSS at 120B, Gemma 4 at 31B, and QwenCoder 2.5 at 1.5B.

Link