Recoleta
GitHub
Source note

Not All RAGs Are Created Equal: A Component-Wise Empirical Study for Software Engineering Tasks

Authors
Qiang Ke, Yanjie Zhao, Hongjin Leng, Shengming Zhao, Haoyu Wang
Source
arXiv
Published
Collection
Software Intelligence
Retrieval Augmented GenerationCode IntelligenceSoftware EngineeringEmpirical StudyRAG Optimization
Open arXivMarkdown source

Summary

This paper tests which parts of a RAG pipeline matter most for Python software engineering tasks. Its main claim is that retrieval choices, especially the retriever, often affect final quality more than the generator.

Problem

  • RAG systems for software tasks require choices about query processing, retrieval, context refinement, and generation, but practitioners lack task-specific evidence for those choices.
  • Prior work mainly studied code generation or one retrieval component, leaving weaker guidance for code summarization and code repair.
  • Poor component choices can feed irrelevant code or documentation into an LLM and increase trial-and-error cost.

Approach

  • The authors build a modular Code RAG testbed that separates query processing, retrieval, context refinement, and generation so each part can be tested alone.
  • They evaluate 3 tasks: code generation, code summarization, and code repair.
  • They compare 4 query processing techniques, 7 retrieval models across sparse, dense, and hybrid retrieval, 4 context refinement methods, and 6 generators.
  • The retrieval corpus combines Stack Overflow, Python API docs, LeetCode, CodeSearchNet, and Code-Contests, with exact-match test-solution decontamination.
  • They also build an LLM-driven adaptive configuration system that maps task features to component choices using rules derived from the experiments.

Results

  • The excerpt provides no final W-Pass@1, CodeBLEU, or embedding-similarity scores, so the claimed performance gains cannot be checked numerically from the supplied text.
  • The study covers over 21 models and methods: 4 query processing techniques, 7 retrieval models, 4 context refinement methods, and 6 generator models.
  • The evaluation uses 4 datasets or dataset variants: a 300-problem APPS subset, 100 long CodeXGLUE Python snippets, an obfuscated CodeXGLUE variant, and a 300-sample DebugBench subset.
  • The retrieval corpus contains 232,633 documents: 164,085 from Stack Overflow, 38,352 from Python API docs, 3,174 from LeetCode, 13,590 from CodeSearchNet, and 13,432 from Code-Contests.
  • The main empirical claim is that retriever-side components influence final RAG performance more than generator choice in many settings, and BM25 performs well across the tested software tasks.
  • The compression setup reports average context sizes of 1,533.47 tokens for 10 chunks, 3,578.81 for 20 chunks, 7,818.43 for 40 chunks, and 17,105.42 for 80 chunks.

Link