Recoleta
GitHub
Source note

Contextualized Code Pretraining for Code Generation

Authors
Chen Liu, Qingyuan Liang, Hanwen Zhang, Zeyu Sun, Yakun Zhang, Lu Zhang
Source
arXiv
Published
Collection
Software Intelligence
Code GenerationCode PretrainingCaller ContextRepository Level CodeStatic Analysis
Open arXivMarkdown source

Summary

CallerGen trains code models to generate missing Python functions using the caller code that already invokes them. The paper claims this caller-conditioned pretraining improves pass@1 on real-repository generation tasks, especially for small models.

Problem

  • Real projects often need a missing function implemented after caller code already exists, so the call site can reveal expected inputs, return shape, errors, and side effects.
  • Common code models are trained mainly on code or natural-language descriptions, so they may ignore caller-side usage when the function signature is under-specified.
  • Retrieval-based repository methods can fail when similar examples do not exist or are not retrieved, which matters for new modules, small repositories, and early-stage code.

Approach

  • The method extracts caller-callee pairs from 800 Python GitHub repositories using AST-based static analysis, symbol tables, import alias resolution, and a function-level call graph.
  • Each training sample pairs a target callee with its direct caller function bodies and optional docstring, then asks the model to generate the missing callee implementation.
  • CallerGen is trained at 60M, 220M, and 0.5B parameter scales with an invocation-aware objective.
  • The paper applies the idea to both encoder-decoder CodeT5-style models and decoder-only Qwen2.5-Coder-style models.
  • CallerEval is introduced as a benchmark of real caller-callee cases with execution-based tests tied to caller-constrained behavior.

Results

  • On CallerEval, CallerGen-220M reaches 16.58% pass@1 and CallerGen-0.5B reaches 22.81% pass@1.
  • On CoderEval with calling contexts, CallerGen-220M reports 21.22% pass@1, compared with 8.78% for CodeGen-350M.
  • CallerGen-0.5B reaches 22.81% pass@1 on CallerEval and is reported to beat Qwen2.5-Coder-32B-Instruct by nearly 2 percentage points.
  • Without calling contexts, CallerGen-220M reaches 10.00% pass@1 from function headers on CoderEval, compared with 7.39% for CodeGen-350M.
  • The evaluation compares CallerGen with nine open-source code models across CoderEval and CallerEval.

Link