Recoleta
GitHub
Source note

Primal Generation, Dual Judgment: Self-Training from Test-Time Scaling

Authors
Yizhu Jiao, Ruixiang Zhang, Richard Bai, Jiawei Han, Ronan Collobert, Yizhe Zhang
Source
arXiv
Published
Collection
Software Intelligence
Code GenerationSelf TrainingTest Time ScalingReinforcement LearningProgram Ranking
Open arXivMarkdown source

Summary

DuST trains a code model to judge its own generated solutions using execution labels, then uses the same trained model for code generation. The paper claims this ranking-only GRPO training improves both judging and pass@1 generation on LiveCodeBench.

Problem

  • Code models often get only a sparse pass/fail signal for one generated program, which gives little information about near-miss solutions.
  • Generate-then-judge test-time scaling samples several programs and ranks them, but standard pipelines discard the ranking signal after inference.
  • Better reuse of that signal matters because it can reduce repeated sampling cost and improve the generator itself.

Approach

  • For each programming problem, the base model samples many candidate programs; the authors use 64 candidates per problem in the main data construction.
  • A sandbox executes each candidate and assigns a binary correctness label: passes all tests or fails.
  • Candidates are grouped in sets of 4, and only mixed groups with at least one correct and one incorrect solution are kept.
  • The model receives the problem and candidate set, outputs a ranking, and gets reward for placing correct programs above incorrect ones. This gives n+ × n- pairwise comparisons per group.
  • GRPO updates the same model parameters used later for generation; the reward is only for ranking quality, and generated code receives no direct correctness reward.

Results

  • Training data: about 10K de-duplicated rSTARcoder problems; 64 sampled candidates per problem; about 6.9K valid queries and 37K training groups for the primary model.
  • Qwen3-30B-Thinking on LiveCodeBench v6: pass@1 rises from 65.4% to 68.5% (+3.1), judgment NDCG from 70.1 to 76.3 (+6.2), and Best-of-4 accuracy from 68.7% to 72.6% (+3.9).
  • Qwen3-30B-Thinking on LiveCodeBench v5: pass@1 rises from 69.2% to 71.0% (+1.8), NDCG from 76.0 to 78.2 (+2.2), and Best-of-4 accuracy from 72.3% to 75.2% (+2.9).
  • GPT-OSS-20B on LiveCodeBench v6 improves Best-of-4 accuracy from 65.2% to 69.4% (+4.2), and on v5 from 67.5% to 72.8% (+5.3).
  • Qwen3-4B-Thinking on LiveCodeBench v6 improves Best-of-4 accuracy from 55.0% to 59.4% (+4.4); Qwen3-30B-Instruct improves from 43.1% to 47.1% (+4.0).
  • In the Qwen3-30B-Thinking ablation on LiveCodeBench v6, off-policy ranking reaches 72.6% TTS accuracy, 68.3% generation pass@1, and 76.3 NDCG, beating on-policy generation at 71.4%, 67.1%, and 74.6 respectively.

Link