Source note
FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale
Summary
FrontierSmith turns closed-ended competitive programming tasks into open-ended optimization-style coding problems for training LLM coders. The paper claims that 200 synthesized problems improve Qwen3.5 models on FrontierCS and ALE-bench.
Problem
- Many real coding tasks have no known optimal solution and need continuous scoring, while most LLM coding training uses binary pass/fail tasks.
- Open-ended coding data is scarce and costly because each problem needs an objective, test cases, and a verifier that can score solution quality.
- The gap matters because open-ended benchmarks remain hard: FrontierCS reports human experts at 95.41 on algorithmic tasks versus 29.37 for Gemini 3.0 Pro, and FrontierCS and ALE-bench contain only about 240 and 40 human-curated problems.
Approach
- FrontierSmith starts with closed-ended competitive programming problems and mutates them by changing goals, restricting valid outputs, or generalizing inputs.
- These mutations convert exact-answer tasks into optimization tasks where solvers can use different heuristics and receive continuous scores.
- A coarse LLM judge filters candidates for open-endedness, then an idea divergence metric keeps problems where sampled solutions use different core algorithms.
- The system estimates idea divergence first with pairwise LLM judgments, then with distances between verifier score vectors across generated test cases.
- Separate agents generate test cases and scoring verifiers, cross-check them, and validated problems enter the seed pool for later synthesis rounds.
Results
- Training Qwen3.5-9B with GRPO on 200 synthesized problems improves FrontierCS by +8.82 points and ALE-bench by +306.36 Elo-rating-based performance.
- Training Qwen3.5-27B improves FrontierCS by +12.12 points and ALE-bench by +309.12.
- Compared with closed-ended HardTests, FrontierSmith training is higher by +5.24 on FrontierCS and +236.40 on ALE-bench.
- Compared with random rewards, FrontierSmith training is higher by +7.58 on FrontierCS and +256.76 on ALE-bench.
- Removing the idea-divergence filter drops FrontierCS performance by 2.05 points, which supports the filter's value in selecting training problems.
- The paper also reports that synthesized problems make agents use more turns and tokens, matching behavior seen on human-curated open-ended problems, but the excerpt does not give exact turn or token counts.