Recoleta
GitHub
Source note

Alipay-PIBench: A Realistic Payment Integration Benchmark for Coding Agents

Authors
Shiyu Ying, Xuejie Cao, Yingfan Ma, Yuanhao Dong, Wenyu Chen, Bowen Song, et al.
Source
arXiv
Published
Collection
Software Intelligence
Coding AgentsSoftware Engineering BenchmarksPayment IntegrationRepository Level EvaluationSecure Software
Open arXivMarkdown source

Summary

Alipay-PIBench is a repository-level benchmark for testing whether coding agents can implement secure, reliable, end-to-end Alipay payment integrations in realistic business applications. It evaluates functional completion and risk hardening across nine product-specific projects and 18 task instances.

Problem

  • General coding benchmarks often miss domain requirements such as server-side credential protection, signature verification, asynchronous notification handling, and consistency between payment and local business states.
  • Payment integration errors can create fund-safety risks, incorrect fulfillment, duplicate state transitions, and unsafe refunds, so source-level code completion alone is insufficient.

Approach

  • The benchmark pairs nine Alipay payment products and business repositories with two progressive scenarios: Basic Functional Payment Completion and Advanced Risk-Aware Payment Hardening.
  • Tasks require repository-level changes across frontend, backend, payment APIs, configuration, and business-state logic; advanced tasks test idempotency, abnormal transactions, notification authenticity, repeated confirmations, and refund safeguards.
  • Scenario-specific rubrics generate deterministic static, unit, integration, and end-to-end checks, supplemented by LLM-assisted assessment for semantic properties such as product fit and state consistency.
  • A weighted Rubric Pass Rate (RPR) gives integration and end-to-end checks weight 2, while static, unit, and LLM-assisted checks receive weight 1.
  • Paired trials compare six models with and without the official alipay-payment-integration skill under fixed task and environment conditions.

Results

  • Across six models and 18 task instances, mean with-skill RPR ranges from 68.58% to 91.37%.
  • Access to the skill improves mean RPR by 10.31 percentage points on average, with gains in 101 of 108 model-product-scenario comparisons.
  • The average improvement is larger for Basic tasks (+11.27 percentage points) than for Advanced tasks (+9.35 percentage points).
  • The benchmark separates source-level completion, executable payment behavior, and payment-domain requirements, showing why multiple evaluation signals and progressive scenarios are needed.
  • The excerpt does not provide complete per-model, per-product, or without-skill score tables, so it supports aggregate skill-effect conclusions more strongly than detailed model rankings.

Link