Recoleta
GitHub
Source note

Rethinking Issue Resolution for AI/ML Systems

Authors
Ahmed Adnan, Mushfiqur Rahman, Antu Saha, Oscar Chaparro
Source
arXiv
Published
Collection
Software Intelligence
AI ML MaintenanceIssue ResolutionSoftware Engineering For MLReproducibilityArtifact Provenance
Open arXivMarkdown source

Summary

The paper argues that AI/ML issue resolution needs workflows that model experimentation, nondeterminism, and artifacts beyond source code. A qualitative study of 100 issues across TensorFlow, scikit-learn, MLflow, and AutoGPT provides preliminary evidence for an iterative, artifact-aware framework.

Problem

  • Traditional issue-resolution frameworks assume mostly deterministic behavior, code-centric debugging, and pass/fail verification, which do not fully fit AI/ML systems.
  • Resolving AI/ML issues can require changes to datasets, prompts, models, hyperparameters, dependencies, and infrastructure, making reproducibility and traceability difficult.

Approach

  • The authors qualitatively analyzed 100 randomly sampled closed issues and associated pull requests from 2020–2025, with 25 issues each from TensorFlow, scikit-learn, MLflow, and AutoGPT.
  • Two authors independently open-coded 988 snippets, reconciled disagreements through consensus, and expanded an existing issue-resolution codebook with AI/ML-specific activities, challenges, and mitigation strategies.
  • The analysis compared traditional resolution stages with AI/ML activities such as model performance monitoring, parameter tuning and training, data modification, and model functionality analysis.
  • Based on the findings, the paper proposes future frameworks supporting iterative cross-stage workflows, reproducibility-aware verification, artifact provenance, heterogeneous-artifact coordination, and human-AI collaboration.

Results

  • The study identified 947 traditional issue-resolution activities and 41 AI/ML-specific activities across the 100 issues; 64 issues were classified as AI/ML, 18 as hybrid, and 18 as non-AI/ML.
  • AI/ML-specific activities occurred at these rates: model performance monitoring in 27% of issues, parameter tuning and training in 11%, data modification in 7%, and model functionality analysis in 5%.
  • Among the 64 AI/ML issues, 28 (45%) required changes outside production code, including prompts, datasets, hyperparameters, dependencies, CI, Docker, or runtime configuration. The corresponding figures were 83% code-only resolution for non-AI/ML issues and 77% for hybrid issues.
  • Traditional stages were also common: issue analysis appeared in 65% of issues, solution design in 62%, implementation in 54%, verification in 53%, reproduction in 42%, and code review in 34%.
  • Reported challenges included scaling fixes to larger datasets or distributed training in 9 issues, understanding model architectures in 6, nondeterministic solution verification in 4, and issue reproduction in 4. Developers used repeated executions, statistical tests, containerized environments, documentation, and distributed validation as mitigations.
  • The evidence is preliminary: it covers four open-source projects and uses qualitative consensus coding without inter-coder agreement metrics, so the patterns may not generalize to other AI/ML systems.

Link