Source note
Developers' Experience with Generative AI Beyond Productivity Assessment -- Insights from an Empirical Mixed-Methods Field Study
Developer ExperienceGithub CopilotCode IntelligenceHuman AI InteractionMixed Methods Study
Summary
This paper studies how 22 SAP developers use generative AI coding tools during real work, with attention to workload, satisfaction, and interaction style rather than productivity alone.
Problem
- Prior studies mainly measure code quality or task speed, so they miss how developers experience GitHub Copilot and similar tools during daily work.
- Organizations need to know when GenAI helps developers and when it adds review work, cognitive load, or workflow friction.
- The paper focuses on interaction types, including in-code suggestions and chat prompts, because different tasks may need different tool behavior.
Approach
- The study followed 22 SAP employees over 4 days, with controlled sessions on the first and last day and normal work between them.
- Controlled tasks covered 6 task types: coding, debugging, code documentation, unit tests, summarization, and brainstorming.
- The researchers combined questionnaires, NASA-TLX workload ratings, screen recordings, keyboard and mouse logs, and wristband physiology; this paper focuses on questionnaires and screen recordings.
- GitHub Copilot was the controlled-session tool, with GPT-4o as the default model and ask mode as the default chat mode.
- The analysis compares no Copilot use, in-code suggestions, chat prompts, and combined use within the same task.
Results
- The excerpt gives no quantitative effect sizes for the paper's main findings, but it reports a 22-person professional field study and about 66 hours of screen recordings.
- Participants had an average of 5 to 10 years of IT/programming experience; the cohort included 12 software developers/engineers, 8 senior software developers/engineers, 1 senior quality specialist, and 1 principal software architect.
- Developers were generally satisfied with GenAI, especially for monotonous, repetitive, and structured tasks.
- The paper claims that in-code suggestions and chat prompts each improve task efficiency and reduce perceived workload when used separately.
- Combining in-code suggestions and chat prompts within one task reduced the benefits, according to the reported findings.
- For development-heavy tasks, perceived cognitive load came from the AI interaction, while perceived productivity depended on AI output quality.