Ironies of Generative AI: Understanding and Mitigating Productivity Loss in Human-AI Interaction

Authors: Auste Simkute, Lev Tankelevitch, Viktor Kewenig, Ava Elizabeth Scott, Abigail Sellen, Sean Rintel
Venue: International Journal of Human–Computer Interaction, 2024
Author's copy (PDF)

AThe paper

A narrative review linking three decades of automation research, in fields such as aviation and driving, to recent studies of people using generative AI tools such as GitHub Copilot. Its literature search ran to January 2024.

  1. Automation researchChallenges drawn from Bainbridge's Ironies of Automation and related Human Factors research on how people work with automated systems.
  2. Generative AI studiesSearches for studies reporting productivity loss or ineffective use of generative AI, mostly in programming, including pre-prints.
  3. Matching challengesFindings on generative AI were grouped under four challenge themes from automation research.
  4. Matching solutionsDesign approaches from automation research were mapped to each of the four challenges.

BWhat the paper argues

Why productivity can fallfour challenges argued from automation research and generative AI studies

  • The paper argues generative AI shifts people from producing work to checking it.Checking is hard because people lack the understanding they would have had from doing the work. Some programmers said an AI coding assistant felt like proofreading.
  • The paper argues this shift to checking can breed over-trust and over-reliance.Early studies of AI-assisted writing found expert and non-expert writers overconfident in AI drafts and failing to revise them thoroughly.
  • The paper argues prompting and adapting AI output add tasks that compete with the main work.These new tasks also break up familiar ways of working.
  • The paper argues unrequested AI suggestions interrupt people's flow.Even a suggestion a person rejects has already pulled that person's thinking away from the task.
  • The paper argues generative AI tends to make easy tasks easier and hard tasks harder.Subtle errors, overwhelming suggestions and disrupted workflows weigh most on demanding work. A study of legal work found AI helped with simple analysis but stumbled on complex reasoning.

What could helpdesign approaches proposed from automation research

  • The paper proposes design approaches from automation research to reduce productivity loss.The approaches include continuous feedback, personalisation, interfaces fitted to each field's work, steadier main tasks and clear task allocation. These still need testing with generative AI.
  • The paper notes its review was narrative and may have missed studies.The review deliberately focused on productivity losses, while acknowledging that some studies have observed gains.

CKey takeaway

Decades of research on the ironies of automation can guide generative AI design that keeps people productive. When AI turns people from makers into checkers, feedback and clear roles may restore the understanding that checking needs.

Made for this site with AI assistance from the verified research brief. It is not the paper's own figure.

Abstract

Generative AI (GenAI) systems offer opportunities to increase user productivity in many tasks, such as programming and writing. However, while they boost productivity in some studies, many others show that users are working ineffectively with GenAI systems and losing productivity. Despite the apparent novelty of these usability challenges, these ‘ironies of automation’ have been observed for over three decades in Human Factors research on the introduction of automation in domains such as aviation, automated driving, and intelligence. We draw on this extensive research alongside recent GenAI user studies to outline four key reasons for productivity loss with GenAI systems: a shift in users’ roles from production to evaluation, unhelpful restructuring of workflows, interruptions, and a tendency for automation to make easy tasks easier and hard tasks harder. We then suggest how Human Factors research can also inform GenAI system design to mitigate productivity loss by using approaches such as continuous feedback, system personalization, ecological interface design, task stabilization, and clear task allocation. Thus, we ground developments in GenAI system usability in decades of Human Factors research, ensuring that the design of human-AI interactions in this rapidly moving field learns from history instead of repeating it.

Research brief

By Auste Simkute, Lev Tankelevitch, Viktor Kewenig, Ava Elizabeth Scott, Abigail Sellen, and Sean Rintel.

Takeaway: Decades of automation research predict where generative AI costs productivity. People shift from producing work to checking output they didn't create, which demands understanding they no longer have.

Why this matters

What was known
Generative AI is expected to raise productivity, and some studies show gains, but users, especially programmers, also report extra cognitive load, frustration and lost time. This suggests that the potential of GenAI systems to boost productivity may not be guaranteed, evenly distributed, or fully exploited. (§1, p. 2)
What this paper adds
Four named causes of productivity loss with GenAI, each matched with design remedies from Human Factors research. Extrapolating from over 30 years of Human Factors research on the "ironies" of human-automation and productivity loss, we synthesize an overview of the usability and productivity challenges observed in recent GenAI user studies. (§1, p. 3)
Who it matters to
Designers and builders of GenAI tools and copilots, and HCI and Human Factors researchers. ensuring that the design of human-AI interactions in this rapidly moving field learns from history instead of repeating it (Abstract, p. 2)

Research questions

Method and data

Design
Narrative review in two stages. It first distilled challenges from Bainbridge's 'Ironies of Automation' and related Human Factors research, then searched the generative AI literature for studies reporting productivity losses or ineffective use.
Setting
GenAI user studies, mainly programming (e.g., GitHub Copilot), plus writing, design, data science, law, consulting and healthcare.
Participants / data
Google Scholar and ACM Digital Library searches (generative AI, large language model, LLM, ChatGPT), cut-off 2024-01-20, including pre-prints. Included GenAI articles are listed in Table 1 with domain, task, method and challenge theme.
Analysis
Thematic synthesis aligning GenAI findings with four Human Factors challenge themes, then mapping Human Factors solutions to each challenge.

Findings

Questions this paper answers

Why does AI help more with easy tasks than hard ones?
Automation research calls this task-complexity polarization. Easy work is easiest to automate, so hard work stays with people, now with added monitoring in a fragmented workflow. Generative AI studies agree. The hardest part, such as breaking a coding problem down, gets little help, and subtle errors make debugging harder. (See §3.4, p. 11; §3.1, p. 4.)
Do people really over-rely on AI?
Automation research predicts it, and early generative AI studies already show it. When people shift to checking AI output, with less awareness of how it was made and more to do, they tend to accept it without enough scrutiny. (See §3.1.3, p. 8.)
What are the 'ironies of automation', and do they apply to AI?
They are a pattern from decades of automation research. The more advanced the automation, the more important the human's remaining role may be, and in some cases the harder it becomes. The same pattern is appearing in studies of generative AI tools. (See §3.1, p. 4; §3.4, p. 11.)

Contributions and implications

Limitations

›

Concepts

Ironies of automation
Bainbridge's term from automation research, which we apply to GenAI: the more advanced an automated system is, the more important the human operator may be (§1, p. 2)
Production-to-evaluation shift
Our name for the first of four GenAI productivity challenges: users' roles have shifted from producing output to evaluating it, often with little contextual information and situational awareness (§1, p. 2)
Situational awareness
Endsley's definition, central to why evaluating AI output is hard: their perception of data and elements of the situation, comprehension of the situation, and the projection of future status (§3.1.1, p. 6)
Unhelpful workflow restructuring
The second challenge, in the words of automation research: Automation can restructure workflows in unhelpful ways by introducing new challenging tasks, disrupting familiar task sequences, and removing informative feedback (§3.2, p. 8)
Task-complexity polarization
Our more precise term for what Human Factors research calls 'clumsy automation': Automation often makes easy tasks easier but fails to reduce the workload during cognitively demanding tasks, and in fact, often makes them harder (§3.4, p. 11)

Figures

Four icon panels. (a) A user shifts from writing code to inspecting AI output. (b) Prompting, output adaptation, lost task sequence, lost feedback. (c) An AI suggestion interrupts a user. (d) Easy tasks succeed while complex ones gain errors.
Figure 1. Productivity challenges of Generative AI automation: (a) the production-to-evaluation shift, in which users' situational awareness of their working environment is reduced, increasing the cognitive demand required to evaluate AI outputs; (b) unhelpful workflow restructuring, including the addition of new challenging tasks of prompting systems and adapting outputs, a loss of task sequence due to AI suggestions or other changes, and a loss of feedback when AI suggestions are presented without the relevant context; (c) task interruptions from automated AI suggestions; and (d) task-complexity polarization, in which automation tends to make easy tasks easier and hard tasks harder when implemented in practice. From the paper, Figure 1, p. 6.
Six icons for feedback features addressing the production-to-evaluation shift: alternative solutions, usage examples, highlighted prompt and output changes, links to documentation, confidence estimates, and errors mapped to inputs.
Figure 2. Examples of approaches to providing continuous feedback and increasing explainability which can help address the challenges of the [production-to-evaluation] shift (Section 3.1). Examples include providing alternative solutions, adding links to relevant documentation or other information sources, providing relevant usage examples, conveying information based on confidence, highlighting prompt changes and resulting output changes, and providing information linking errors to inputs. From the paper, Figure 2, p. 13.

How to cite

Auste Simkute, Lev Tankelevitch, Viktor Kewenig, Ava Elizabeth Scott, Abigail Sellen, and Sean Rintel. 2024. Ironies of Generative AI: Understanding and Mitigating Productivity Loss in Human-AI Interaction. International Journal of Human–Computer Interaction. https://doi.org/10.1080/10447318.2024.2405782

This brief was prepared with AI assistance and published by Sean Rintel. It is not the authors' published text. Page numbers refer to this PDF.

BibTeX

@article{simkute2024ironies,
author = {Simkute, Auste and Tankelevitch, Lev and Kewenig, Viktor and Scott, Ava Elizabeth and Sellen, Abigail and Rintel, Sean},
title = {Ironies of Generative AI: Understanding and Mitigating Productivity Loss in Human-AI Interaction},
year = {2024},
month = {October},
url = {https://doi.org/10.1080/10447318.2024.2405782},
journal = {International Journal of Human–Computer Interaction},
doi = {10.1080/10447318.2024.2405782},
}