Research brief
Takeaway: Metacognition (monitoring and controlling one's own thinking) offers a way to understand why generative AI is hard to use. Users must state goals, decompose tasks and calibrate confidence. Tools could support users by prompting planning and self-evaluation, and could demand less by explaining outputs and offering the right controls.
Why this matters
- What was known
- User studies have documented many usability challenges with generative AI, in prompting, evaluating outputs, and deciding how to use it in workflows.
However, there is not yet a coherent understanding of the usability challenges of GenAI, much less one grounded in a theory of human cognition.
(§1, p. 1) - What this paper adds
- We bring research on metacognition from psychology and cognitive science to bear on generative AI user studies, identify the metacognitive demands these systems impose, and propose two directions for addressing them, with open research questions.
The metacognitive demands of working with GenAI systems parallel those of a manager delegating tasks to a team.
(§1, pp. 1–2) - Who it matters to
- Designers and researchers of human-centred generative AI systems and human-AI interaction, and metacognition researchers.
Designing truly human-centered GenAI systems [34, 104] means grappling with these metacognitive demands.
(§5, p. 18)
Research questions
- How can metacognition (the ability to monitor and control one's own thoughts and behaviour) help us understand the usability challenges of generative AI in prompting, evaluating and relying on outputs, and deciding on an automation strategy?
- How can these metacognitive demands be addressed, both by improving users' metacognition and by designing systems that reduce the demands?
Method and data
- Design
- Conceptual analysis: a simplified descriptive framework of metacognition applied to a simplified generative AI workflow, with illustrative hypothetical design examples.
- Setting
- Generative AI use in work and everyday tasks, with evidence drawn mainly from programming, writing, design and data science.
- Participants / data
- Published research on metacognition in psychology and cognitive science, recent generative AI user studies, and research on metacognitive interventions and human-AI interaction.
- Analysis
- Synthesis of the literature into metacognitive demands at each stage of use (prompt formulation, prompt iteration, output evaluation, automation strategy), proposed responses, and open research questions summarised in tables.
Findings
Prompting makes the implicit explicit. Drawing on user studies of generative AI, mostly in programming, writing, design and data science, we argue that prompting requires people to be aware of their own goals and to break tasks into parts. Goals that stay implicit when working manually must be spelled out for the AI. #
“For example, when writing an email to a senior colleague, one might implicitly know to adopt a certain tone. Many GenAI systems require specification that the email is to a senior colleague and needs an appropriate tone.”
Was it me, or the machine? When an output disappoints, users must decide whether to blame their prompt, the settings, the system or chance, and adjust their approach. Because generative AI gives different outputs to the same prompt, this calls for well-adjusted confidence in one's prompting ability and the flexibility to change strategy. #
“Is my prompt specific and clear enough; are system parameters set appropriately; is system performance generally poor on this task; or is this an 'unlucky' probabilistic output?”
Judging output needs calibrated confidence. Judging AI output requires confidence in one's own judgement that matches one's actual ability. Research on metacognition suggests that fast, fluent output may inflate that confidence, and that the effort of checking long outputs may lower the bar people set before accepting them. #
“when problems took longer to solve, participants appeared to compromise on their confidence criterion, and were willing to provide solutions with less confidence”
Users must decide when to use AI. Because generative AI can be applied to so many tasks, deciding whether, when and how to use it falls to the user. That calls for awareness of how it affects one's work, confidence about doing tasks with and without it, and flexibility in changing workflows. #
“I turned off autosuggest and that made a huge difference. Now I'll use it when I know I'm doing something repetitive that it'll get easily”
Tools could build better habits. Systems could improve users' metacognition by building in support for planning, self-evaluation and self-management, adapted to each user. For example, in a study by Wu and colleagues, breaking a task into a chain of smaller prompts helped people think through the task. #
“Chaining also increased users' self-awareness of their goals: the ability to decompose tasks led some participants to create more generalizable outputs better suited to their broader goals”
Tools could carry some of the load. Systems could also lower the metacognitive demands themselves, by explaining outputs and uncertainty and by offering the right amount of customisation, though such support can add cognitive load of its own. #
“by providing contextual and performance information alongside the system inputs and outputs, explainability should help partly offload metacognitive processing from users' minds and onto the system interface”
Questions this paper answers
- Why is it so hard to write good prompts for AI?
- Prompting asks you to know exactly what you want and break the task into parts. You then judge whether a poor result came from your prompt, the settings, the system or chance, and adapt. These are metacognitive demands (monitoring and controlling your own thinking). (See §3.1, p. 6; §3.1, p. 7.)
- Can fast, fluent AI answers make me overconfident?
- We argue they may. Quick, extensive output could act as a cue that raises confidence in the output and in your ability to judge it. That can reduce the effort you put into checking. Research measuring this with generative AI is still needed. (See §3.2, p. 9.)
- Can AI tools help people think better about their own work?
- We argue they can. In a study by Wu and colleagues, breaking a task into a chain of smaller AI steps made some people more aware of their goals. Systems could build in similar nudges to plan and reflect, acting like a coach at key moments. (See §4.1, p. 13.)
- Who decides when to use generative AI at work?
- Largely the user. Because generative AI applies to so many tasks, users must judge whether it helps their workflow and how confident they are doing a task themselves versus with AI. They then adapt their workflow accordingly. (See §3.3, p. 11.)
Contributions and implications
- Some effort introduced by metacognitive support and explanations may be worth it, questioning the assumption that interfaces should always be effortless.
We propose that some potential effort introduced by metacognitive support strategies and explanations may be justified
(§4.3, p. 18) - Metacognitive prompts could be adapted and gradually faded out as users internalise the strategies, reducing cognitive load over time.
- The balance between too much and too little customisability needs exploring, ideally combined with metacognitive support.
- Interaction with generative AI is also a paradigm for advancing basic understanding of metacognition.
interaction with GenAI offers a powerful paradigm for advancing our foundational understanding of metacognition
(§5, p. 18)
Limitations
- Evidence on how generative AI changes workflows comes mostly from short studies of programming.
The above reports are limited to short study contexts, focusing only on programming.
(§3.3, p. 10) - Research on self-confidence and reliance has mainly used discriminative AI models, not generative AI.
Recent work has investigated the role of self-confidence in evaluating and relying on AI output, although with discriminative models, rather than GenAI.
(§3.2, p. 8) - Support for metacognition may itself add cognitive load. The net effect is hypothesised, not shown.
There is a risk that strategies to address metacognitive demands may also increase cognitive load
(§4.3, p. 17)
›Concepts
- Metacognition
- Our working definition, from psychology and cognitive science:
the psychological ability to monitor and control one's own thought processes
(§1, p. 1) - Metacognitive demand
- Distinct from cognitive load (the total mental effort a task requires), though it contributes to it:
the need for extensive metacognitive monitoring and control for a task
(§3, p. 5) - Well-adjusted confidence
- Confidence is one's self-assessment of one's own abilities. It is well-adjusted when it
distinguishes objectively correct and incorrect performance, and accurately matches one's abilities
(§2.2, p. 3) - Metacognitive flexibility
- Shifting strategy when new information arrives, a strategy is not working, or the task changes. The paper's example:
I recognize that my formal tone in my emails does not match the more conversational style of my new co-workers. I should therefore adjust my approach while still maintaining professionalism
(§2.2, p. 3) - Automation strategy
- Deciding whether, when and how to use generative AI in one's workflow, which the paper likens to a manager's choice about delegating:
Moreover, they need to decide whether, when, and how to even delegate tasks in the first place.
(§1, pp. 1–2) - Fuzzy abstraction matching
- Sarkar and colleagues' term for a problem that arises because generative AI works at many levels of abstraction:
it becomes difficult for users to discern a system's capabilities and to match one's intent and prompting accordingly
(§3.1, p. 8)
Figures
How to cite
Lev Tankelevitch, Viktor Kewenig, Auste Simkute, Ava Elizabeth Scott, Advait Sarkar, Abigail Sellen, and Sean Rintel. 2024. The Metacognitive Demands and Opportunities of Generative AI. CHI '24. https://doi.org/10.1145/3613904.3642902
This brief was prepared with AI assistance and published by Sean Rintel. It is not the authors' published text. Page numbers refer to this PDF.