Work in progress method paper, tested in a pilot with two people who use spreadsheets for their work. Each pilot session lasted about one hour.
Choose a chatbotResearchers compared four generative AI systems and chose Bing Chat, mainly because it draws on information from the web.
Build prompting strategiesResearchers developed prompting strategies over several weeks of trial and error.
Pilot with two peopleA convenience sample of two spreadsheet users each worked on a problem from their own work.
Revise the protocolObservations from the two pilot sessions were used to revise the protocol and the study script.
BThe method
Ground it in real workA semi-structured interview centres on a decision or problem from the participant's own work.
Participant asksThe participant says what to ask the generative AI system.
Researcher rewritesThe researcher rewrites the request using prepared prompting strategies and sends it to the system.
Read and reflectThe participant reads the response, reflects on it and is guided to the next request.
Interview afterA post-activity interview follows the rounds of asking and reflecting.
In the pilot, participants got through 5 to 6 rounds in about an hour.
CWhat the study found
What the pilots showedtwo pilot sessions, not claimed as findings
•In two pilot sessions, the protocol worked as intended at very low cost.The paper argues the method improves on studies where a researcher secretly simulates the AI, because participants use a real AI model.
•In the pilots, a few rounds an hour still produced a wide range of insights.Participants got through 5 to 6 rounds. Most time went on reflecting on each response and deciding what to ask next.
•In the pilots, participants started with short, search-style requests.The researchers added guidance explaining that longer, conversational requests are possible.
•In the pilots, the chatbot's verbose creative mode was no more likely to make things up.The longer answers of creative mode also gave participants more to reflect on, so the researchers chose it for the study.
Limits of the methodwhat the paper notes
•The paper notes researcher help hides where participants' own prompting would fail.Participants are asked how they would have changed prompts, which partly offsets this.
•The paper notes its prompting strategies are subjective and hard to reproduce.The strategies came from trial and error and the researchers' intuitions.
DKey takeaway
Participatory prompting can ground user research on generative AI in what working systems can actually do, with a researcher prompting a chatbot on participants' own problems. This can cheaply improve on simulated AI.
Made for this site with AI assistance from the verified research brief. It is not the paper's own figure.
Abstract
Generative AI, such as image generation models and large language models, stands to provide tremendous value to end-user programmers in creative and knowledge workflows. Current research methods struggle to engage end-users in a realistic conversation that balances the actually existing capabilities of generative AI with the open-ended nature of user workflows and the many opportunities for the application of this technology. In this work-in-progress paper, we introduce participatory prompting, a method for eliciting opportunities for generative AI in end-user workflows. The participatory prompting method combines a contextual inquiry and a researcher-mediated interaction with a generative model, which helps study participants interact with a generative model without having to develop prompting strategies of their own. We discuss the ongoing development of a study whose aim will be to identify end-user programming opportunities for generative AI in data analysis workflows.
Research brief
By Advait Sarkar, Ian Drosos, Robert DeLine, Andy Gordon, Carina Negreanu, Sean Rintel, Jack Williams, and Ben Zorn.
Takeaway: Participatory prompting grounds user research on generative AI in what real systems can do. A researcher crafts prompts to a working chatbot for participants' own problems, cheaply improving on Wizard-of-Oz.
Why this matters
What was known
Low-fidelity methods such as Wizard-of-Oz and paper prototyping are cheap and flexible, but with generative AI they risk drifting from what real systems can do, and participants fill the gaps with inflated expectations. for a Wizard-of-Oz study to have direct implications for design, the Wizard protocol must correspond to the actually existing capabilities of the system(s) that are eventually built(§1, p. 2)
What this paper adds
A protocol, study script and set of pre-tested prompting strategies that combine contextual inquiry and participatory design with an off-the-shelf AI system, so no fine-tuning, customisation or interface building is needed. (§4, p. 9)
Who it matters to
Researchers and designers trying to understand where generative AI could help in knowledge workflows where it is not yet widely used, such as data analysis in spreadsheets. A key question for researchers and designers at this point in time is how to study the needs of users involved in such workflows.(§1, p. 2)
Research questions
How can researchers engage end-users in realistic conversations about generative AI that balance its actually existing capabilities with the open-ended nature of their workflows?
Where are the opportunities for generative AI assistance in end-user data analysis workflows in spreadsheets (the aim of the planned full study)?
Method and data
Design
Work-in-progress method paper. A semi-structured interview, grounded in a decision or problem from the participant's own work, run alongside researcher-mediated turns with a generative AI system. Each turn has 5 steps: the participant asks, the researcher rewrites the query using pre-identified prompting strategies and sends it, the participant reads the response, reflects on it, and is guided to the next query. A post-activity interview follows.
Setting
Data analysis workflows in spreadsheets. Bing Chat was chosen after comparing OpenAI playground, ChatGPT, Google Bard and Bing Chat, mainly because it draws on information from the Web. Researchers developed prompting strategies over several weeks.
Participants / data
Pilot with a convenience sample of two participants who are familiar with spreadsheets and use them for their work (n=2).
Duration
About 1 hour per pilot session. Participants got through 5-6 turns.
Analysis
Qualitative observations from the pilots, used to revise the protocol and the study script.
Findings
Feasible, and cheap. In two pilot sessions with people who use spreadsheets for work, the protocol worked as intended at very low cost, and the pilots led to improvements in the protocol and study script. #
“We believe this protocol is an improved adaptation of the traditional Wizard-of-Oz approach for studies of generative AI, since participants interact with a real AI model, but the implementation costs were extremely low.”
Few turns, many insights. Participants got through only 5-6 rounds of asking, reading and reflecting in an hour, yet these produced a wide range of qualitative insights. Most time went on reflecting on each response and deciding what to ask next. #
(§2.3, p. 6)
Longer answers worked better. In the pilots, Bing Chat's verbose 'creative' mode was no more likely to make things up than its 'precise' mode. Its longer answers also gave participants more to reflect on, so we chose it for the study. #
“We initially used "precise" mode because we believed that it would be the least likely to hallucinate misinformation”
Search habits shaped first prompts. Used to search engines, pilot participants started with short queries that could not convey their context, so we added guidance explaining that longer, conversational requests are possible. #
“This is unsurprising given Bing Chat's positioning within a more traditional search interface.”
Early hints of real barriers. The pilots also surfaced early insights, which we do not claim as findings. For example, one participant felt that having to check possibly made-up data by hand undercut the value of the AI system. #
“They said they felt it severely limited the benefits of Bing Chat.”
Questions this paper answers
How can I study what people want from generative AI when they don't use it at work yet?
One way is participatory prompting. The participant brings a real problem, and a researcher rewrites their requests using prepared prompting strategies and sends them to a working AI system. The participant then reflects on each response. A two-person pilot showed the approach is feasible. (See §2.4, p. 8; §2.3, p. 6.)
Does a user study with an AI chatbot need many exchanges to be useful?
Not necessarily. In pilot sessions where a researcher prompted a working chatbot on each person's behalf, people got through only 5-6 rounds of asking and reflecting. Yet these produced a wide range of qualitative insights. Most time went on reflecting on each response and deciding what to ask next. (See §2.3, p. 6.)
What trips people up when they first use a chat AI to help with data analysis?
Mostly, conveying context and trusting the output. In pilot sessions, people began with short, search-engine-style queries that could not convey their context, and one felt that checking possibly hallucinated data by hand severely limited the tool's benefits. These are early pilot observations, not established findings. (See §2.3, p. 7; §2.4, p. 8.)
Why not simply let participants prompt the AI themselves?
Because non-experts often cannot develop prompting strategies that get good results from a raw or generic model, so a study would mostly show prompting difficulties. Mediated prompting explores more of the workflow, at a cost. It cannot show where a participant's own prompting would have failed. (See §2.3, p. 7.)
Contributions and implications
Ask participants to think of a suitable problem before the session, to leave more time for the interactive turns. our recommendation is that if possible, the participant should be asked to think of a suitable problem ahead of the scheduled study session(§2.3, p. 6)
Researchers may nudge a stuck participant with a suggested query, but not in consecutive turns, to limit bias.
Find out what kind of output (for example images or tables) the participant expects and ask for it explicitly in the prompt, and refer back to earlier output to keep data consistent.
An open question is whether some of the researcher's work in the session could itself be supported by AI. raises the question of whether some activities carried out by the human experimenter could be supported with AI(§3, p. 8)
Limitations
Because the researcher helps craft prompts, the method does not show where a participant's own unsupported prompting would have failed. We ask participants how they would have changed prompts to partly offset this. results following this protocol may not give a clear understanding of where and when a user's unsupported prompting would have had issues(§3, p. 8)
The prompting strategies come from trial and error and researchers' intuitions, so they are subjective and hard to reproduce. our resulting prompts are subjective and difficult to reproduce(§2, p. 4)
Our environment and prompts emphasise spreadsheets, so participants' moves from prompting to spreadsheets may be less organic than without mediation.
Bing Chat could not regenerate a response to a specific earlier query, which made changing course within a time-limited session impractical.
The pilot had only two participants and the protocol was adapted live, so we do not claim its observations as usable findings. we do not claim these are usable findings due to currently running a pilot of n=2(§2.4, p. 8)
›
Concepts
Participatory prompting
Our method: the researcher rewrites the participant's requests using prepared prompting strategies, within an interview about the participant's own problem. In short: a researcher-mediated interaction between a study participant and a working generative AI system(§1, p. 2)
Critical integration
Sarkar's description of how generative AI shifts knowledge work (deciding where to use AI, how to instruct it and how to fold its output into the work), which we build on. from material production to critical integration(§1, p. 1)
Inner and outer loops of AI assistance
From Sarkar, building on double-loop learning. Inner loop: adding AI to steps of existing workflows. Outer loop: reshaping workflows, or creating new ones, around AI. (§1, p. 1)
Actually existing AI (AEAI)
A term from Siddarth and colleagues for AI as it really works today, as opposed to inflated public expectations of it. Our method aims to keep participants grounded in it. focus not on this collective mirage of what AI might be, but on "actually existing AI (AEAI)"(§1, p. 2)
How to cite
Advait Sarkar, Ian Drosos, Robert DeLine, Andy Gordon, Carina Negreanu, Sean Rintel, Jack Williams, and Ben Zorn. 2023. Participatory prompting: a user-centric research method for eliciting AI assistance opportunities in knowledge workflows. Proceedings of the 34th Annual Conference of the Psychology of Programming Interest Group (PPIG 2023).
This brief was prepared with AI assistance and published by Sean Rintel. It is not the authors' published text. Page numbers refer to this PDF.
BibTeX
@inproceedings{sarkar2023participatory,
author = {Sarkar, Advait and Drosos, Ian and DeLine, Robert and Gordon, Andy and Negreanu, Carina and Rintel, Sean and Williams, Jack and Zorn, Ben},
title = {Participatory prompting: a user-centric research method for eliciting AI assistance opportunities in knowledge workflows},
booktitle = {Proceedings of the 34th Annual Conference of the Psychology of Programming Interest Group (PPIG 2023)},
year = {2023},
month = {August},
url = {https://seanrintel.com/publications/participatory-prompting-a-user-centric-research-method-for-eliciting-ai-assistance-opportunities-in-knowledge-workflows.html},
}