Research brief
Takeaway: Before and during meetings, generative AI adaptive interfaces could support meeting intentionality by reducing effort. They could infer goals and implicit phases from the invitation, focus discussion where attendees disagree, and bring up apps and files, in 2D or VR, as talk changes phase.
Why this matters
- What was known
- Productive meetings take effort before and during the meeting, but meeting systems do little to reduce it, and the videoconferencing stage has barely changed since the 1990s. Commercial AI meeting features cover only part of what generative AI could do.
The COVID-19 pandemic showed that constant exposure to this stage is not optimized for meeting effectiveness and is also fatiguing
(§2, p. 2) - What this paper adds
- Rather than adding AI features to the standard grid of faces beside one shared screen, we treat the whole meeting workspace (flat or virtual) as something AI can reshape around goals. We also share the prompts behind our probes so others can build on them.
provocative designs for GenAI-driven meeting systems that adapt the working space in line with meeting goals and contextual activities
(§1, p. 2) - Who it matters to
- Designers and researchers of meeting and videoconferencing systems, including VR meeting platforms, who want to support meeting intentionality.
We provide example prompts that could facilitate design research on adaptive GenAI-driven meeting interfaces.
(§1, p. 2)
Research question
- How could generative AI reduce the effort of planning and running meetings by foregrounding meeting goals and adapting the meeting interface to each meeting phase, in both 2D videoconferencing and virtual reality? (Stated aim. The paper has no formal research questions.)
Method and data
- Design
- Design and implementation of two technology probes, shaped by three design approaches: collective feedback on objectives, clarifying needs and resources for each phase, and human-on-the-loop control.
- Prototype
- Two prototypes sharing one generative AI back end (GPT-3.5, built in Unity): CoExplorer2D, a videoconferencing client for Windows computers with LiveKit video, and CoExplorerVR, a virtual reality meeting room on Meta Quest 2. Before the meeting, the system reads the organiser's invitation and shows each attendee the meeting's aim, its likely phases (with expected durations and activities) and a generated Meeting Focus Tool, here a feature-ranking aid. It then uses everyone's answers to narrow the phases to points of disagreement. During the meeting, speech is transcribed with the Microsoft Azure Speech API and each utterance is matched to a phase. When the phase changes, the system asks everyone whether to halt the switch. If no one objects, it moves to a tiled layout of the apps and files planned for that phase, for example slides and a shared notepad, then a spreadsheet and calculator. CoExplorerVR instead places windows around a virtual room, taking account of who can see each spot and what nearby objects mean. Nothing was compared. The prototypes were demonstrated on a fictional scenario.
Figure 2: CoExplorer2D listens for meeting attendees transitioning between phases, then notifies the attendees that a change has been detected and modifies the layout to accommodate the new phase. From the paper, Figure 2, p. 4. - Setting
- A fictional decision-making meeting of a cross-functional product team choosing features for the "Strata Headphones 3".
- Participants / data
- No user study. The probes were demonstrated on the scenario.
- Analysis
- Design reflection on what the probes could and could not do. Formal evaluation is explicitly out of scope and planned as future work.
Findings
AI can draft the plan. Given only a brief meeting invitation, the system generated the meeting's aim, its likely phases and a tool for attendees to state their preferences. For a consistent experience across users we then used a pre-generated version of the tool, with manual fixes to confusing details such as button colours. #
“During our experiments, we observed that GenAI is indeed proficient at devising a tool for this need.”
Acting unprompted hides the lag. In the 2D prototype, the system recognises a phase change from the transcribed talk and, unless someone aborts, switches to that phase's layout. Running each utterance through a language model adds a delay, which matters less than it might: #
“since CoExplorer strives to proactively take action, rather than react to user prompts, this latency was not especially noticeable and not directly relevant to the tasks”
The model could not lay out screens. Asking the language model to size and position windows freely did not work with the model we had, so the 2D prototype tiles windows to fill the screen instead. #
“While GPT-4 was adept at creating well-fitted window layouts, GPT-3.5 - the version available to us during prototyping - fell short in this capacity.”
In virtual reality, place has meaning. Moving from a flat screen to a virtual room changes the layout problem. Where a window appears must suit who can see it (a panel on the speaker's table is private, a large display behind them is public). It must also suit what nearby objects mean, and whether it can easily be pointed at. #
“CoExplorerVR expands this functionality into the third dimension, allocating virtual windows to appear around a room in meaningful spatial arrangements”
Quick to build, hard to trust. Building with generative AI involves a trade-off. Development is fast, but the AI's outputs must be validated, because it tends to propose changes when none are needed. Language models also run slower than speech-to-text. #
“GenAI drastically cuts the time and data requirements for developing systems like CoExplorer by eliminating the usual steps of data collection, model training, and testing”
Questions this paper answers
- Can AI work out what a meeting is for from its invitation?
- It can draft a plan before the meeting, supporting meeting intentionality. From a brief invitation, the CoExplorer prototype's AI proposed the meeting's aim and likely phases. It also built a tool for attendees to state preferences, narrowing discussion to where views diverge. We have not formally evaluated this. (See §3.3, p. 5.)
- Should an AI meeting assistant ask before it acts?
- Not every time, we argue. People in meetings have little spare attention for prompts. So the CoExplorer prototypes act on their own and only give everyone a chance to abort a change (a human-on-the-loop approach). That demands more trust than asking first, so the AI's judgements must be reliable. (See §3.3, p. 5; §4, p. 6.)
- How is arranging content in virtual reality different from arranging it on a screen?
- A screen is one flat surface shared by all. A room gives places meaning. Where content appears decides who can see it, and nearby objects can carry information about people. Placement can make a window hard to select, so layout must follow the room as well as the task. (See §3.4, p. 6.)
- Are today's AI models good enough for live meeting support?
- Partly. In the CoExplorer prototypes, GPT-3.5 could draft meeting plans and match talk to meeting phases. But it could not lay out windows well, tended to over-react, and ran slower than speech-to-text. GPT-4 already produced well-fitted layouts, and we expect latency to fall as models improve. (See §3.3, p. 5; §3.3, p. 5; §3.3, p. 5; §4, p. 6.)
Contributions and implications
- Drawing on meeting-practice guidance, we steer live discussion to the issues where attendees disagree, since live meetings suit dynamic discussion and areas of agreement can be handled asynchronously or later.
- Validating generative AI outputs is critical for future development of such systems.
validating the outputs fromGenAI remains critical for future development
(§4, p. 6) - A future system could combine AI-detected action items with adaptive windowing, placing action items under the video of the relevant person.
- We plan to evaluate CoExplorer2D as a technology probe, and then test effects on meeting effectiveness with surveys and telemetry.
Limitations
- We did not formally evaluate the system's capabilities or their effect on meeting effectiveness.
More direct evaluation on the effect these capabilities have on meeting effectiveness remains for future work.
(§2, p. 3) - Due to time limitations we did not complete the full meeting scenario in CoExplorer2D.
Due to time limitations, we did not complete the full meeting scenario in CoExplorer2D.
(§3.3, p. 5) - Generative AI tends to over-respond, producing changes where none are needed.
GenAI tends to generate non-null, [extensive-change] responses due to its sensitivity, leading to inappropriate output.
(§4, p. 6) - LiveKit did not support Unity clients, so we used workarounds, which is a problem for low-latency real-time communication.
›Concepts
- Implicit meeting phases
- The stages a meeting actually moves through, which may match, span or subdivide agenda items, and which each need different files and apps.
while explicit items on the agenda direct the discourse, the meeting actually often transitions through implicit phases
(§3.1, p. 3) - Human-on-the-Loop (HOTL)
- The control model we chose, because meeting participants have little capacity for prompts outside the conversation itself:
HOTL characterizes human-machine interactions in which the automated system mainly allows humans to abort the machine's decisions
(§3.1, p. 3) - Meeting Focus Tool
- A tool generated before the meeting, whose results the system uses to refocus objectives and phases on areas of disagreement. In our scenario, it is
a Meeting Focus Tool that lets meeting participants assess the implications of including specific features in a product from their role perspective and voice their preferences on key features
(§3.3, p. 5) - Spatial context
- The paper's definition, which CoExplorerVR uses when placing windows around a virtual room:
Spatial context refers to the significance attributed to specific locations within the 3D scene
(§3.4, p. 6)
Figures
How to cite
Gun Woo (Warren) Park, Payod Panda, Lev Tankelevitch, and Sean Rintel. 2024. CoExplorer: Generative AI Powered 2D and 3D Adaptive Interfaces to Support Intentionality in Video Meetings. CHI EA 2024. https://doi.org/10.1145/3613905.3650797
This brief was prepared with AI assistance and published by Sean Rintel. It is not the authors' published text. Page numbers refer to this PDF.