Meeting in the Middle: The Interpretation Gap Between People and Machines

Research brief

By Anastasia Kuzminykh and Sean Rintel.

Takeaway: Machines describing meetings to visually impaired colleagues must bridge an interpretation gap. A cue people call simple and objective, like someone 'listening to both of them', is many pose and gaze labels added up. That is because social attention is selective sense-making, not just what can be seen.

Why this matters

What was known

Machine vision can already label basic visual information such as pose and gaze. Research has yet to systematically address the significant gap between levels of granularity and interpretation of machine labels and of human comprehension. (Abstract, p. 1)

What this paper adds

Preliminary results from interviews with visually impaired employees and a narration study, yielding a structural model of how people describe social attention in meetings. the output viewed by human users as elemental and objective is, in fact, often highly interpretive and requires several levels of algorithmic accumulation of basic recognized events (§INTRODUCTION, p. 2)

Who it matters to

AI and HCI researchers building systems that translate machine recognition into output for people, such as voice assistants and assistive technology for visually impaired meeting participants. we strongly encourage specialists from both fields to start considering the degrees of minimal required interpretability in AI systems (§CONCLUSION, p. 4)

Research questions

• How do visually impaired people experience social dynamics in work meetings, and what additional information would they find useful?
• How do people narrate the visual social attention cues of a meeting for a visually impaired person, and what structure do those descriptions have?

Method and data

Design

Two preliminary studies: semi-structured interviews, then a quasi-experimental narration study.

Setting

Work meetings. In Study II, participants narrated muted video recordings of meetings to an imaginary visually impaired person, focusing on social dynamics.

Participants / data

Study I: ten visually impaired employees from diverse types of organizations. Study II: 15 participants, each shown six ~2 min videos of work meetings with 4 to 6 meeting participants.

Analysis

Qualitative analysis of interview accounts and of narrated descriptions, used to build a structural model of social attention description.

Findings

• Blind employees want body language. Ten visually impaired employees from diverse organisations wanted to know what other people's body language and attention conveyed in meetings, much of which rests on signals that machine vision could potentially recognise. #

  “It would be really helpful to know what people's body language was saying. So you know, bored, or engaged, or distrusted, or stressed.”

• Raw cues, not verdicts. They saw the information they wanted as objective and elemental, and preferred to receive it unprocessed so that they could interpret it themselves. #

  “And I think "someone is taking notes", "someone is reading notes", "someone is looking at the window" is fine.”

• Structured, but interpretive. When 15 people narrated muted meeting videos for an imagined visually impaired listener, their descriptions were structured enough for us to model, but their elements were still accumulative and interpretive. #

  (§Study II: How people narrate the social attention cues of a meeting, p. 3)

• Groups take more work to recognise. Attention to one person might be recognised from a single video frame, but attention to a group, as in this narration, has to be assembled from several individual acts of attention. #

  “The lady is sitting and listening to both of them”

• Verbs that go beyond what is seen. Narrators named acts of attention with many interpretive verbs, which go well beyond the visual information they were derived from. #

  “"to look", "to watch", "to listen", "to pay attention", "to direct attention", "to be attentive to", "to be engaged", "to talk to", "to have a conversation with"”

Questions this paper answers

What do visually impaired people want to know about what's happening in a meeting?

Body language and attention, such as whether others are engaged, attentive, confused or supportive. Visually impaired employees interviewed about meetings saw this as objective information. They preferred to receive it unprocessed, so that they could interpret it themselves. (See §Study I: How visually-impaired people experience social dynamics in meetings, pp. 2–3; §Study I: How visually-impaired people experience social dynamics in meetings, pp. 2–3.)

Can machine vision describe social dynamics in meetings to blind users?

Many relevant signals, such as gestures and body pose, are potentially recognisable by machine vision. But the descriptions people expect are interpretive and accumulative, so a system would need to combine basic labels into higher-level events before they are meaningful to a listener. (See §Study I: How visually-impaired people experience social dynamics in meetings, pp. 2–3; §Study II: How people narrate the social attention cues of a meeting, p. 3; §Study II: How people narrate the social attention cues of a meeting, p. 3.)

Is a description like 'she is listening' objective?

Not to a machine. People treat it as plain observation, but it interprets gaze and posture as an act of attention, and attention to a group must be pieced together from several individual acts. Such summaries spare a listener overload, so systems need rules for building them. (See §Study II: How people narrate the social attention cues of a meeting, p. 3; §Study II: How people narrate the social attention cues of a meeting, p. 3; §Study II: How people narrate the social attention cues of a meeting, p. 3.)

Contributions and implications

• Research should explore how people build accumulative descriptions and turn them into machine recognition rules, since these reduce the listener's cognitive overload. it is important that research continues to explore further the mechanisms of accumulative descriptions and formulate them as machine recognition rules (§Study II: How people narrate the social attention cues of a meeting, p. 3)
• AI and HCI researchers should work together, with HCI articulating the requirements for applying machine recognition in human-oriented technology, including the minimal interpretability systems need. we strongly encourage specialists from both fields to start considering the degrees of minimal required interpretability in AI systems to advance the possibilities of systems applications (§CONCLUSION, p. 4)

Limitations

• The results are preliminary, and understanding of the right degree of interpretability is at a very early stage. We are still at the very early stages of understanding how to identify the appropriate degree of interpretability (§CONCLUSION, p. 4)

Figures

How to cite

Anastasia Kuzminykh and Sean Rintel. 2019. Meeting in the Middle: The Interpretation Gap Between People and Machines. CHI 2019.

This brief was prepared with AI assistance and published by Sean Rintel. It is not the authors' published text. Page numbers refer to this PDF.