Research brief
Takeaway: Headphones can become context-aware input devices by augmenting social signals people already make. Lifting an earcup to hear a nearby colleague, for example, can mute the video call.
Why this matters
- What was known
- Headphone interaction has focused on media and audio control, with buttons, taps and touch sensors.
most of these implementations are decontextualized interactions where the gestures and ear interactions do not leverage the context within which the user performs the task
(§2.1, p. 2) - What this paper adds
- A design space for headphone interaction across two dimensions (type of input and context of the action), and an annotated portfolio of working prototypes for workplace and gaming scenarios.
The insight that each interaction could use multiple types of input and sense multiple contexts at the same time opens up a vast design space for future exploration around headphones
(§1, p. 2) - Who it matters to
- Interaction designers and ubiquitous computing researchers, and designers of other wearables.
We hope this work will encourage designers to re-envision headphones as a more general input/output wearable.
(§7, p. 11)
Research questions
- What is the potential of headphones as a general-purpose input device, for both foreground motion gestures and background sensing of user activity?
- How can headphone-situated interactions be organised into a design space that motivates future systems?
Method and data
- Design
- Research through Design with a material-centric practice, evaluated by demonstration through an annotated portfolio of functional prototypes.
- Prototype
- Off-the-shelf wireless headphones augmented with sensors. First came a motion sensor (IMU) with buttons and rotary knobs on a hot-swappable magnetic mount. Later a LiDAR depth sensor was taped above the earcup, with a MobileNet v2 neural network trained to recognise hand gestures near the ear. The wearer behaves naturally or makes a familiar gesture: turning away from a call, looking at another screen, lifting or cupping an earcup, leaning. In response, connected applications blur the video and mute the microphone, switch the shared screen or move audio and game-controller input to the device being looked at, raise the volume, animate an avatar for a camera-off participant, or steer a game character. Deliberately built from low-fidelity materials (cardboard, breadboards, tape) and evaluated by demonstration, not compared with other systems.
Figure 1: We explore interaction with headphones as a wearable sensor-enhanced input peripheral-as opposed to an output device with audio control functionality. Rooting user gestures within their context give rise to several application possibilities. From the paper, Figure 1, p. 1. - Setting
- Use of headphones in the workplace and for gaming. Prototypes were built in a lab during COVID-19 social distancing.
- Participants / data
- Six of the authors recorded themselves wearing wireless headphones in everyday settings (each at least 20 minutes, covering multi-device use, a real-world interruption and changing tasks) to seed the designs.
- Analysis
- Recurring gestures from the recordings shaped iterative prototypes. Annotating the portfolio of prototypes with the interaction qualities they share led to the design space.
Findings
Lifting an earcup can mute a call. In video recordings that six of us made of ourselves wearing headphones in everyday settings, people tended to lift the headphones off when someone approached. Our prototype turns that familiar gesture into muting both sound and microphone. #
“Through this gesture, the interaction design minimizes the need to search for a button and simultaneously signals conversation acceptance to the nearby person in a socially acceptable and familiar way.”
Turning away blurs your video. In our video-call prototype, a sustained head turn away from the screen, sensed by the headphones, blurs the user's video and mutes their microphone while they talk to someone in the room. Looking back restores both, with no button pressed. #
“the blurred video notifies other attendees that the user is temporarily away from the call”
Where you look, input follows. Because people tend to look at the device they are using, the prototype treats head orientation as a sign of attention. During a presentation it switches the shared screen to whichever display or tablet the user looks at. In gaming it can move audio or controller input to another device. #
“the user may transfer the game controller input to the other non-gaming device in order to answer a phone call or change the music audio in the room without moving their hands off the game controller”
Taking part without a camera. For a participant with their camera off, the prototype animates an avatar from the headphones' head-orientation data and the user's voice (for lip movement). In games, leaning the head can swerve a car or peek around a corner. #
“Users could join in an audio-only mode but participate more equally with other visually-represented participants with an avatar representation”
Inputs and contexts combine. When we placed our eleven prototypes on the design space, neither the input types nor the contexts turned out to be mutually exclusive. A single interaction can draw on several of each. #
“controlling an avatar might use both the body context (to detect gestures such as nods) and the app context (to detect being in an application with avatars), as well as multiple kinds of inputs”
Fewer gestures to learn. We argue that reusing behaviours people already perform, rather than adding more buttons, keeps an eyes-free headphone interface simple while expanding what it can control. #
“This shifts complexity from the user to the system, co-opting a natural user behavior and giving it a dual-purpose, context-dependent digital meaning.”
Questions this paper answers
- Why use headphones as an input device?
- They travel with the user from device to device, sit on a part of the body rich in social meaning, and are socially accepted. A few sensors can then capture head turns, posture and hand gestures near the ear, cues that already signal attention or intent to other people. (See §3.2, p. 5; §3.2, pp. 6–7; §3.2, p. 7.)
- Is a gesture either intentional or not?
- Not always. Looking away from a video call may be spontaneous, not meant as a command to the headphones, yet meaningful socially. Treating intentionality as a continuum lets designers act on such behaviour while still guarding against accidental triggers. (See §3.2, p. 5.)
- Can a device do more without becoming harder to use?
- Potentially, by reading context rather than adding buttons, which add things to learn and control only the device. Sensing the application, the body and the surroundings lets the same few gestures mean different things. One button could then answer a phone or switch a light depending on where you look. (See §4.3, pp. 9–10; §1, p. 2.)
Contributions and implications
- The design space of input type and context may extend to other wearables.
the considerations of the input gesture and the context of use regarding the application, user's body, and the environment are also extensible to other kinds of wearables
(§6, p. 11) - Implementations need safeguards against accidental triggers, such as time thresholds.
looking away from a screen for a short time should not mute the user in a video call, nor change the screen share
(§6, p. 10) - Richer sensors bring ethical questions for everyday devices.
designers and engineers are strongly advised to consider the ethical implications of using some of these sensors in devices meant for everyday use
(§6, p. 11)
Limitations
- Prototypes were built in a lab during COVID-19 social distancing, so robust prototypes and ecologically valid experiments were not possible.
developing robust prototypes and conducting ecologically-valid experimentation were not possible
(§6, p. 11) - The gestures are a subset drawn from the researchers' own cultures and may not suit other cultures or scenarios.
These gestures may be unnatural in different cultures and scenarios, and their implementation in actual software should be done with care.
(§6, p. 10)
›Concepts
- Type of input gesture
- The first of the design space's two dimensions. The abstract names both:
(1) type of input (touch, mid-air, or head orientation); and (2) the context of user action (application, body, or environment)
(Abstract, p. 1) - Context of input gesture
- The second dimension: whether a gesture depends on the application, the user's body, or the physical and social environment. Its baseline category:
Context free gestures produce a similar result regardless of the active application, what the user is doing, or the user's social or physical environment.
(§4.2, p. 8) - Foreground and background interaction
- Building on earlier work on foreground and background attention, applied to headphones:
the foreground is the user's direct interaction with the device and the background is their interaction with the environment
(§2.2, p. 3) - Annotated portfolio
- A Research through Design method in which a set of prototypes is described together, with annotations of the interaction qualities they share.
Within the RtD method of annotated portfolios, an annotation is any textual description that accompanies a design artefact
(§3.2, p. 5) - Intentionality as a continuum
- Our argument against treating every action as simply intentional or not:
a more nuanced perspective of intentionality as a continuum allows for a desired outcome which could be the result of an unintended action
(§5.1, p. 10)
Figures
How to cite
Payod Panda, Molly Jane Nicholas, David Nguyen, Eyal Ofek, Michel Pahud, Sean Rintel, Mar Gonzalez Franco, Ken Hinckley, and Jaron Lanier. 2023. Beyond Audio: Towards a Design Space of Headphones as a Site for Interaction and Sensing. DIS'23: ACM Designing Interactive Systems Conference 2023. https://doi.org/10.1145/3563657.3596022
This brief was prepared with AI assistance and published by Sean Rintel. It is not the authors' published text. Page numbers refer to this PDF.