Ecological Validity and the Evaluation of Avatar Facial Animation Noise

Authors: Marta Wilczkowiak, Ken Jakubzak, James Clemoes, Cornelia Treptow, Michaela Porubanova, Kerry Read, Daniel McDuff, Marina Kuznetsova, Sean Rintel, Mar Gonzalez-Franco
Venue: Workshop Series on Animation in Virtual and Augmented Environments (ANIVAE 2024)
Author's copy (PDF)

AThe study

Two experiments in augmented reality with 56 participants in Washington, USA, each wearing a HoloLens 2 headset. The same people judged facial glitches on cartoon avatars, in short clips and in a full conversation.

  1. 56 participantsRecruited in Washington, USA, by a research recruitment firm.
  2. Animated avatarsProfessional actors' motion capture animated two cartoon avatars in an architect and client conversation about remodelling a house.
  3. Clip taskParticipants used sliders to set acceptable glitch levels on 20-second clips, one glitch type at a time and then combined.
  4. Conversation taskParticipants watched full conversations and flagged expressions as appropriate or not, while glitches rose over four levels without warning.
  5. AnalysisAcceptability thresholds, statistical tests of response rates, and a comparison of high and low scorers on an empathy questionnaire.

BWhat the study found

Which glitches matterglitch types and frame rate, in both tasks

  • In both tasks, jittering lips hurt acceptability more than any other glitch.With all glitches combined, almost all participants chose mouth jitter as one to remove. A mouth that would not fully close came second.
  • Participants generally treated 15 frames per second as good enough.That frame rate made up 87.5% of responses on isolated clips, and 90% when watching the conversation.

Clips versus conversationhow the test setting changed tolerance

  • Watching a full conversation, participants tolerated much more noise than when tuning clips.In the conversation, appropriate responses far outnumbered inappropriate ones, a highly reliable difference, p < 0.0001. In the clip task, combining glitches lowered tolerance further.
  • The authors cannot rule out that task order lowered the clip thresholds.The same participants did both tasks, so the order of tasks may have shaped the noise levels they accepted.

Who was watchingempathy and gender, in the conversation task

  • In the conversation, lower-empathy participants flagged more lip-jitter expressions as inappropriate.Empathy was measured by questionnaire, and the difference was reliable, p < 0.05. Empathy made no significant difference to thresholds set on isolated clips.
  • Responses varied with the characters' gender and whether the talk ended in agreement.Response rates differed by video version, p = 0.03. A female architect who agreed drew the most appropriate responses, and one who disagreed the fewest.
  • The authors suggest possible bias against women in professional roles shaped these judgements.Participants only watched the conversation and did not take part in it. No zero-glitch baseline was collected, because zero noise is not achievable in practice.

CKey takeaway

Testing for ecological validity can change what avatar animation seems acceptable. People may tolerate more facial glitches in conversation than in isolated clips, while empathy and social bias shape judgements.

Made for this site with AI assistance from the verified research brief. It is not the paper's own figure.

Abstract

Facial animation noise levels affect the acceptance of avatars in communication systems. However, there is no standard for evaluation, especially with regard to ecological validity. We investigate low and high ecological validity on two within-subjects experiments conducted in Augmented Reality on a Hololens2. We simulated facial-expression noise introduced on stylized cartoon avatars, and found that in the high ecological validity experiment, subjects were less sensitive to noise parameters, but their judgement was more influenced by empathy scores and gender biases. This highlights the importance of considering both technical parameters and user experience when designing communication systems. We make some general recommendations for evaluating issues of avatar acceptance given the trade-offs between the approaches, and propose the ‘Triple C’ factors of Context, Culture and Character as an important set of ecological factors to consider.

Research brief

By Marta Wilczkowiak, Ken Jakubzak, James Clemoes, Cornelia Treptow, Michaela Porubanova, Kerry Read, Daniel McDuff, Marina Kuznetsova, Sean Rintel, and Mar Gonzalez-Franco.

Takeaway: Ecological validity changes avatar verdicts. Viewers watching a realistic conversation seemed to forgive more facial animation glitches than when judging isolated clips. But in conversation their judgements were swayed by context, culture and character, such as empathy and gender bias.

Why this matters

What was known
Research on how animation noise affects avatars is sparse, tends to focus on one's own avatar, is often low in ecological validity, and gives no direct evidence on acceptable noise thresholds. Parameters assumed to be universal, such as animation noise thresholds, seem to change from study to study, reducing comparability, repeatability, and extrapolation of work (§1, p. 1)
What this paper adds
Instead of a single study, we pair a classic threshold-tuning experiment with a realistic conversation-watching one, and derive recommendations for when each kind of test can be trusted. Running high and low ecologically valid conditions on the same cohort, on the same device, with the same assets, brings to light how experimental design impacts the results of a parameter evaluation. (§5, p. 5)
Who it matters to
Researchers and developers of avatars for communication systems who need to set technical requirements and evaluate avatar acceptance. the level of ecological validity changes the usefulness of these results for researchers and developers of avatars (§5, p. 6)

Research questions

Method and data

Design
Two within-subjects experiments in augmented reality on a HoloLens2. Experiment 1 (low ecological validity): participants used sliders to set acceptable noise levels on 20-second clips, for each noise type in isolation and then in combination, and chose noises to remove. Experiment 2 (high ecological validity): participants watched full conversations and pressed thumbs-up or thumbs-down for appropriate or inappropriate facial expressions while noise was introduced at 4 levels without their knowledge.
Setting
Motion-captured professional actors animated two stylized cartoon avatars in an architect and client house remodelling conversation. 4 versions varied the gender of the architect and whether the conversation ended in agreement or disagreement.
Participants / data
N=56 participants recruited in Washington, USA by a research recruitment firm.
Analysis
Cumulative acceptability thresholds (90% of participants), counts of noise priorities, response rates analysed with ANOVA, and comparison of top and bottom 25% scorers on the Toronto Empathy questionnaire.

Findings

Questions this paper answers

Which avatar facial glitch bothers people most?
Jittering lips. When people tuned glitches on cartoon avatars in augmented reality, almost all included mouth jitter among the three glitches they chose to remove. It also hurt acceptability most, whether people tuned noise on clips or watched a full conversation. (See §4.1.1, p. 4.)
Does the way you test avatars change the results?
Yes. Short isolated clips make people stricter about glitches than a realistic conversation does, while a realistic conversation lets personality and social attitudes, such as empathy and gender, shape judgements. Thresholds from one kind of test should not be assumed to hold in the other. (See §4.1, p. 4; §5, p. 6; §5, p. 6.)
What frame rate is good enough for avatar faces in augmented reality?
Animating avatar faces at 15 frames per second was generally treated as good enough in augmented reality. That held in 87.5% of people's responses when judging isolated clips, and 90% when watching a full conversation. (See §4.2, p. 4.)
Do avatar glitches bother people while they follow a conversation?
Less than when they are tuning noise directly. When people were focused on a social task and not told about the noise, they judged most expressions appropriate. They seemed to tolerate much more noise than when tuning it, although more noise still meant a worse experience. (See §4.1, p. 4.)

Quantitative results

MeasureResultWhere
ParticipantsN=56§3.3, p. 2
Response rate, 'appropriate' vs 'inappropriate' facial expressions (high ecological validity)F(1) = 86.7 , p < 0.0001§4.2, p. 4
Response rate by video condition (gender of characters and outcome)F(3)=3, p=0.03§4.2.1, p. 4
Frame rate treated as good enough15 fps: 87.5% of responses (isolation), 90% (social context)§4.2, p. 4
Lower empathy and 'inappropriate' responses with mouth jitterp < 0.05§4.3, p. 5

Contributions and implications

Limitations

›

Concepts

Ecological validity
Our definition, which also says why it matters: ecological validity, which is the extent to which an experiment reproduces the situation and environment in which the technology will be used, and hence the extent to which the research findings generalize (§1, p. 1)
Facial animation noise
The degradations we tested: We explore different types and levels of degradation, including lip and brow jitter, full closure of the eyes and mouth, facial asymmetries, and drops in frame rate. (§1, p. 1)
Triple-C factors
Context, Culture and Character: three broad ecological factors to take into account when evaluating avatars. high validity experimentation requires considering how the Triple-C factors matter to achieving balanced, generalizable, and replicable results that evaluate and reduce the effects of bias (§5, p. 6)

Figures

Four panels: actors in motion-capture suits, two stylised avatar heads, a sketch of a HoloLens wearer watching two avatars talk and answering on a tablet, and an avatar face labelled with brow, eye and mouth noise types.
Figure 1: A. Mo-cap of a social interaction scenario for the experiment. B. The mo-cap was remapped to high-end stylized avatars. C. Participants experienced the social interaction wearing a HoloLens2 and responded to questions on a tablet app. D. Facial features of the avatars were degraded with different types and severity of animation noises. From the paper, Figure 1, p. 1.
Left: bar chart of noises turned off, mouth jitter most often, then mouth closing, asymmetry and brow jitter, eye closing least. Right: mean response rates by video type, with 'appropriate' higher than 'inappropriate' in all four.
Figure 5: (Top) Participants selected the following noises as priority for them to remove when they were all in combination. (Bottom) Engagement, measured as responses count depending on whether participants found the facial expressions 'appropriate' or 'inappropriate' for the different recorded conditions. From the paper, Fig. 5, p. 5.

How to cite

Marta Wilczkowiak, Ken Jakubzak, James Clemoes, Cornelia Treptow, Michaela Porubanova, Kerry Read, Daniel McDuff, Marina Kuznetsova, Sean Rintel, and Mar Gonzalez-Franco. 2024. Ecological Validity and the Evaluation of Avatar Facial Animation Noise. Workshop Series on Animation in Virtual and Augmented Environments (ANIVAE 2024). https://doi.org/10.1109/vrw62533.2024.00019

This brief was prepared with AI assistance and published by Sean Rintel. It is not the authors' published text. Page numbers refer to this PDF.

BibTeX

@inproceedings{wilczkowiaksheher2024ecological,
author = {Wilczkowiak, Marta and Jakubzak, Ken and Clemoes,  James and Treptow, Cornelia and Porubanova, Michaela and Read, Kerry and McDuff, Daniel and Kuznetsova, Marina and Rintel, Sean and Gonzalez-Franco, Mar},
title = {Ecological Validity and the Evaluation of Avatar Facial Animation Noise},
organization = {IEEE Virtual Reality 2024},
booktitle = {Workshop Series on Animation in Virtual and Augmented Environments (ANIVAE 2024)},
year = {2024},
month = {March},
url = {https://doi.org/10.1109/vrw62533.2024.00019},
doi = {10.1109/vrw62533.2024.00019},
}