Methodology and Participation for 2020 Diary Study of Microsoft Employees Experiences in Remote Meetings During COVID-19

Research brief

By Sean Rintel, Priscilla Wong, Advait Sarkar, and Abigail Sellen.

Takeaway: A sudden shift in work can be studied as it unfolds by combining guided diaries with weekly polls that react to emerging findings. Each entry can pair rotating prompts with the same ratings. Rolling recruitment captured people at different stages of working from home, though most diarists dropped out early.

Why this matters

What was known

The sudden shift to mandatory working from home in March 2020 created an unprecedented opportunity to study exclusively all-remote meetings over time. Diary studies have captured information workers' longitudinal experiences well. (§1.1, p. 4)

What this paper adds

This technical report records the study's logistics, data collection instruments (onboarding survey, guided diary, weekly polls) and participation demographics. (§1, p. 4)

Who it matters to

Designers and developers of meeting technologies, and researchers using the study's data. Understanding how meeting technologies are used in this broader context is crucial to future design and development. (§1.1, p. 4)

Research questions

• RQ1: What are the benefits, challenges, and opportunities of exclusively non-hybrid online meetings, conducted from home, in a global crisis?
• RQ2: How do the above evolve over time?
• RQ3: What development is required to support more effective, engaging, inclusive, and sustainable online meetings?

Method and data

Design

Longitudinal diary study with an onboarding survey, a guided diary of 24 entries in three cycles of eight topics, per-entry Likert pulse questions, and ten optional weekly polls.

Setting

Knowledge workers at a global technology company during mandatory working from home in the COVID-19 pandemic.

Participants / data

Rolling self-selected recruitment via an opt-in advertisement sent to 10,000 randomly selected employees and internal mailing lists. 715 participants filled in at least one diary entry.

Duration

Mid-April to mid-August 2020. Diary prompts approximately every three days for two months.

Analysis

This report covers logistics and participation (demographics, completion and poll response counts), not substantive findings.

Findings

• Broad reach across the company. Recruited through an opt-in invitation sent to randomly selected employees and to internal mailing lists, 715 employees filled in at least one diary entry, from almost all regions and operational groups of the company. #

  (§3.5.1, p. 21)

• Most stopped before the end. Participation dropped off and varied over the two months. Of the participants, 266 completed the first cycle of eight entries, 140 completed two cycles and 106 completed all three. Others filled in scattered entries without finishing a cycle. #

  “Given the size of this diary study and the situation in which it was conducted we expected a drop-off in participation”

• Polls drew a steady minority. The ten optional weekly polls, on topics such as parallel chat, turning video off and hybrid meetings, averaged 131 respondents each, ranging from 79 to 198. #

  (§4.2.1, p. 41)

• About half answered a poll. 357 people answered at least one weekly poll, which the report gives as just under half of diary participants. 13 of them filled in polls but no diary entries. #

  (§4.2.2, p. 42)

• A loyal core, until hybrid meetings came up. Later polls were answered mostly by a 'hardcore' group who took four or more polls, with few people taking part only once. The final poll, on the future of hybrid meetings, broke the pattern: #

  “The final poll on hybrid meetings shows a big spike of one-timers, suggesting that a lot of people were motivated to weigh in on this.”

• Start date didn't predict poll-taking. Whether participants joined early or late in the rolling recruitment was not correlated with filling in polls. #

  (§4.2.3, p. 45)

Questions this paper answers

How do you study a change in work while it is happening?

With a longitudinal diary. A study of remote meetings at a global technology company asked employees for a guided entry about every three days for two months. Entries cycled three times through eight meeting topics, so each could be revisited. The same eight ratings ended every entry. (See §3.5.1, p. 21; §3.5.1, p. 21.)

How many people keep up a long voluntary diary study?

Fewer as it goes on. When employees of a global technology company kept a voluntary diary of remote meetings during the pandemic, 715 made at least one entry but only 106 completed all 24. Many others contributed partially or irregularly. (See §3.5.1, p. 21; §3.5.1, p. 21.)

Do optional extra surveys reach the same people as the main study?

Partly. Just under half of the people keeping the diary answered at least one optional weekly poll. Most polls were answered mainly by a 'hardcore' group who took four or more, and 13 people answered polls without keeping a diary at all. (See §4.2.1, p. 41; §4.2.2, p. 42.)

What brings occasional participants back into a study?

Perhaps a topic they care about. In a diary study of remote meetings at a global technology company, later weekly polls drew mostly a loyal core. But the final poll, on the future of hybrid meetings, saw a spike of one-time respondents, apparently keen to weigh in. (See §4.2.3, p. 44.)

Quantitative results

Contributions and implications

• Rolling recruitment meant participants started at different points of mandatory working from home, which the authors considered valuable as a possible filter during analysis. (§1.6, p. 6)

Limitations

• Participants self-selected and were not obliged to continue, so participation dropped off. (§3.5.1, p. 21)
• Because of rolling recruitment, not all participants could answer every poll. Having a slightly different subset of participants responding to the different poll questions was considered a reasonable trade-off (§1.6, p. 6)
• Some groups, including human resources, legal and very senior employees, could not take part.

Figures

How to cite

Sean Rintel, Priscilla Wong, Advait Sarkar, and Abigail Sellen. 2020. Methodology and Participation for 2020 Diary Study of Microsoft Employees Experiences in Remote Meetings During COVID-19. Microsoft Research.

This brief was prepared with AI assistance and published by Sean Rintel. It is not the authors' published text. Page numbers refer to this PDF.