Research brief
Takeaway: Secondary-school students learning from history texts understood and remembered more three days later when they took notes, alone or alongside an AI chatbot, than when using the chatbot alone. The chatbot possibly did the summarising work for them, yet students enjoyed it more and found it easier.
Why this matters
- What was known
- Studies of LLMs in education mostly measure task performance, sometimes with the LLM available during assessment, and mostly in higher education, so little is known about effects on genuine learning. (§1, pp. 1–2)
- What this paper adds
- A large, pre-registered randomised experiment in secondary schools comparing LLM use with note-taking and with both combined, measuring comprehension and retention days later, plus engagement and the kinds of prompts students wrote. (§1, p. 2)
- Who it matters to
- Educators, schools and policymakers deciding how to bring AI tools into classrooms, and designers of LLM-based learning tools. (§2.4, p. 5)
Research questions
- 1) How does using an LLM for comprehension-focused reading affect students' text comprehension and retention compared to the more traditional learning activity of note-taking as well as compared to using an LLM alongside note-taking?
- 2) How does using an LLM for comprehension-focused reading affect students' emotional, cognitive, and behavioural engagement compared to the more traditional learning activity of note-taking as well as compared to using an LLM alongside note-taking?
- 3) What types of prompts do students use when interacting with the LLM?
Method and data
- Design
- Pre-registered randomised controlled experiment with within- and between-participant design elements. Each student studied two history passages, each with a different activity: Group 1 used LLM and Notes, and Group 2 used LLM and LLM + Notes, with LLM as the common reference. A test session followed three days later.
- Setting
- Seven secondary schools in England, during regular school hours, between March and July 2024. A custom chatbot based on GPT-3.5 turbo (maximum 20 prompts) and a notepad in a web app. Passages on Apartheid in South Africa and the Cuban Missile Crisis.
- Participants / data
- 405 Year 10 students aged 14-15. 344 retained for analysis (Group 1: 184, and Group 2: 160).
- Duration
- Two sessions three days apart: a learning session (10 to 15 minutes per passage) and a test session.
- Analysis
- Linear mixed-effects regressions for literal retention, comprehension and free recall, with students as a random effect and multiple imputation for missing covariates. Paired permutation tests for free recall. Paired t-tests with Bonferroni correction for engagement. GPT-4-assisted, human-checked qualitative coding of preference explanations and of 4929 prompts.
Findings
Notes beat the chatbot. Three days after studying a history passage, 14-15-year-old students understood and remembered it better if they had taken notes, alone or alongside an AI chatbot, than if they had used the chatbot alone. #
(§4.2, p. 12)Notes alone did best. Note-taking alone also improved free recall, which notes alongside the chatbot did not. In an indirect, exploratory comparison, its advantage over the chatbot alone tended to be larger than that of the combined activity. #
(§4.2, p. 12)Enjoyed more, learned less. Compared with note-taking, students enjoyed the chatbot more, found it easier and rated it more helpful for understanding and learning the text, even though they scored lower after using it. #
“Having an LLM do some of the work of summarising a passage or explaining a concept may feel more enjoyable and efficient but can reduce the cognitive engagement necessary for deep comprehension and long-term retention”
Less effort, and copied notes. With the chatbot alone, students reported less effort and spent slightly less time on the task, although time did not predict their scores. With chatbot and notepad together, about a quarter copied most of their notes straight from the chatbot's answers. #
“This indicates that students' own note-taking efforts were reduced, and that the effectiveness of note-taking may be diminished.”
The combination was preferred. Students who tried both preferred the chatbot plus notes over the chatbot alone by a wide margin, and the combination did not reduce their enjoyment. #
“Rather than viewing these as competing alternatives, we could think of them as complementary that, when thoughtfully integrated, may enhance learning outcomes in ways that neither can achieve alone.”
Curious questions, some off-topic. Students' prompts fell into four kinds, most often asking for more information and deeper understanding. Some explored ideas beyond the text, and about one in ten prompts were off-topic. #
“in a passage on the Cuban Missile Crisis, another student asked, "Why was America afraid of communism?"”
Questions this paper answers
- Do students learn more from a text with an AI chatbot or by taking notes?
- Taking notes worked better in an experiment with 14-15-year-olds. Three days later, students who took notes, alone or alongside a chatbot, understood and remembered history texts better than those who used only the chatbot. The tests covered only the given texts, not broader learning. (See §4.2, p. 12; §4.2, p. 12.)
- Can students tell which study methods help them learn?
- Perhaps not. Students enjoyed the chatbot more and rated it more helpful for understanding and learning than note-taking, yet scored lower after using it. We suggest students may have limited metacognitive knowledge of what helps them learn, and underweight the 'desirable difficulties' of effortful activities such as note-taking. (See §5, p. 16.)
- Why might studying with an AI chatbot lead to less learning?
- Possibly because the chatbot can do the work that builds understanding. Learning grows with cognitive engagement, and having a chatbot summarise or explain may feel easier but reduce it. Students reported less effort with the chatbot alone, and their prompts suggested they treated it mainly as a knowledgeable source. (See §4.3.3, p. 15; §5, p. 16.)
- Does an AI chatbot have any learning benefits?
- Possibly, though not on the measures used. Students used it to clarify words, condense the text and ask about background, and some explored questions beyond the text, such as why America feared communism. Testing covered only the given texts, so that broader learning went unmeasured. (See §5, p. 16.)
Quantitative results
| Measure | Result | Where |
|---|---|---|
| Sample | 405 recruited; 344 analysed (Group 1: 184 students, 53.5 %; Group 2: 160, 46.5 %) | §3.2, p. 5 |
| Literal retention, Notes vs LLM | β = 1.92, t = 7.50, p < 0.001, d = 0.44 | Table 3, p. 13 |
| Literal retention, LLM + Notes vs LLM | β = 0.57, t = 2.06, p = 0.040, d = 0.13 | Table 3, p. 13 |
| Comprehension, Notes vs LLM | β = 0.95, t = 5.73, p < 0.001, d = 0.38 | Table 3, p. 13 |
| Comprehension, LLM + Notes vs LLM | β = 0.35, t = 1.98, p = 0.048, d = 0.14 | Table 3, p. 13 |
| Free recall, Notes vs LLM; LLM + Notes vs LLM | Notes: β = 1.02, p = 0.017, d = 0.21; LLM + Notes: p = 0.854 (not significant) | Table 3, p. 13 |
| Advantage of Notes over LLM + Notes (relative to LLM; indirect, exploratory comparison) | Literal retention: estimated mean difference = 1.35, z = 3.60, p < .001; comprehension: 0.59, z = 2.43, p = .015 | §4.2, p. 12 |
| Activity preference | Group 1: LLM over Notes 42.0 %, Notes over LLM 26.9 %; Group 2: LLM + Notes over LLM 50.5 %, LLM over LLM + Notes 16.2 % | Table 4, p. 14 |
| Copying from LLM output into notes (LLM + Notes) | 25.63 % of students over 70 % trigram overlap; 16.25 % over 90 % | §4.3.3, p. 15 |
| Prompts analysed; most frequent archetype | n = 4929; seeking additional information and deeper understanding, 2265 prompts | §4.4, p. 15 |
Contributions and implications
- Separate LLM use from note-taking, or block pasting into the notepad, so that students synthesise information themselves.
if the LLM and note-taking are used simultaneously, pasting into the notepad could be blocked
(§5.1, p. 17) - Students need more comprehensive training to use LLMs in goal-focused ways aligned with active and constructive learning strategies. Software could also limit distracting, off-topic use.
- Educators could use students' LLM prompts, in a privacy-protecting way, to see what concepts students struggle with or are curious about, and tailor lessons.
educators could leverage insights from students' interactions with the LLM to better understand what concepts they are struggling with or what they are curious about
(§5.1, p. 17)
Limitations
- Students received instructions and a demonstration but no in-depth training or practice, which may have disadvantaged the less familiar LLM conditions.
One limitation of the present study is that students received no in-depth training for the different learning activities.
(§5.2, p. 17) - There was no reading-only control condition, and Notes and LLM + Notes could be compared only indirectly, so that comparison needs caution.
these comparison outcomes still need to be considered with caution because of the potential interaction effects (each group could have experienced the LLM differently)
(§5.2, p. 18) - Qualitative coding was not independently double-coded, and prompts were analysed singly rather than as dialogues. Engagement used single-item measures.
- A single, isolated learning activity with GPT-3.5 turbo, a model less advanced than more recent ones. Repeated or classroom use could yield different results.
the study was limited to a single, isolated learning activity outside of the context of normal use throughout an entire course of study
(§5.2, p. 18)
›Concepts
- Reading retention
- The memory outcome, tested days after reading:
Reading retention is the process of storing the comprehended content from a text in long-term memory.
(§2.1, p. 2) - Literal retention versus comprehension
- The study's two outcomes. Comprehension questions required inferences across the text and from prior knowledge. Literal retention questions did not:
Literal retention questions (short response and multiple choice) required literal recall or recognition of information from the passage to provide a correct response.
(§3.5.1, p. 7) - Comprehension-focused reading
- Reading with active, constructive or interactive strategies, such as paraphrasing, making connections or dialogue with a partner.
Note that the generation of questions (in this case LLM prompts) itself can be considered as a constructive process if the questions encourage students to think beyond the immediate text
(§2.2.1, p. 3) - Cognitive offloading to the LLM
- The risk that relying on an LLM for answers replaces the thinking that builds understanding.
When learners depend excessively on LLMs for answers and explanations, they may be less inclined to employ self-explanation and elaboration strategies that are essential for comprehension and meaningful learning
(§2.2.1, p. 3)
Figures
How to cite
Pia Kreijkes, Viktor Kewenig, Martina Kuvalja, Mina Lee, Jake Hofman, Sylvia Vitello, Abigail Sellen, Sean Rintel, Daniel G. Goldstein, David Rothschild, Lev Tankelevitch, and Tim Oates. 2025. Effects of LLM Use and Note-Taking On Reading Comprehension and Memory: A Randomised Experiment in Secondary Schools. Computers & Education. https://doi.org/10.1016/j.compedu.2025.105514
This brief was prepared with AI assistance and published by Sean Rintel. It is not the authors' published text. Page numbers refer to this PDF.