Learning behavior accounts for background-related advantage in AI-assisted education

Authors: Jingwei Yi, Yueqi Xie, Jiyan He, Rui Ye, Junming Huang, Bin Zhu, Sean Rintel, Yu Xie, Xing Xie, Fangzhao Wu
Venue: arXiv, July 2026
Preprint (arXiv)

AThe study

Randomised trial with 318 university students new to game theory or Python, in one online session of up to 125 minutes. Two in three had a GPT assistant, and all sat the exam without AI.

Learn, 40 minutesAssignment, 20 minutesReview, 20 minutesExam, 20 minutes AI group158 in game theory, 54 in Python With GPT With GPT With GPT No AI No-AI group79 in game theory, 27 in Python No AI No AI No AI No AI

BWhat changed

Exam scores by how students used AIstudents chose these behaviours themselves

  • ↑AI group: students who tried first and checked the AI scored higher than the no-AI group.Trying problems before asking, checking answers or fixing AI mistakes. Students chose these behaviours, so this is an association, not proof of cause.
  • ↓AI group: in Python, students who ignored or copied the AI scored lower than without AI.Mean exam scores of 1.83 for ignoring, with only 5 students, and 2.94 for copying, against 4.47. This is an association, not proof of cause.
  • –AI group: in game theory, students ignoring or copying the AI scored like the no-AI group.Mean exam scores of 4.92 and 4.39, against 5.09. Neither difference was statistically reliable.

Who used AI wellstudent background, within the AI group

  • AI group: students from higher-ranked universities were more likely to try first and check the AI.So were students who scored higher on a prior-knowledge quiz. In game theory, both links were statistically reliable.
  • AI group: once AI use was accounted for, the exam advantage of stronger backgrounds shrank.How students used AI accounted for 13.7% to 18.9% of the advantage in game theory and 30.5% to 31.7% in Python.
  • –AI group: extra gains from AI for stronger-background students were not statistically reliable.Exploratory analyses pointed that way, but the samples were too small to detect such differences reliably.

What students saidsurveys before and after the session

  • AI group: students who tried first more often said they turned to AI when stuck.Self-reported. Concern about relying too much on AI grew more among students who ignored or copied it.

↑ better than the comparison group   ↓ worse than the comparison group   – no reliable difference

CKey takeaway

Proactive and critical engagement, trying problems before checking AI answers, may help students learn more. Stronger-background students tend to do this, so equal access can still mean unequal learning.

Made for this site with AI assistance from the verified research brief. It is not the paper's own figure.

Abstract

Generative AI has been found, and will likely be found increasingly, useful in education. However, existing AI-for-education studies provide inconsistent evidence on its average effects. More broadly, research on prior educational technologies shows that average effects often mask substantial heterogeneity across student populations. Motivated by this evidence, this study examines heterogeneity in students' learning behavior with AI, which students benefit from AI assistance, and how learner profiles and learning behavior shape these patterns. To this end, we recruited 318 university students to participate in structured learning experiments lasting up to 125 minutes. Our findings indicate that students' learning behavior is strongly associated with learning outcomes, with behaviors characterized by proactive and critical engagement, rather than limited engagement, associated with significantly better performance. These behavioral differences are related to learner profiles, with students from higher-ranking universities and those with greater prior knowledge tending to benefit more, consistent with their greater likelihood of adopting proactive interaction strategies. Accounting for learning behavior substantially weakens or eliminates the associations between learner profiles and learning outcomes, suggesting that how students use AI is a key pathway through which background differences are linked to learning gains. Overall, this work provides a deeper understanding of AI assistance in education by showing how differences in learner profiles and learning behavior shape who benefits from AI-supported learning. These insights can help educators and students better navigate and integrate AI into educational practices.

Research brief

By Jingwei Yi, Yueqi Xie, Jiyan He, Rui Ye, Junming Huang, Bin Zhu, Sean Rintel, Yu Xie, Xing Xie, and Fangzhao Wu.

Takeaway: University students who engage with AI proactively and critically (trying problems before asking and checking its answers) score higher on later AI-free exams. Because better-prepared students do this more, equal access may yield unequal learning.

Why this matters

What was known
Studies of AI in education report positive, null and negative average effects, and earlier educational technologies showed that averages hide large differences between students. One reason for these divergent conclusions may be that average effects obscure variation across students and forms of AI use. (§A Study design and procedure, p. 2)
What this paper adds
Annotated logs of how each student actually used GPT, linked to their later exam scores and to their backgrounds (university ranking and prior knowledge). we conducted a randomized controlled experiment involving 318 university students with diverse learner profiles to examine who benefits from AI assistance and how learning behavior is associated with these benefits (§A Study design and procedure, p. 3)
Who it matters to
Educators, students, policy makers, and designers of AI-supported learning systems. (§Discussion, p. 9)

Research question

Method and data

Design
Randomised controlled trial in two course contexts (Python and game theory). Within each course, students were allocated 2:1 to an experimental group with GPT access (during learning, assignment and review) or a control group with no LLM tools. All took the final exam without GPT access.
Setting
A fixed, stage-based online study platform: pre-task survey and prior-knowledge quiz, 40-minute course learning, 20-minute assignment, 20-minute review, 20-minute exam and post-task survey, screen recording for compliance, and performance-based pay.
Participants / data
318 adult university students from a wide range of academic tiers, with no prior study of the target subject (game theory: 158 experimental, 79 control, and Python: 54 experimental, 27 control).
Duration
One structured session lasting up to 125 minutes.
Analysis
Manual annotation of assignment-phase chat logs into five behaviour patterns. One-sided Brunner-Munzel tests (Benjamini-Hochberg adjusted). OLS linear probability and nested OLS models of exam score with bootstrap indirect associations. OLS group-by-profile interaction models. Pre/post survey analysis. Behaviour comparisons are treated as observational.

Findings

Questions this paper answers

Does using an AI chatbot while studying help students learn?
It depends on how they use it. Among students learning with a GPT assistant, those who tried problems first, checked its answers or corrected its errors did better on a later AI-free exam. Those who ignored it or copied its answers did no better, or worse. These are associations. (See §Heterogeneity in learning behavior shapes learning outcomes, p. 5; §Heterogeneity in learning behavior shapes learning outcomes, p. 5.)
Does AI widen gaps between stronger and weaker students?
It may. Students from higher-ranked universities or with more prior knowledge were more likely to use AI proactively and critically. They also tended to gain more, although those gain differences were exploratory and not statistically significant. (See §Discussion, p. 9; Results, p. 7.)
Why do some students benefit more from AI than others?
Part of the background advantage appears to run through behaviour. Taking into account whether students engaged proactively and critically weakened the link between background and exam score, in one course to the point of non-significance. These are associations, not causal estimates. (See §Discussion, p. 9; Results, p. 7.)
Do students who lean on AI answers realise they depend on it?
Partly. Concern about over-relying on AI grew more among students who ignored or copied the AI than among those who engaged critically. Ready-made answers can make AI feel useful while also making students more aware of their dependence on it. (See §Participant feedback analysis, p. 7.)

Quantitative results

MeasureResultWhere
Sample318 students; game theory 158 experimental, 79 control; Python 54 experimental, 27 controlTable 1, p. 3
Exam score vs control (control 5.09), game theory: abstention; rote-adoption4.92 (not significant); 4.39 (not significant)Fig. 2, p. 4
Fig. 2 (read from the exam-score panel)
Exam score vs control (control 4.47), Python: abstention (n = 5); rote-adoption (n = 28)1.83 (significantly lower); 2.94 (significantly lower)Fig. 2, p. 4
Fig. 2 (read from the exam-score panel)
Profile predicts proactive-critical engagement, game theory (linear probability)auniv = +0.171, P = 0.010; aprior = +0.161, P = 0.002Results, p. 5
Profile predicts proactive-critical engagement, Pythonaprior = +0.217, P = 0.046; auniv = +0.161, P = 0.127; mid/high vs low ranking proportion difference = +0.32, P = 0.035Results, p. 5
Share of profile-exam score association accounted for by learning behaviour (university ranking, prior knowledge)game theory 13.7% and 18.9%; Python 30.5% and 31.7%Results, p. 7
Indirect association through learning behaviour, Python prior knowledge0.87 (95% CI, 0.032-2.105)Results, p. 7
Experimental-by-profile interaction on exam score (exploratory; not significant)Python β3 = +1.13 (ranking), +1.05 (prior knowledge); game theory β3 = +0.13 and +0.03Results, p. 7

Contributions and implications

Limitations

›

Concepts

Limited engagement
Two of the five behaviour patterns we annotated, grouped together: Abstention denotes completing the assignment without using the assistant despite having GPT access. Rote-adoption denotes direct use of AI-generated answers with minimal independent reasoning. (§Heterogeneity in learning behavior shapes learning outcomes, p. 5)
Proactive and critical engagement
The other three patterns, grouped together: Active-trial denotes independent attempts before consulting the AI. Verification denotes explicit checking or follow-up questioning of AI responses against one's own reasoning. Error-correction denotes identifying and revising incorrect AI outputs. (§Heterogeneity in learning behavior shapes learning outcomes, p. 5)
Learner profiles
Students' background, measured here as university ranking tier and a timed prior-knowledge quiz. (§Measures, p. 10)
Abstention
Having GPT access but not using it during the assignment. (§Heterogeneity in learning behavior shapes learning outcomes, p. 5)

Figures

Example chats for five AI-use behaviours. Per-student behaviour shares. Exam score violins where active-trial, error-correction and verification score above control, and abstention and rote-adoption do not.
Figure 2. Students adopt distinct AI-use learning behavior patterns, and proactive and critical engagement is associated with higher exam score. a, Schematic examples of five annotated learning behavior patterns among experimental participants during assignment work: abstention, rote-adoption, active-trial, verification, and error-correction. These patterns distinguish participants who forgo available AI support, directly adopt AI output, attempt independently before seeking help, verify AI responses, or identify and correct AI-generated errors. b, Student-level composition of learning behavior across assignment problems in game theory and Python. Each horizontal bar represents one experimental participant, and colored segments indicate the share of assignment problems classified into each behavior pattern. c, Exam score distributions by learning behavior pattern in game theory and Python. Violin plots show score distributions with overlaid box plots. Legends report group sizes (n), and µ denotes mean exam score. Significance annotations are based on one-sided Brunner-Munzel tests for selected pairwise comparisons, adjusted using the Benjamini-Hochberg procedure. From the paper, Fig. 2, p. 4.
Proactive-critical AI use grows with university ranking and prior knowledge. Behaviour accounts for 13.7% to 31.7% of profile associations. AI gains look larger for high-profile students, mostly in Python, but not significantly.
Figure 3. Students with stronger learner profiles show more proactive-critical AI use and larger gains in exam score. a, Learning behavior composition across university ranking and prior knowledge strata in game theory and Python. Stacked bars show the share of experimental participants in each behavior pattern. b, Exam score by learning behavior group, with separate lines for learner profile strata based on university ranking and prior knowledge. Points show median exam scores with bootstrap 95% confidence intervals. Line overlap indicates smaller exam score differences between learner profile strata after learning behavior is fixed. Aβ denotes the proportion of the profile association accounted for by learning behavior, and Iβ denotes the indirect association through learning behavior. c, Exam score differences between the experimental and control groups across learner profile strata. Points show mean exam scores with bootstrap 95% confidence intervals. Shaded bars show differences computed as experimental minus control. β3 denotes the experimental-group-by-resource interaction coefficient from an OLS regression. From the paper, Fig. 3, p. 6.

How to cite

Jingwei Yi, Yueqi Xie, Jiyan He, Rui Ye, Junming Huang, Bin Zhu, Sean Rintel, Yu Xie, Xing Xie, and Fangzhao Wu. 2026. Learning behavior accounts for background-related advantage in AI-assisted education. arXiv preprint arXiv:2607.10101. https://doi.org/10.48550/arXiv.2607.10101

This brief was prepared with AI assistance and published by Sean Rintel. It is not the authors' published text. Page numbers refer to this PDF.

BibTeX

@article{yi2026learning,
author = {Yi, Jingwei and Xie, Yueqi and He, Jiyan and Ye, Rui and Huang, Junming and Zhu, Bin and Rintel, Sean and Xie, Yu and Xie, Xing and Wu, Fangzhao},
title = {Learning behavior accounts for background-related advantage in AI-assisted education},
year = {2026},
month = {July},
url = {https://doi.org/10.48550/arXiv.2607.10101},
journal = {arXiv preprint arXiv:2607.10101},
doi = {10.48550/arXiv.2607.10101},
}