What ten randomized trials say about AI tutors
Randomized studies of AI tutoring range from large gains in short, tightly designed lessons to lower grades in a semester-long university trial. Differences in how the tools were designed and used run through the results.
Key points
- Short, structured trials in a Harvard physics course, in Ghana and in Nigeria found sizable learning gains from AI tutors.
- A high school trial in Turkey found that an unrestricted chatbot raised practice scores but lowered exam scores; a version with teacher-designed safeguards avoided the drop.
- Two trials in Tennessee middle schools found small or marginal gains, and a University of Maryland trial found lower final grades.
- Seven of the ten studies are working papers or preprints, not yet peer reviewed.
This digest draws on ten randomized studies in Edictai's research library, published between 2024 and October 2026, that tested AI tutors or AI help with coursework. They differ in age group, subject, country, length and tool, so their effect sizes cannot be added up or averaged. Read side by side, they show where AI tutoring has helped and where it has not.
Large gains in short, structured lessons
In a randomized crossover trial in an introductory physics course at Harvard, students who worked through two lessons with a custom GPT-4 tutor at home had median learning gains more than double those of students taught the same lessons through in-class active learning1. The median AI-tutored student spent 49 minutes on task, against about 60 minutes in class2. The tutor was built around pedagogical best practices and pre-written solutions.
Two trials outside the United States also found gains. In Ghana, students in grades 3 to 9 who used Rori, a math tutor on WhatsApp, in two 30-minute sessions a week had substantially higher math growth over eight months, an effect size of 0.363. In Benin City, Nigeria, a six-week after-school program in which secondary students used Microsoft Copilot with teacher guidance raised scores on an end-of-program assessment by 0.31 standard deviations4 and on the regular end-of-term English exam by 0.21 standard deviations5.
AI that helps the tutor
One trial put the AI in the hands of human tutors rather than students. Students whose virtual tutors had Tutor CoPilot, a tool that suggests expert-like responses in real time, were 4 percentage points more likely to master math topics6. Gains were largest for students of lower-rated and less experienced tutors7.
Better practice, worse exams
A preregistered trial in a high school in Turkey separated practice from learning. Access to GPT-4 during practice raised practice grades by 48 percent with a standard chat interface and by 127 percent with a tutor version carrying teacher-designed safeguards8. On a later exam taken without AI, students who had used the standard chatbot scored 17 percent lower than students who never had access, while the safeguarded tutor's effect was statistically indistinguishable from no AI9. Most students in the standard arm used it to get answers10.
Small effects at scale
The two largest school trials, both in Hamilton County, Tennessee, found modest results. Over two years in 18 middle schools, assignment to Khan Academy's Khanmigo tutor in daily remedial math sessions raised achievement by 1.3 national percentile ranks per term11, gains similar to those from Khan Academy practice without AI12. Nearly all students tried the tutor, but the median student messaged it on only a third of practice days13.
A second experiment in the same district, with more than 6,000 students, found that AI support made students slower but more accurate after a mistake14. The strongest evidence of learning on a test a week later appeared when the AI was built into a mastery workflow, and those gains were only marginally significant15.
Lower grades in a university trial
At the University of Maryland, College Park, instructors were randomized to give their course sections a GPT-4o-based study assistant built into the learning management system. Among sections of the same course, access reduced final grades by 0.37 standard deviations and participation in online course activities by 0.90 standard deviations16. Estimated grade losses for first-generation students were more than twice those for continuing-generation students17.
A four-year trial at California State University, Northridge tested a different use: an AI-enabled text-messaging chatbot for student support rather than coursework. Its effects were concentrated on completing time-sensitive administrative tasks, with no detectable effect on academic performance or persistence18.
Hints written by AI
An online experiment with adult learners practicing algebra and statistics found that ChatGPT-written hints produced learning gains with no significant difference from hints written by human tutors19. But 32 percent of the ChatGPT hints first failed quality checks, a rate the researchers cut sharply with an added checking step20.
| Study | Setting | Status | Main result |
|---|---|---|---|
| AI tutoring vs. in-class active learning | Harvard physics course, 2023 | Peer reviewed | Median gains more than double |
| Rori math tutor | Ghana, grades 3-9 | Preprint | Effect size 0.36 |
| Copilot after-school program | Nigeria, secondary | Working paper | +0.31 SD; +0.21 SD on term exam |
| Tutor CoPilot | US district, grades 3-6 | Preprint | +4 percentage points in topic mastery |
| GPT-4 in math practice | Turkey, high school | Peer reviewed | -17% on exam with standard chatbot |
| Khanmigo | Tennessee, grades 6-8 | Working paper | +1.3 percentile ranks per term |
| AI in mastery practice | Tennessee, grades 6-8 | Working paper | Marginal delayed gains |
| Course-integrated AI tutor | University of Maryland | Working paper | -0.37 SD in final grades |
| Student support chatbot | CSU Northridge | Working paper | No academic effect |
| ChatGPT hints | Online adult learners | Peer reviewed | Gains equal to human hints |
Source: Edictai research library, October 2026.
Reading them together
Three patterns recur. In the one trial that compared them directly, a tutor with teacher-designed safeguards avoided the exam losses of an open chatbot, and the largest gains elsewhere came from tools built around pre-written solutions or teacher guidance. Use was often thinner than designers intended: in the Khanmigo trial the median student messaged the tutor on a third of practice days, and in Turkey most students used the open chatbot to get answers. And in Turkey, practice scores were a poor guide to what students could do later without AI.
Seven of the ten are working papers or preprints that have not been peer reviewed. The Harvard, Turkey and adult-learner studies were published in peer-reviewed journals.
Notes
- AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting, Scientific Reports, 3 Jun 2025. Results, Learning gains.
The median learning gains for students, relative to the pre-test baseline (M = 2.75, N = 316), in the AI-tutored group were over double
- AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting, Scientific Reports, 3 Jun 2025. Results, Time on task.
70% of students in the AI group spent less than 60 minutes on task, while 30% spent more than 60 minutes on task. The median time on task for students in the AI group was 49 minutes.
- Effective and Scalable Math Support: Experimental Evidence on the Impact of an AI-Math Tutor in Ghana, arXiv, 15 Feb 2024. Abstract.
We find that the math growth scores were substantially higher for the treatment group and statistically significant (p < 0.001). The effect size of 0.36 is considered large in the context of educational interventions
- From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria, World Bank, 19 May 2025. Abstract.
The intervention demonstrated a significant improvement of 0.31 standard deviation on an assessment that included English topics aligned with the Nigerian curriculum, knowledge of artificial intelligence and digital skills.
- From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria, World Bank, 19 May 2025. Introduction, p. 2.
We calculate an ITT effect of being selected for the program on the performance in the third-term exam of 0.21 standard deviations.
- Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise, arXiv, 3 Oct 2024. Abstract.
students whose tutors used Tutor CoPilot were 4 percentage points more likely to master mathematics topics (p<.01)
- Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise, arXiv, 3 Oct 2024. Results, RQ2.
lower-rated tutors experienced a 9 p.p. increase in students passing their exit ticket (56%→ 65%student passing rate from control to treatment). In Figure 2b, less experienced tutors showed a 7 p.p. increase (61%→ 68%).
- Generative AI without guardrails can harm learning: Evidence from high school mathematics, Proceedings of the National Academy of Sciences, 25 Jun 2025. Abstract.
having GPT-4 access while solving problems significantly improves performance (48% improvement in grades for GPT Base and 127% for GPT Tutor)
- Generative AI without guardrails can harm learning: Evidence from high school mathematics, Proceedings of the National Academy of Sciences, 25 Jun 2025. Main results.
on the exam, students in the GPT Base arm perform statistically significantly worse than students in the control arm by 17%; this negative effect is essentially eradicated in the GPT Tutor arm, though we still do not observe a positive effect.
- Generative AI without guardrails can harm learning: Evidence from high school mathematics, Proceedings of the National Academy of Sciences, 25 Jun 2025. Introduction.
we find evidence that students do not perceive any reduction in their learning or subsequent performance as a consequence of copying solutions
- One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment, Annenberg Institute at Brown University (EdWorkingPapers), Aug 2026. Abstract.
Assignment raises math achievement by 1.3 national percentile ranks per term, or about 0.06 to 0.08 standard deviations over a school year; the implied effect of a full year of active participation reaches 0.14 standard deviations.
- One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment, Annenberg Institute at Brown University (EdWorkingPapers), Aug 2026. Abstract.
These gains resemble those from Khan Academy practice without AI assistance.
- One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment, Annenberg Institute at Brown University (EdWorkingPapers), Aug 2026. Abstract.
96 percent of students tried Khanmigo at least once, but the median student messaged it on only a third of the days they practiced, and in only 17 percent of the exercise sessions in which they made a mistake.
- Making AI Tutoring Productive: Evidence from a Mastery-Based Math Practice Experiment, National Bureau of Economic Research, Aug 2026. Abstract.
AI students progressed more slowly and attempted fewer questions, but answered more accurately conditional on reaching an attempt. The clearest mechanism appeared after mistakes: AI improved next-attempt correctness
- Making AI Tutoring Productive: Evidence from a Mastery-Based Math Practice Experiment, National Bureau of Economic Research, Aug 2026. Abstract.
The most encouraging delayed-test evidence appears when AI is embedded in the mastery workflow, with marginally significant gains concentrated on practiced Exercise 1 material.
- The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial, Annenberg Institute at Brown University (EdWorkingPapers), Oct 2026. Abstract.
Among sections of the same course, tutor access reduced final grades by 0.37 standard deviations and learning management system participation by 0.90 standard deviations.
- The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial, Annenberg Institute at Brown University (EdWorkingPapers), Oct 2026. Introduction, p. 4.
the estimated final-grade losses among first-generation students were more than twice those among continuing-generation students in both samples
- Sustaining AI-Enabled Student Support: A Four-Year Implementation and Impact Study, Annenberg Institute at Brown University (EdWorkingPapers), Jul 2026. Abstract.
Students remain receptive over time, and impacts are concentrated in improved completion of time-sensitive administrative tasks, with no detectable effects on academic performance or persistence.
- ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills, PLOS ONE, 24 May 2024. Abstract.
We find that only the ChatGPT condition produces statistically significant learning gains compared to a no-help control, with no statistically significant differences in gains or time-on-task observed between learners receiving ChatGPT vs human tutor help.
- ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills, PLOS ONE, 24 May 2024. Abstract.
Notably, ChatGPT-generated help failed quality checks on 32% of problems. This was, however, reducible to nearly 0% for algebra problems and 13% for statistics problems after applying self-consistency
About this article
Written by Claude, an AI model, from Edictai’s records and research library. Before publication, every note is checked automatically against the record or study it cites. Descriptions are as of the data release named above.
Corrections: none.