Harvard AI tutor nearly doubled learning gains in 194-student physics study

What happens when students can turn to a physics tutor at any hour, learn at their own pace and ask the same question as many times as they need?

A Harvard experiment involving 194 students offers an intriguing answer. Students who learned with a custom-built AI tutor recorded learning gains that were about twice those of students taught through an instructor-led active learning approach, according to preliminary findings reported by the Harvard Gazette.

The study was led by Harvard lecturers Gregory Kestin and Kelly Miller and focused on students enrolled in Physical Sciences 2, a course for life sciences majors. The findings were reported in September 2024, while the researchers were still preparing the final research for publication.

Students learned more and reported greater engagement

The experiment involved two lessons taught over consecutive weeks. One group first participated in an instructor-guided active learning session, while the other learned the material using the AI tutor at home. The groups switched approaches the following week.

Researchers assessed students through pre-tests and post-tests to measure how much they learned. They also collected feedback on engagement, enjoyment, motivation and growth mindset.

The preliminary results showed that students using the AI tutor made learning gains roughly twice as large as those in the classroom-based active learning group.

Students also reported being more engaged and motivated while working with the AI tutor.

The finding was notable because the comparison was not between AI tutoring and a conventional lecture. Harvard's classroom approach already relied on active learning and research-backed teaching methods.

“It was shocking, and super exciting,” Miller said, according to the Harvard Gazette.

The tutor was built to teach, not simply answer

The experiment did not involve giving students unrestricted access to a standard ChatGPT conversation.

Kestin and Miller designed the AI system to behave more like an experienced instructor. It was built using the GPT application programming interface and incorporated carefully developed prompts, pre-vetted conversations and structured feedback.

The approach was designed to make the interaction more personalised.

Students who understood a concept could move ahead, while those who struggled could spend more time asking questions and working through problems without the pressure of keeping pace with an entire class.

That flexibility could also reshape how teachers use classroom time.

If students can receive personalised support while learning basic concepts outside class, teachers could potentially devote more classroom time to complex problem-solving, projects, discussion and collaborative work.

Researchers caution against treating AI as a replacement

The Harvard researchers also stressed that the findings should not be interpreted as evidence that AI can simply replace instructors.

Kestin cautioned that AI could either strengthen or undermine learning depending on how it is designed and used. The objective, he argued, should be to encourage students to think critically rather than allow AI to do the thinking for them.

The Harvard team was also exploring whether the approach could work beyond physics. Versions of the AI tutor were being considered for subjects including multivariable calculus, with Harvard's Derek Bok Center for Teaching and Learning and Harvard University Information Technology preparing additional pilots.

The larger takeaway from the experiment was therefore not simply that AI can answer students' questions.

It was that a tutor built around a teacher's expertise, structured feedback and clearly defined learning goals could give students more personalised support while making learning more active and engaging.