NBER Study Puts AI Tutoring Claims to a School-Level Test

A recent study by the National Bureau of Economic Research (NBER) has put the claims of AI tutoring to the test in a real-world educational setting, examining the impact of Khan Academy's AI tutor, Khanmigo, on student performance in mathematics.

Study Overview

The NBER study, titled "One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment," conducted a two-year cluster randomized trial across 18 middle schools in Tennessee. The research aimed to evaluate the effectiveness of Khanmigo, an AI-powered tutor, in enhancing students' math achievement during remedial sessions. The study found that students using Khanmigo showed an increase in math achievement by 1.3 national percentile ranks per term, equating to about 0.06 to 0.08 standard deviations over a school year.

Operational Implications

The study's findings suggest that while AI tutors like Khanmigo have the potential to improve educational outcomes, their success heavily depends on student engagement. Despite 96% of students trying Khanmigo at least once, the median student interacted with the AI on only a third of their practice days. This highlights a critical operational challenge: ensuring consistent and meaningful student interaction with AI tools. Schools and educators may need to explore strategies to boost engagement, such as integrating AI tutoring more seamlessly into the curriculum or providing incentives for regular use. Additionally, understanding the role of AI literacy in education could be crucial for maximizing the benefits of AI tutors.

Commercial Considerations

For companies developing AI educational tools, the study underscores the importance of designing systems that not only provide access but also encourage active participation. The commercial success of AI tutoring platforms may hinge on their ability to foster sustained engagement. This could involve enhancing the interactivity of AI tutors or developing features that better capture students' attention. Companies like Khanmigo might consider these factors in future iterations of their products. Furthermore, insights from the Harvard Business School on educational technology could inform strategic decisions.

Limitations and Next Steps

While the study provides valuable insights, it also points to limitations that warrant further investigation. The gains observed with Khanmigo were similar to those achieved through traditional Khan Academy practice without AI assistance. This raises questions about the unique value proposition of AI tutors. Future research could explore how AI tools can be optimized to deliver distinct advantages. Additionally, stakeholders should verify the scalability of these findings across different educational contexts and subjects. The Digital Promise report offers further context on these challenges.

Further Reading and Resources

For those interested in exploring the broader implications of AI in education, several resources are available. The EdWorkingPapers and Poverty Action Lab reports provide additional context on AI's role in personalized learning. The ResearchGate publication explores AI's potential in scientific education. For a deeper dive into the technical aspects, consider reading the Microsoft AI Agent Guide and the OpenAI Strategic Futures Blog. Additionally, the EdTech Innovation Hub provides insights into AI's impact on science education. For governance perspectives, the NBER taxonomy offers a comprehensive overview of AI's role in education policy. Finally, the NBER paper on AI governance discusses broader implications for educational technology.

Decision-makers should scrutinize AI tutoring's operational evaluation and governance. Assess limitations and procurement processes, ensuring robust validation. Verify alignment with strategic goals and workforce readiness, as seen in workforce initiatives and AI model routing advancements.