AI Tutor Delivers Record Learning Gains in Dartmouth Physics Course

A new AI tutor built on GPT-4 achieved effect sizes of 0.71 to 1.30 standard deviations on exam performance in an introductory physics course at Dartmouth College. The results, published in a workshop paper for the Intelligent Textbooks 2025 workshop, represent some of the largest learning gains ever recorded for an automated tutoring system.

The Setup

The study ran in Dartmouth's PHYS 13 course, a calculus-based introductory physics class. Students used the AI tutor as a supplemental tool during the winter 2024 term. The system, called "TutorAI" (not the final name), was integrated into the course's existing learning management system.

How It Works

TutorAI uses GPT-4 to generate real-time, context-aware hints and explanations. When a student submits an answer, the system evaluates correctness and provides targeted feedback. For wrong answers, it doesn't just say "incorrect"—it generates a hint specific to the student's mistake, often referencing the relevant physics principle.

The system was designed with three key features:

  1. Adaptive hint generation: GPT-4 analyzes the student's answer and the problem context to produce a hint that guides without giving away the solution.
  2. Multi-turn dialogue: Students can ask follow-up questions, and the tutor maintains context across the conversation.
  3. Instructor oversight: All AI-generated content is logged and reviewable by the course instructor.

The Numbers

The study compared exam scores of students who used TutorAI (n=47) against those who didn't (n=53). The effect size of 0.71-1.30 SD is massive. To put that in perspective, typical effect sizes for human tutoring are around 0.4-0.8 SD. The 1.30 SD figure came from a subset of students who used the system most intensively.

Why It Matters

This isn't just another "AI in education" press release. The paper provides a rigorous, controlled experiment with a clear methodology. The authors controlled for prior GPA and used a pre-post test design. The results suggest that LLM-based tutors can match or exceed human tutors in certain contexts—at scale and at a fraction of the cost.

Technical Details

The paper includes specific implementation details:

  • Model: GPT-4 (gpt-4-0613) via OpenAI API
  • Prompt engineering: The system prompt includes the problem statement, the student's answer, and a set of pedagogical rules (e.g., "Do not provide the answer directly")
  • Context window: 8K tokens, which allowed the system to retain the full problem and conversation history
  • Latency: Median response time of 2.3 seconds per query

The authors also implemented a "confidence threshold"—if GPT-4's log probabilities indicated low confidence, the system would fall back to a pre-written hint from the instructor.

Limitations

The study has caveats. It was a single course at a single institution. The sample size is modest. And the authors note that the effect was strongest for students who voluntarily used the system more—self-selection bias may play a role. Additionally, the paper doesn't address long-term retention; exams were within the same term.

What's Next

The team is expanding the study to multiple courses and institutions this year. They're also open-sourcing the TutorAI codebase (Python, Flask, PostgreSQL) on GitHub. For developers, this presents an opportunity to experiment with LLM-based tutoring in their own educational contexts.

Editor's Take

I've spent years building educational software, and I've seen countless "AI tutors" that were little more than glorified chatbots. This one is different because it has the data to back it up. The 1.30 SD effect size is eye-popping—but I'm skeptical about generalizability. Dartmouth students are a self-selected, high-achieving group. I'd love to see this replicated at a community college with a more diverse student body. Still, the open-source release is a smart move. If you're building edtech, grab that code and run your own experiments. The paper's methodology is solid, and the potential for impact is real.

Developer Insights

  1. Study the prompt engineering patterns used in the paper—they're available in the appendix and can be adapted for other LLM-based tutoring applications.
  2. The confidence threshold fallback mechanism is a critical reliability pattern for production LLM systems; consider implementing similar safeguards in your own AI-powered features.
  3. The open-source codebase (Python/Flask/PostgreSQL) is a practical starting point for building your own AI tutor or similar educational tool.