TutorMoments is a new framework from Allen AI for evaluating whether large language models can balance the pedagogical trade-off between scaffolding (making a problem easier) and pushing for rigor (letting the student do more reasoning). The evaluation is built on 462 de-identified transcripts of real one-on-one math tutoring sessions with U.S. students in grades 2-7, drawn from a high-dosage tutoring program. Experienced math teachers annotated over 1,500 key moments where the tutor had to decide whether to offer support or challenge the student. TutorMoments then replays the session from that decision point: the LLM acts as tutor for five turns with a simulated student, and an automated scoring pipeline rates whether the model’s moves were appropriate—scaffolding when needed, pushing for rigor when appropriate, and avoiding over-scaffolding.

Results from seven LLMs show a clear tendency to over-help: with a plain prompt that only says “tutor well,” models rarely push for rigor. When the prompt explicitly describes the trade-off (scaffolding vs. rigor vs. over-scaffolding), every model improves, but the gap to human tutoring remains large. The human tutors in the dataset scored 0.458 on appropriate scaffolding, 0.182 on appropriate rigor, and 0.496 on avoiding over-scaffolding—all below the best evaluation-aware model scores. The authors caution that human tutors are a naturalistic reference, not a ceiling, because the dataset deliberately focuses on moments where tutoring could have been better. The scoring measures tutor behavior, not student learning, and uses an oracle simulated student.

TutorMoments is a preview with several stated limitations: the dataset is narrow (U.S. math, grades 2-7, annotated by one pool of educators), the scoring pipeline is less reliable for rigor moments, and the findings may not generalize to other subjects or settings. The team is releasing the dataset, code, and replays for reproducibility, and plans to build a larger multimodal dataset and stronger scoring pipeline. The work is supported by the Gates Foundation and Learning Commons.