AI-Powered Video Analytics
As Project Lead for a research initiative exploring the use of computer vision to measure early childhood classroom environments, I investigate how different computer vision and AI approaches can be applied to answer key research questions from classroom video. Early childhood classrooms present particular challenges for off-the-shelf computer vision models, including crowded scenes, frequent occlusion, substantial variation in children’s size and movement, and complex interactions among multiple people. Our broader research therefore explores a combination of task-specific trained computer vision models and multimodal AI models, with each approach contributing different capabilities.
The examples below focus specifically on what multimodal large language models (LLMs) can contribute to this toolkit. I prompted a multimodal LLM to “watch” and interpret classroom video, using clips from A Video Dataset for Classroom Group Engagement Recognition (Lu et al., Scientific Data, 2025), a published dataset of human-annotated classroom footage.
The Engagement Visualization runs the model across a 20-clip sample (10 clips originally rated High engagement, 10 rated Medium) and compares its predicted engagement level for each video against the paper’s human-coded label.
The Activity Visualization goes deeper on a single 15-second clip, coding every student’s behavior second-by-second into eight observable activity types, then validating each of the observations by hand against the source video, surfacing accuracy by activity type, by second, and by student.
Together, these examples demonstrate how multimodal AI models can transform unstructured classroom video into structured, quantitative data that can be analyzed at scale. In our analysis, LLMs may complement trained models by providing additional flexibility for interpreting complex behaviors and interactions. While substantial work remains to validate both approaches across diverse classrooms and research questions, advances in computer vision and multimodal AI are rapidly expanding what is possible. Combining these tools has the potential to dramatically reduce the time and cost of classroom observation while enabling researchers to measure educational environments at a scale that was previously impractical.