Middle school students file into the computer lab and take their seats at the monitors. As they work through the math problems on the screen, they receive immediate feedback about their performance. Students who excel are given a harder set of problems to work through, while those who are struggling practice more basic math skills.

Meanwhile, their teacher observes her class through a pair of mixed-reality glasses that show icons hovering over the students' heads. Some icons indicate the students are doing well on the lesson. Others suggest the students are having trouble but are likely to arrive at the right answer. Some icons reveal that the student is stuck.

It's not a dream scene from the future but a real-life pilot project led by Vincent Aleven, PhD, a researcher in human-computer interaction at Carnegie Mellon University and a member of CMU's LearnLab.

LearnLab's mission is to create educational technologies that help students learn, while also serving as a platform through which to study the cognitive principles and mechanisms of human learning. In most LearnLab projects, both of those things are happening at once. Students might be learning algebra or Chinese with computer-based tutoring systems, for instance, while at the same time those systems embed experiments and collect data on what factors influence learning.

The interdisciplinary project draws collaborators from fields including psychology, computer science and human-­computer interaction. It's a sizable undertaking, with 10 faculty members—each with a set of graduate students—plus postdoctoral researchers, research assistants and programmers.

Though LearnLab is big, it's goal is even bigger, says lab director and cognitive psychologist Ken Koedinger, PhD, a professor of human-computer interaction and psychology. "We're really striving to be a model for every school or university by applying psychological research to what it does."

Real-world research

LearnLab got its start as the Pittsburgh Science of Learning Center, one of three national learning science centers funded by the National Science Foundation in 2004. A joint program of CMU and the University of Pittsburgh, the Pittsburgh learning center—also directed by Koedinger—was an interdisciplinary program focused on advancing a practical science of student learning.

When NSF funding for the program ended in 2014, the Pittsburgh center shut down, along with the official partnership between CMU and the University of Pittsburgh. But Koedinger and his colleagues decided to keep the mission alive. LearnLab receives funding through grants and through CMU's Simon Initiative, a program that takes a cross-disciplinary approach to measurably improving student learning outcomes.

Turning classrooms into learning laboratoriesDespite its name, a standout feature of LearnLab is that it takes the research into real-world middle school, high school and college classrooms, rather than conducting experiments in a lab. That approach allows for longer studies with students engaged in genuine academic learning. And from a practical point of view, it means researchers can spend less time recruiting and scheduling lab participants, and more time working with students in schools.

"The central idea is to study learning in real learning environments, but with the same kind of methodological rigor that we have historically incorporated into psychology labs," says Koedinger.

LearnLab isn't the only group studying learning in an applied setting, but it takes a unique approach, says Paulo Carvalho, PhD, a postdoctoral researcher at LearnLab since 2016. "Many times, when people do applied research, they do a different kind of experiment than they would do in a laboratory. Here it's the same materials, the same approach, we just do it in collaboration with teachers and schools, instead of in a lab with participants."

Straddling the line between applied science and basic, theory-oriented research on learning can be challenging at times, says Rony Patel, PhD, a former psychology graduate student of Koedinger's who now works at Google.

"Sometimes people on the educational side thought we were too theoretical, and people on the theoretical side thought we were too practical. We had to figure out how to negotiate that space," he says. "But what I liked about LearnLab was that I was in actual schools getting actual feedback from teachers, administrators and other stakeholders in the education fields, in addition to education researchers."

Shaking things up

With so many scientists and students from a variety of departments, it's no surprise that LearnLab covers a lot of research territory.

One recurring theme is the development of intelligent tutoring programs, which grew out of cognition research by CMU psychologist John R. Anderson, PhD. Intelligent tutors are computer-based learning systems that provide immediate and customized instruction and feedback to students as they work through a series of problems.

Such systems don't replace teachers, but are used in classrooms alongside face-to-face instruction, Koedinger says. Think of them as interactive textbooks that provide students opportunities to test their developing knowledge, practice new skills and get personalized instruction in response to their own needs. Meanwhile, LearnLab researchers can embed experiments into the system to advance understanding of the processes of conceptual change and develop better ways to teach.

During his graduate work, for instance, Patel used an intelligent tutoring system to try to improve the way students learn fractions. "There's a lot of research showing a huge knowledge gap in terms of fractions, even relative to other mathematical operations," he says. Traditionally, students learn about fractions using a strategy known as blocking—first, they learn fraction addition, then move on to fraction multiplication. But many students confuse the operations, Patel says.

Using intelligent tutoring systems with fourth- through eighth-graders, he compared the effectiveness of blocking with interleaving—which mixes problem types so that students practice fraction addition and multiplication at the same time—and found that students performed better with interleaving. He also discovered that reversing the order of the blocks—teaching fraction multiplication before addition—also led to better learning outcomes compared to the traditional method (Proceedings of the 38th Annual Conference of the Cognitive Science Society, 2016).

While LearnLab has demonstrated great success in applying intelligent tutoring systems, Koedinger wondered if those systems might be limited by the constraints of learning on a flat screen. To explore that idea, his graduate student Nesra Yannier used depth camera technology and artificial intelligence vision to develop an intelligent tutoring system that watched 4- to 8-year-olds as they played a game that involved predicting and explaining what makes block towers fall on a simulated earthquake table.

To test the potential added benefit of doing hands-on science, Yannier developed a computer version of the game where kids watched videos of towers from the earthquake table. Compared with students engaged in computer-based inquiry, those who observed real-life demonstrations learned more, both about physics principles and building sturdier towers (ACM Transactions on Computer-Human Interaction, Vol. 23, No. 4, 2016).

"The kids learn almost five times more from the 3-D physical interaction than they do from the exactly analogous flat screen interaction. And it's more fun! They're super excited about it," Koedinger says.

Other LearnLab projects have begun to look beyond student learning to explore better ways to support teachers, such as Aleven's project to provide teachers with real-time feedback about student performance through mixed-reality glasses. "Software can give different students different sets of problems, and I'd argue that's a good thing. But it's not without its challenges for teachers," he says.

Providing teachers with real-time insight into their students' thinking allows them to focus their attention where it's needed. "Students who ask for help aren't always the ones who need it the most," Aleven says. "Time moves fast in a classroom, lots of things are happening, and it's important to give the teacher the highlights."

Driving by data and collaboration

Computational modeling and data analysis is another key element of LearnLab's mission. The lab oversees DataShop, a central repository for the data collected from educational software. Those data are freely available to researchers around the world and integrated into all the work being done in LearnLab.

"It's an incredible resource to have just lying around. We sometimes call it the Large Hadron Collider of Learning because it's like having a particle accelerator for educational data mining," says Erik Harpstead, PhD, who completed his dissertation in human-computer interaction in Aleven's lab in 2017 and now works as a systems scientist at CMU. "If someone has an interesting idea, they can pull some data sets to explore it without having to set up a whole study."

That data repository allows researchers to continuously refine their models and interventions, says Carvalho. "We go to the old data to look for patterns before starting new research. It's about closing the loop: creating models, applying data, seeing the results and then going back to the model," he says. "It's going full circle to create better and better interventions."

Carvalho, for instance, has been combing through previously collected data to compare the benefits of reading course materials versus working through activities or exercises in online courses. In a phenomenon he calls the "doer effect," students who engage in more activities have better grades than those who spend more time reading. "We see the doer effect in pretty much every data set we could find," he says.

If access to reams of data is one of LearnLab's highlights, its embrace of interdisciplinary collaboration is another, says Amy Ogan, PhD, an educational technologist and LearnLab faculty member who received her doctorate in the department under Aleven in 2011. Ogan studies the social context of the use of educational technologies, such as how students engage with educational software when their home language and the language of instruction differ. Though her background is in computer science, such research requires partnerships with psychology, she says. People with technical know-how frequently turn to psychologists for theory, and those with psychology training know who to ask for help if they need to build a system to test their ideas, she says. "Having people with all of these backgrounds lets us push things to a level we might not be able to otherwise."

To foster those connections, graduate students in LearnLab are required to do an interdisciplinary project with another student outside their field. It's a welcoming environment to both students and faculty members who are interested in getting involved, says Harpstead. "There's a core team of LearnLab researchers. But if you're a student in an affiliate department or someone who is only peripherally involved, it's really easy to walk up and say, ‘Can I work on something like this?' There's a kind of flexibility to bring people in from project to project."

Those partnerships have been fruitful. LearnLab has improved student learning by improving educational technologies, and educational companies, such as Pearson and Kaplan, have adopted evidence-based practices from the lab's research. The lab has also launched startup companies that are putting intelligent tutoring systems into schools.

The research coming out of LearnLab is often highly technical, requiring a deep understanding of cognitive principles and complex data analysis. But ultimately, the work is less about the educational systems and more about the people using them, Ogan says. "We think a lot about students and how to support them through personalization of their learning experience. That's the big connecting thread running through LearnLab."

"Lab Work" illuminates the work psychologists are doing in research labs nationwide. To read previous installments, go to www.apa.org/monitor/digital/ and search for "Lab Work."

Additional reading

Data Mining and Education
Koedinger, K.R., D'Mello, S., McLaughlin, E.A., Pardos, Z.A., & Rosé, C.P. WIREs Cognitive Science, November 2015

Instructional Complexity and the Science to Constrain It
Koedinger, K.R., Booth, J.L., & Klahr, D. Science, November 2013

The Knowledge-Learning-Instruction Framework: Bridging the Science-Practice Chasm to Enhance Robust Student Learning
Koedinger, K.R., Corbett, A.T., & Perfetti, C. Cognitive Science, 2012

International Handbook of Metacognition and Learning Technologies
Azvedo, R. & Aleven, V. (Eds.), 2013