Learning is not a spectator sport

However, a study published in the Proceedings of the Second (2015) ACM Conference on Learning @ Scale shows that this central approach of MOOCs -- having students watch to learn -- is ineffective. Instead, the emphasis on interactive activities as advocated by Carnegie Mellon University's Simon Initiative helps students learn about six times more.

CMU's Simon Initiative, named for the late Nobel and Turning laureate Herbert Simon, aims to measurably improve student learning outcomes by harnessing a learning engineering ecosystem that has developed over several decades at CMU. The Simon Initiative approach uses CMU's Open Learning Initiative (OLI) courses, which are built to mimic intelligent tutors in order to provide adaptive feedback and hints during learning by doing.

"Learning by doing gives students deliberative practice opportunities to address a course's objectives," said Ken Koedinger, professor of human-computer interaction and psychology and co-coordinator of the Simon Initiative. "With OLI, students get immediate feedback. If they do not master a concept, they have to go back to re-watch or re-read and then demonstrate they have learned before they are able to move on."

Koedinger and his team set out to understand the difference between MOOCs and OLI courses, specifically whether OLI features help students learn more than MOOC lecture videos. They compared two uses of an Introduction to Psychology as a Science class: 18,645 students took it as a MOOC only, while 9,075 enrolled in it as a combined MOOC and OLI course. Eleven weekly quizzes and a final exam were given to all students.

First, the researchers compared how each group's students performed on the final exam. MOOC-only students had an average score of 57 percent, and the MOOC and OLI students averaged 66 percent. This significant difference remains after adjusting for other contributors to student success including their prior educational background and their incoming psychology knowledge.

"Do students learn more with OLI? The answer is a clear and resounding 'yes,'" Koedinger said.

Then, the team investigated how different patterns of student use corresponded with different student learning outcomes. They found that while more watching, reading and doing all predict better learning outcomes, the amount of learning associated with each activity done was six times greater than for each video watched or page read.

"Most of MOOCs' attention has been on scaling teaching -- making lectures available to more people," said Norman Bier, director of OLI and executive director of the Simon Initiative. "This study shows that students can be better served if educators and course creators focus on what we can scale -- learning. More attention needs to be placed on designing, developing and improving the learning experience in online courses -- with a focus on learning by doing that is well-aligned with outcomes and assessments."

Because MOOCs have a history of retention problems, the researchers also compared course dropout and completion rates. They found that MOOC and OLI students were 30% more likely to finish the course and take the final exam than those in the MOOC-only class. Also, participation in the weekly quizzes -- meaning students stayed in the courses longer -- was always higher for the MOOC and OLI students.

In addition to Koedinger and Bier, the CMU research team included Jihee Jim, Julianna Zhuxin Jia, and Elizabeth A. McLaughlin.

The Bill and Melinda Gates Foundation funded the courses' development and delivery. CMU's LearnLab, a Science of Learning Center funded by the National Science Foundation, supported data storage and analytics.