The Action Lab

Research meets Impact
The Action Lab is a dedicated teaching and learning laboratory within EdPlus that engages in deep research in digital teaching and learning science enabling continuous improvement in online programs which ultimately result in student success.
The Action Lab is a dedicated teaching and learning laboratory within EdPlus that engages in deep research in digital teaching and learning science enabling continuous improvement in online programs which ultimately result in student success.

New Research Frameworks: Drawing from ASU’s leadership in digital learning and ASU’s research in cognitive science, adaptive learning and social science, the Action Lab will creates new methods of digital learning research that translate into improve student outcomes at scale.

Core Values: The Action lab at EdPlus strives to achieve new insight into the learning science behind students’ learning behavior in a digital world and engage in research that drives instruction design innovation and student success.

Featured research: Making Digital Learning Work

The Action Lab at Arizona State University and the Boston Consulting Group worked with six leading universities and community colleges to analyze case study data and develop recommendations for institutions looking to scale digital learning. With support from the Bill & Melinda Gates Foundation, the resulting Making Digital Learning Work report provides actionable information that addresses student access, retention and graduation rates as well as ways to reduce operational costs for institutions. 

Good research impacts digital education

Only 59% of students who began seeking bachelor's degree at a 4-year institution in 2009 completed that degree within 6 years.

source:nces.ed.gov

The Action Lab’s work is grounded in the belief that success in educating and graduating workforce-ready students, requires reimagining the student digital service experience and determining how technology can facilitate that success. The Action Lab will employ an evidence centered design approach providing actionable data and information that results in continuous improvement of ASU’s leading online education programs.

"Research should be applicable and practical in ways that result in continuous improvement in digital learning innovation."

- Lou Pugliese, Action Lab Managing Director and Senior Innovation Fellow

Existing Action Lab projects
Digital learning efficacy

ASU Online Efficacy Study Phase I - This project compares the efficacy of 252 ASU Online courses with their on-ground counterparts.

ASU Online Efficacy Study Phase I

Research Team: Phil Arcuria, Will Morgan, Carlos Valcarcel, Tom Fikes, Lou Pugliese

The Action Lab is analyzing the efficacy of the online graduate and undergraduate platform, ASU Online, studying outcomes of the courses offered from 2010 through 2016. The focus of this phase is on 252 of the largest enrollment courses. Our research compares online courses with their on-ground counterparts at ASU, with consideration for course completion and mastery with a wide variety of demographic and learning-analytic predictor variables. Results indicate comparable outcomes for face-to-face and online versions of courses across the sample, with small advantage for online in course completion, a small advantage for face-to-face in passing, and no significant difference in mastery. These results are interpreted as evidence for the efficacy of online learning and the effectiveness on the ASU Online program in particular.

 

ASUO efficacy, Phase II - This phase focuses on the student success of ASU Online students.

ASUO efficacy, Phase II

Research team: Phil Arcuria, Will Morgan, Tom Fikes, Lou Pugliese

The overarching strategic objective of Phase II is to evaluate and inform the efficacy of the teaching and learning programs at ASU Online with the goal of increasing student success. Phase II will expand upon the findings of Phase I as it seeks to answer:

  • Are ASU Online students successful and does success vary by course and student characteristics?

  • How best can EdPlus design ASU Online courses to maximize student success?

  • How best can EdPlus deliver ASU Online courses to maximize student success?

  • How do non-cognitive factors (e.g., goal orientation, time management, etc.) relate to student success and are those factors malleable?

  • Are ASU Online students meeting the intended student learning outcomes?

The outcomes of this research will increase student success.

 

Course length and student success - Through three retrospective studies, the Action Lab examines whether course grades and completion rates differed between students taking short and long courses

Course length and student success

Research team: Tom Fikes, Phil Arcuria, Jim Cunningham

Like many universities, ASU offers courses in a variety of lengths. The Action Lab conducted three retrospective studies to examine whether course grades and completion rates differed between students taking short (6.5 – 8 weeks in length) versus long (14.5 – 16 weeks in length) courses. Nearly a million observations from 262 ASU Online, face-to-face, and iCourse (online courses offered to students enrolled in on-campus programs) courses. The second study included 7,656 student-course observations across six ASU Online courses. The final study evaluated potential differences in student success in College Algebra (MAT117). Research concluded that students in longer versions of the course are more likely to withdraw. Given the observational nature of the data, the results are suggestive and not definitive. More research is needed to determine causation.

 

Socio-economic mobility and learning gaps - An efficacy analysis suggests largely positive results for traditionally underrepresented populations of students in the digital domain.

Socio-economic mobility and learning gaps

Research Team: Derrick Anderson (Presidents Office), René Kizilcec, Tom Fikes

Access to-, and success in higher education provides a powerful engine of socioeconomic mobility. An initial efficacy analysis of the ASU Online program across more than 250 courses and 100,000 online students suggested largely positive results for traditionally underrepresented populations in the digital domain. This study focuses on ASU Online undergraduate students from lower socioeconomic backgrounds, as defined by eligibility - at any point in an undergraduate’s ASU Online career, for the Pell Grant. Performance gaps (hi vs. low socioeconomic status) are calculated descriptively and modeled for a number of grade-based outcome measures for both on-campus and ASU Online students. This allows us to assess whether high socioeconomic students are advantaged by online learning environments.

Academic probation - This project provides insight into interventions that may improve probationary student retention and success.

Academic probation

Research Team: Mark Wiederspan & Jeongeun Kim
 

ASU is dedicated to developing effective interventions to improve students’ academic success and persistence toward degree completion. This project provides insight into interventions that may improve probationary student retention and success. To uncover such findings, this project examines probationary students both online and face-to-face through two research studies. One study determines whether academic success courses at ASU improve probationary students’ GPA. The purpose of academic success courses is to help students with study habits, time management, and how to overcome personal and academic barriers. The second study investigates if probationary students are more likely to leave college than those not on probation and what factors (such as engagement with online course material and the type of classes taken) increase the likelihood of absence from required academic success courses.

Non-cognitive, social and emotional factors in student achievement - The ASU Action Lab investigates how different aspects of “soft skills” predict academic achievement, as well as post-course career development.

Non-cognitive, social and emotional factors in student achievement

Research Team: Elle Wang, Phil Arcuria, René Kizilcec, Jim Cunningham

 

This study leverages existing and planned non-cognitive assessment tools within the ASU Online environment, and investigates how different aspects of “soft skills” predict academic achievement, as well as post-course career development. This project aims at bridging the scarcity of scalable non-cognitive skills assessments in digital educational environments for learning at scale while informing design social and emotional learning program for the increasing population of online learners.

 

Targeted and Timely Interventions to Support Learners at Scale - This research identifies the best intervention methodologies for target populations of students.

Targeted and Timely Interventions to Support Learners at Scale

Research Team: René Kizilcec

Prior work finds that brief reflection and writing interventions can help students regulate their learning process and lower psychological barrier such as stereotype threat. This research project develops an adaptive intervention strategy that provides targeted support at scale by leveraging machine learning methods to identify who would benefit from what kind of intervention and at what time. The goal is to target and time interventions based on latent psychological processes such as self-regulation skills, or perceived social belonging by mining  observed student characteristics and behavior. The initial research context is the Global Freshman Academy.

Student success beyond the classroom - Ths study identifies a framework for learner data to understand long-term student success.

Student success beyond the classroom

Research team: Elle Wang, Phil Arcuria

Despite the promising outlook, few empirical studies have delved into links between large-scale online learning programs and learner post-course development such as career advancement. This deficit in research may slow our understanding of how online learning benefits learners and lead us to solutions that only benefit learners in the short-term. This study takes into account learner data before, during and after course completion and will develop a framework for collecting and analyzing post-course development data on ASU Online and Global Freshman Academy students. Following this framework, pre-course data (e.g., assessment of non-cognitive skills) and within-course interaction data (e.g., LMS clickstream data) will be married with post-course data (e.g., enrollment in advanced-level courses and graduate programs, degree attainment, participation in professional associations, career advancement, etc.) to better understand student success beyond the classroom and the factors that contribute to it.  

Open scale courseware

Online peer role model efficacy - This research identifies unique challenges faced by learners and designs effective role model interventions to improve learner achievement.

Online peer role model efficacy

Research Team: Elle Wang, Tom Fikes, Pat Pettyjohn

Past research on effective peer role models revealed that, contrary to conventional assumptions, physical presence of peer role models are not necessarily required for improving learning outcomes. Role models who are similar to learners and thus easier for students to envision are better than those who are highly successful but distant from the students. Built on these past findings, this project includes two steps toward improving learning outcomes: 

  1. Identifying unique challenges faced by ASU Online/GFA learners;
  2. Designing effective peer role model interventions that highlight both the achievement as well as primary challenges faced by online learners.

Effectiveness of Earned Admission - The Action Lab team determines the success of the Global Freshman Academy program in terms of subsequent academic achievement at ASU.

Effectiveness of Earned Admission 

Research team: René Kizilcec, Tom Fikes

The Earned Admission program allows specific student populations with insufficient qualifications for admission to ASU to demonstrate their preparedness to enroll in ASU/ASUO by successfully completing multiple introductory college courses offered through the Global Freshman Academy. We conduct a quasi-experimental regression discontinuity analysis and a multilevel Bayesian analysis to estimate this program’s effectiveness in terms of subsequent academic achievement at ASU.

GFA efficacy study - Researchers examines the efficacy of GFA courses with a focus on persistence, mastery, engagement, and non-cognitive outcomes.

GFA efficacy study

Research team: Tom Fikes, Pat Pettyjohn, Jim Cunningham, René Kizilcec

The Global Freshman Academy (GFA) offers large-scale open courses with the option of university credit, with a goal of increasing access to higher education by offering high-quality open courses that mirror the learning objectives, activities, and rigor of traditional first-year ASU undergraduate courses. This study examines the efficacy of GFA courses from 2015 through 2017, focused on persistence, mastery, engagement, and non-cognitive outcomes for the broad diversity of learners. As with MOOC courses, our the "open scale” GFA courses attract both learners to completing and a large number of "course samplers” with goals other than completing. One innovative aspect of our efficacy study is to segment learners in this way, and to compare the performance of “committed learners” to to a matched set of learners in synonymous traditional ASU courses.

Motivation and self-regulation - The Action Lab surveys learners to identify their needs and to develop custom support for high risk students.

Motivation and self-regulation 

Research team: Pat Pettyjohn, Mike Meaney

Past research identified four motivational and self-regulatory factors that positively predict student engagement and course completion: goal setting, self-efficacy, task value, and self-regulation strategies. First, goal setting involves determining what students expect to achieve during their learning engagement. Second, self-efficacy is a student’s beliefs about their ability to succeed in a course. Third, task value is the extent to which a student finds the material interesting and relevant to their broader aspirations. Fourth, self-regulation strategies comprise study habits, learning strategies, time management and other metacognitive skills. We developed surveys to assess these constructs and added them to all open scale courses as of Fall 2017. The survey data will help us identify different learning needs, tailor customized support to meet needs of high risk students, and evaluate the effectiveness of our support efforts and course design. This study explores how patterns in course behavior are associated with survey responses to identify and support students in need.

Equity and design in GFA - By working with data from the Global Freshman Academy and comparable MOOCs, this research identifies how to design and target solutions for struggling students.

Equity and design in GFA

Research team: Mike Meaney, Cambridge University

This study seeks to identify whether Global Freshman Academy will provide better access to traditionally underrepresented college students. Survey data from GFA will help evaluate the composition of students presently taking GFA courses. Initial survey results are compared to traditional MOOCs in the market to identify differences and similarities in behavior patterns, persistence trajectories, and completion rates. Through this research we seek to better understand how we might design and target interventions for struggling students.

 

Emerging technologies

Heterogeneous effects of adaptive college algebra tutoring - This research discovers how a policy and course requirement change affected student math achievement and academic trajectory.

Heterogeneous effects of adaptive college algebra tutoring

Research team: René Kizilcec, Tom Fikes

Most scientific evidence suggests that developmental (or remedial) math courses do not succeed in preparing students to master subsequent math classes. In Fall 2016, ASU removed its developmental math class, upgraded its college algebra call with an adaptive tutoring system called ALEKS, and allowed students who were failing in their first semester to “stretch” their enrollment into a second semester. This project evaluates how this policy change has affected students’ math achievement and academic trajectory overall and differentially based on socio-demographic characteristics and incoming math ability (based on math placement scores). To better understand how delivery modality influences the effectiveness of this new approach, we compare outcomes when instruction was fully online verus hybrid.

Adaptive learning and college algebra success - This study examines what characteristics define groups which excel in a newly administered math program.

Adaptive learning and college algebra success
Research team: Jim Cunningham
In April of 2016, ASU began offering an open scale version of college algebra through the Global Freshman Academy (GFA). At the heart of the course is the adaptive learning environment of ALEKS, a product of McGraw Hill. The self-paced course targets 383 math concepts that have been identified as essential to the understanding of college algebra. Since its inception, over 1,100 learners have worked through the entire curriculum of the math course. Almost every country in the world has at least one enrollee. Over 3 million algebra problems have been worked and over 2 million math skills have been tested. This course has been adapted to high school environments, schools located in refugee camps, and is being considered by some of the largest community college districts in the U.S. This study examines data from the students including survey and demographics to understand which groupings of students are successful in this course, and how student outcomes can vary depending on the environment. 

Action Lab staff

Lou Pugliese
Managing Director
Tom Fikes, Ph.D.
Director of Research Operations
Pat Pettyjohn, Ph.D.
Research Scientist
Jim Cunningham, Ph.D.
Research Scientist
Elle Yuan Wang, Ph.D.
Research Scientist
René Kizilcec, Ph.D.
Research Fellow
Phil Arcuria, Ph.D.
Research Scientist
Sammie Govindasamy
Enterprise Data Architect
Wendy Van Duyn
Admin Assistant
Mehul Shah
Research Associate
Michael Meaney
Visiting Research Fellow