Does Online Instruction Work? (Part 3)

Here is the fundamental question that public policymakers (e.g., federal and state officials, local school board members and superintendents) have to answer when making decisions that involve children and youth compelled to attend public school. Such a question, however, about the effectiveness of online instruction in raising student’s academic achievement and producing other desirable outcomes such as increased attendance, higher graduation and lower dropout rates, and college admissions—that is what I mean by “work”– gives educational leaders heartburn.

Why heartburn? Because of the tortuous role that research plays in policymaker decisions about adopting and implementing technologies in schools, especially the current clamor for online instruction. Over the past few decades, there have been thousands of K-12 studies that have sought an answer to the question.

The answers provided by scores of studies have been contested because most have had serious design and methodological flaws. Moreover, many of these studies lumped together full-time virtual schools, hybrids, and online courses, And the results have been underwhelming.  That is where heartburn enters the picture.[i]

Even when researchers over the past few decades have performed meta-analyses of a smaller number of studies that have met higher standards of quality they found that virtual instruction in its various modes, at best, is equivalent to regular face-to-face classroom instruction. At worst, some studies showed less achievement gains than traditional teaching. And keep in mind that these meta-analyses were of studies where online instruction occurred in mostly math, reading, and science courses—not other academic subjects. The overall picture is considerably less than promoters of full- and part-time virtual schooling have promised or leaders had expected.[ii]

What complicates matters is that findings drawn from research studies on the effectiveness of online instruction are only one of many interlocking tiles in a mosaic that policymakers assemble in judging whether to adopt virtual instruction for children and youth. Policymakers are often torn by the push-and-pull of conflicting impulses over determining what kinds of evidence of effectiveness matter and how much evidence is necessary to inform, shape, and justify a policy decision?[iii]

Consider the push impulse for evidence. Using research and other forms of evidence to make a decision is rational, a value highly prized in life much less for policymaking where the stakes in power, influence, and resources run high. Gathering, sifting, and analyzing data to make a personal, professional, or organizational decision is what is expected of those in decision-making posts. So when it comes to public policy decisions, in the best or worst of fiscal times, policymakers have to make a strong public case anchored in ample evidence to convince voters and key stakeholders (e.g., school boards, chambers of commerce, teacher unions, state officials, etc.) to buy and deploy new technologies in classrooms and have students use them regularly. Evidence, including research studies, that these technologies will help students learn more, faster, and better is expected.  Furthermore, current top-down pressure among business, civic, and educational leaders for “research-based practice” and “data-driven decisions” hammers school decision-makers to have solid proof in their pockets or snazzy PowerPoint presentations filled with studies that tout the effectiveness of students receiving online instruction.[iv]

Where the pull impulse enters the policy picture is that these very same policymakers have an equally strong tug to buy and deploy glittering new technologies as quickly as possible without waiting for researchers to come up with supportive findings. At professional conferences, national leaders pitch the virtues of “revolutionary” changes springing from virtual instruction and placing new technologies in classrooms. Pulled by first-hand experiences and stories of “transformations” in student learning and astute marketing by vendors, policymakers, technology leaders and school officials do not need to read research studies, visit other districts, or attend more conferences. They know in their gut the answer to the question of what works; they have faith in their intuition and, like entrepreneurs and ambitious decision-makers elsewhere, they want to forge ahead and implement virtual schooling swiftly because they believe it will help students learn.

These policymakers are not irrational. There is a political logic in mandating online courses for every student as a graduation requirement, starting pilot tablet and laptop programs, and encouraging a principal and cadre of teachers to create a technological innovation tailored to their school They consult with key stakeholders in the community before inviting charter management organizations like Rocketship Schools to establish blended learning programs in their schools. These decision-makers do not need researchers to tell them that these new technologies “work.” They believe in their heart that they will work. Push-and-pull conflicting urges pit solid research studies against strong beliefs and leave unanswered the question of what kinds of evidence matter. Too often beliefs trump facts.[v]

In asking the fundamental policy question that too often goes unanswered about whether online instruction is effective in teaching and learning, it helps to examine other reforms that have “worked” where responsible decision-makers did not rely on stories, beliefs, and intuitive snap judgments but, instead, were guided by solid research evidence.

Take preschool education. Well-designed study after study of three and four year-olds who were in preschool programs (e.g., Perry pre-schools, Abecedarian) followed their progress through schools and into adulthood. These studies show short- and long-term gains in academic achievement, adult behaviors, and post-graduation earnings. Or consider the research and evaluations of career-technical academies where students get prepared for both college and career.  Researchers doing experimental and quasi-experimental studies have found over the past four decades a range of positive student outcomes for graduates of these programs.[vii]

Considering these examples where research studies in K-12 schooling clearly show that certain investments in programs and practices work, why do so few policymakers who buy, deploy, and mandate different forms of virtual schooling seldom cite studies to determine whether online instruction is effective? Why is the return on investment of taxpayer monies so often ignored?

Three reasons come to mind.

Research results are scant and mixed. As stated above, the results of meta-analyses of K-12 studies do not show a decided edge for students taking online courses or in virtual full-time schools performing even marginally above students who are in teacher-led classrooms.  More striking, however, is that only a few studies of virtual instruction in K-12 schools meet the minimum quality threshold for design, sampling, and methodologies. From a recent (and often cited) meta-analysis of studies, researchers stated:

Few rigorous research studies of the effectiveness of online learning for K–12 students have been published.  (original italics). A systematic search of the research literature from 1994 through 2006 found no experimental or controlled quasi-experimental studies comparing the learning effects of online versus face-to-face instruction for K–12 students that provide sufficient data to compute an effect size. A subsequent search that expanded the time frame through July 2008 identified just five published studies meeting meta-analysis criteria.[viii]

The authors of the meta-analysis conclude that these five studies:

comprises a very small number of studies, especially considering the extent to which secondary schools are using online courses and the rapid growth of online instruction in K–12 education as a whole. Educators making decisions about online learning need rigorous research examining the effectiveness of online learning for different types of students and subject matter as well as studies of the relative effectiveness of different online learning practices.[ix]

In short, given the results of the few K-12 studies that have been done, there is clearly insufficient evidence to launch major online initiatives in either elementary or secondary schools. For those policymakers who seek to appear rational and prize research findings, the pantry is nearly empty.

The research being done is shoddy. While only a few analyses of online instruction approach the gold standard of experimental or quasi-experimental studies, a great deal of research has been (and continues to be) done. Unfortunately, much of it is poor quality. Most studies fall far below minimum standards researchers have established to determine the effectiveness of an educational program or procedure. Bias is evident in the sampling of students and teachers included in studies. Bias also appears in studies funded by technology vendors. Moreover, there is far too much reliance on teacher and principal surveys and self-reports of student engagement and achievement. Finally, among those studies that claim higher test scores as a result of online instruction, few studies control for obvious factors that could explain the rise in test scores. [x]

Slip-shod research, of course, has seldom stopped champions of online instruction from pressing policymakers to include such studies in their recommendations and use such research to persuade practitioners of the merits of virtual schooling for children and youth. Thus, poorly designed studies loaded with lethal flaws that show student gains in test scores often made media headlines for millions of readers and viewers while occasional well-designed studies that show modest or no gains turned up in academic journals read by a few hundreds researchers.

Nonetheless, policymakers have decided again and again to have more and more elementary and secondary school students in blended schools and taking online courses to solve one or more of the problems described in Part 2.  Were sensible observers of the contemporary policy scene to watch top-level district, state, and federal leaders push ambitious virtual programs, these observers would note the frequent absence of convincing evidence for the sharp expansion of online instruction. These observers would easily conclude that decision-makers made policy on grounds other than research findings. They would hardly miss that when policymakers did cite research studies in making the decisions, citations would be selective and, more often than not, had justified a policy already decided upon. Why is that? Here is where I offer a third reason for the minor role that research plays in making decisions about virtual learning.

Symbolic, political, and budgetary reasons carry far more weight in making policy decisions about online instruction than research findings.

State and local school boards and superintendents adopt elements of virtual schooling because they want to be seen as technologically innovative and ahead of other districts. In this culture, the value of technology is equal to social and economic progress. Even the term “high tech”—like high fashion, high church, high class, high society—conveys a whiff of superiority relative to “low tech” methods and materials. Symbolically, high-tech is high status. Students using new technologies signals that schools are modern, up-to-date, and preparing the next generation to enter higher education or go directly into the labor market with sufficient skills and knowledge to find jobs. Being in the vanguard of innovation—schools buying iPads for every kindergarteners—signals voters, taxpayers, and parents that the district wants to raise achievement through engaging students while bringing the real-world into classrooms to prepare children and youth for an information-driven economy. Not adopting new technologies, even when funds are short, sends a clear message that district leaders are failing their students, mindlessly reinforcing traditional instruction, and neglecting grave educational problems. (xi)

For policymakers to be seen as ahead of the game in technology garners public support. Too often school critics forget that local boards of education are completely dependent upon voters for funding. Those school boards that rely upon local and state resources to raise funds for schools are politically smart, then, to buy computers, whiteboards, and expand online instruction as high-status symbols to cement community support for future tax levies and bond referenda. They are also politically smart in spending monies to adopt and implement virtual schooling because in the long-term—about a decade—it will reduce significantly the cost of schooling children and youth.

Finally, policymakers also know that business, civic, and community expect of them unceasing efforts to improve students’ academic performance through better school organization, governance, curriculum, and instruction—including the adoption of technology. Since World War II, job number one has been reform. Unrelenting reform is, in short, a policymaker strategy for political survival. [xii]

For these three reasons, policymaker use of research studies on online instructional effectiveness matters little.  The truth is that even were there more than a handful of rigorously designed studies showing strong student effects from taking online courses, such results would be used to justify after-the-fact policy decisions. Of course, such solid studies are missing from the research pantry. The fact remains that no one knows for sure for which students virtual schooling works, in what subjects, and under what conditions.

If that is the case now, it does not mean it will be so forever. Recall that I pointed out how rigorous research designs, sampling, and methodologies have produced findings over time that have accumulated into convincing caches of evidence (e.g., preschool and career-technical academies) sufficient to give policymakers a rock-solid foundation for making decisions if the political conditions and resources were favorable for such policies.

Sure, that final “if” clause is crucial but it is realistic in light of the history and practice of making and implementing school policy over the past half-century. Political, economic, and social conditions influence which reforms get identified and adopted. With the current excitement over virtual learning and blended schools unlikely to abate in the immediate future and interest in spending ever larger amounts of money on online instruction, asking decision-makers about the evidence supporting expansion of online instruction is, at the least, a question that demands answers that can be reviewed and analyzed publicly.


[i] Gene Glass, “The Realities of K-12 Virtual Education,” p. 5.

[ii] Cathy Cavanaugh, et. al. “The Effects of Distance Education on K–12 Student Outcomes: A Meta-Analysis.” 2004 Naperville, IL: Learning Point Associates ; Rosina Smith, et. al.  “A Synthesis of New Research on K-12 Online Learning”. 2005,  Naperville, IL: Learning Point Associates; Barbara Means, et. al., “Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies,” (U.S. Department of Education, Office of Planning, Evaluation, and Policy Development, 2010).

[iii] See, for example, Charles Lindblom and David Cohen, Usable Knowledge: Social Science and Social Problem Solving (New Haven, CT: Yale University Press, 1979); Carol Weiss, et. al., “The Fairy Godmother and Her Warts: Making the Dream of Evidence-Based Policy Come True,” American Journal of Evaluation, 2008, 29(1) at: http://aje.sagepub.com/cgi/content/abstract/29/1/29

[iv] For a typical example of calls for practitioners to use data-driven decision making, see:  Pamela Shorr, “10 Things You Always Wanted To Know about Data-Driven Decision Making,” Scholastic Administr@tor, September 2003, at: http://www.scholastic.com/browse/article.jsp?id=423

[v]Michelle Davis, “States, Districts Move to Require Virtual Classes,” Education Week, October 9, 2011; Kelsey Sheehy, “States, Districts Require Online Ed for High School Graduation,” U.S. News, October 24, 2012 at: http://www.usnews.com/education/blogs/high-school-notes/2012/10/24/states-districts-require-online-ed-for-high-school-graduation

[vi] Districts sometimes give up laptops. See Winnie Hu, “Seeing No Progress, Some Schools Drop Laptops,” New York Times, May 4, 2007; Jonathan Schorr and Deborah McGriff, “Future Schools,” Education Next, 2011 at: http://educationnext.org/future-schools/

[vii] James Heckman, “Skill Formation and the Economics of Investing in Disadvantaged Children,” Science, Vol. 312,  June 30, 2006, pp. 1900-1902; James Kemple, “Career Academies: Impact on Work and Educational Attainment,” March 2004, MDRC

[viii] Barbara Means, et. al., “Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies,” (U.S. Department of Education, Office of Planning, Evaluation, and Policy Development, 2010), p. xiv.

[ix] Ibid., p. 54.

[x] Yong Zhao et. al., “What Makes the Difference? A Practical Analysis of Research on the Effectiveness of Distance Education,” Teachers College Record,  2005, 107(8), pp. 1836-1884; Means, et. al., “Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies.” For an example of vendor bias in reporting research, see: Intel White Paper: Education Transformation, “The Positive Impact of eLearning—2012 Update.” A capsule summary that contains the both the high-and-low of research on online instruction as well as studies of educational technology is on p,8:

While few rigorous experimental or controlled quasi-experimental studies on eLearning’s benefits have been published, a critical mass of evidence indicates that investments in eLearning can deliver substantial positive effects.

[xi] In the discussion on symbolism of technology in K-12 schools and in the larger culture I draw from Kathryn Henderson, On Line and On Paper (Cambridge, MA: MIT Press, 1998, chapter 8; John Meyer and Brian Rowan, “Institutional Organizations: Formal Structure as Myth and Ceremony, in Walter Powell and Paul DiMaggio (Eds.) The New Institutionalism in Organizational Analysis (Chicago: University of Chicago Press, 1983), pp.41-62; Jeffrey Pfeffer, “Management as Symbolic Action: The Creation and Maintenance of Organizational Paradigms,” Research in Organizational Behavior, 1981, 3, pp. 1-52.

[xii] Frederick Hess, Spinning Wheels: The Politics of Urban School Reform (Washington, D.C.: Brookings Institution Press, 1998).

About these ads

30 Comments

Filed under school reform policies, technology use

30 responses to “Does Online Instruction Work? (Part 3)

  1. I wonder if any of those studies looks into the “hidden curriculum” portion of school and whether online education is effective in helping students develop “character”, perseverance, and other traits that are sometimes explicitly and sometimes implicitly among the goal of schools.

  2. Marilyn May

    As a retiring teacher and former Apple Classrooms of Tomorrow teacher, I have been happy to find your blog. You have always been a sane voice, one that I have followed for years as I integrated technology into my classroom and taught other teachers how to do so. Our 5th through 8th grade middle school is planning a one-to-one device program next year. The explicit reasons for this appear to be the ones you stated above: “to cement community support for future tax levies and bond referenda” and “not adopting new technologies, even when funds are short, sends a clear message that district leaders are failing their students, mindlessly reinforcing traditional instruction, and neglecting grave educational problems.” However, there is also bit of a hidden agenda. Students will be required to take the new Smarter Balanced Test in our state next year and it is dependent upon students using a device to complete the test. I imagine ours is not the only school district or state where this is the reality.

    I hope that the powers that be are reading your blogs and asking themselves questions about some of the points you so intelligently raise.

    • larrycuban

      Marilyn,
      Former ACOT teacher–attaway! So much was learned through ACOT experiences about integration of technology into daily teaching but little of which had been taken up by technology-driven policymakers and administrators eager to get their hands on the latest new thing.

      Your point about the coming of online tests in the next few years for ACT, SAT, and to determine if Common Core standards in math and reading may be behind a current surge of districts buying bulk trailer loads of tablets and laptops is one I had not considered.

      Thanks for the comments.

  3. Ian Rae

    The lack of good research is really unfortunate. Is that also true in other educational issues, such as ‘new math’, ‘whole language’, and other trends of the day?

    In a larger context, education may be going through the same disruption that other industries went through:

    “In industry after industry, Christensen discovered, the new technologies that had brought the big, established companies to their knees weren’t better or more advanced—they were actually worse. The new products were low-end, dumb, shoddy, and in almost every way inferior. But the new products were usually cheaper and easier to use, and so people or companies who were not rich or sophisticated enough for the old ones started buying the new ones, and there were so many more of the regular people than there were of the rich, sophisticated people that the companies making the new products prospered. Christensen called these low-end products “disruptive technologies,” because, rather than sustaining technological progress toward better performance, they disrupted it.”

    from: http://www.newyorker.com/reporting/2012/0/14/120514fa_fact_macfarquhar

    • larrycuban

      As you do, Ian, online instruction enthusiasts quote Christensen and others who seek the spread of virtual schooling in K-12.

      • Ian Rae

        Wouldn’t exactly call it enthusiastic to quote someone saying virtual schooling will be worse :) Christensen’s quote is evidence that something has happened to almost every other part of society, so is it likely that education will escape? If one concludes that it is unlikely, then adaption is perhaps a better strategy.

        Who knows — I’ve been reading about Finland’s excellent education system which combines quality and lower-cost (or at least lower cost than some US or Canadian jurisdictions). And no standardized testing.

      • larrycuban

        Thanks for commenting, Ian.

  4. Pingback: Does Online Instruction Work? (Part 3) | OTC | ...

  5. As an online educator, I can agree with many of the comments described in your blog. The sector of online/virtual and blended learning modalities are in their early days. Impressive forwards steps are being made rapidly, especially in the last five years but concrete data is still elusive. We are in an innovation period of development and the long term role of virtual education (part or full time) in the educational landscape is unclear.

  6. Pingback: Does Online Instruction Work? (Part 3) | iEARN ...

  7. Bob Calder

    So there is this thing that we don’t know much about, which may or may not work. And we are inviting a company that is expert at doing this thing better than anybody else into our actual workplace because we admit that we are a failure at not doing it. Is that right?

    • larrycuban

      Yep, you got the situation nailed down, Bob.

      • Bridget

        And in the meantime, we may or may not actually be educating the students “virtually”. How do we “fix” the virtual miseducation, if indeed the research eventually finds that is the case??? Seems like a dangerous experiment. Can we afford the costs to children’s lives for those programs that are failures?

      • larrycuban

        As you know, Bridget, much educational policy that is reform-driven is shaped more by political values (e.g., Common Core standards will produce skilled graduates who can compete for knowledge-based jobs) and far less by research results on their effectiveness as policies.

  8. Pingback: Does Online Instruction Work? (Part 3) | Educat...

  9. Pingback: Does Online Instruction Work? (Part 3) | Docenc...

  10. Pingback: Does Online Instruction Work? (Part 3) | Aprend...

  11. Pingback: Does Online Instruction Work? (Part 3) | El rin...

  12. Pingback: Does Online Instruction Work? (Part 3) | Jewish...

  13. Larry:

    You say “Because of the tortuous role that research plays in policymaker decisions about adopting and implementing technologies in schools, especially the current clamor for online instruction.”

    If only that were true. You need to look at the lobbying efforts the commercial online program providers like K12, Inc put into getting that legislation in place to enable state support for online schools. Your post missed that point. In many states its commercial (aka for profit) interests that are driving the policymaker decisions, nor research, not quality, but pure and simple lobbying and all that that entails.

    Don’t confuse the issue by ignoring the basic way decisions are being made, rather than how we wish they were being made.

    Be well
    Ray

  14. Hi Larry,

    Very thoughtful post. May I repost it on my blog?

    http://educationadvocating.blogspot.ca/

    Brad

  15. Pingback: Just Another Meaningless Policy Paper | educationrealist

  16. Pingback: Ready or not, K-12 is moving to increased online instruction | Cost of College

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s