The Lack of Evidence-Based Practice–The Case of Classroom Technology (Part 2)

In  the No Child Left Behind Act (2001), the phrase “scientifically based research” is mentioned 110 times. Not a typo. Evidence-based practice, a variation of the NCLB phrase, and data-driven decision-making are popular among policymakers, administrators, and researchers. What is common to all of these phrases is the idea that systematic inquiry into a question or problem–either through evaluation or research (or both) will yield solid data useful to educators in making and implementing policy.

Yet the historical record is rich in evidence that research and evaluation findings have played a subordinate role in making educational policy. Often, policy choices were (and are) political decisions. This is not a criticism of politics or even ideology in schooling but a recognition that tax-supported public schools are political institutions where stakeholders with competing values vie for resources.

There was no research or evaluation, for example, that found establishing public schools in the early 19th century was better than educating youth through private academies. No studies persuaded late-19th century decision-makers to import privately-funded kindergartens into public schools. Ditto for introducing desktop computers into schools a century later.

So it is hardly surprising, then, that many others, including myself, have been skeptical of the popular idea that what policymakers and teachers should do is pursue unrelentingly evidence-based policy-making and data-driven instruction. The strong belief persists among educators that when policy and practice are anchored in scientifically researched findings, then and only then, rational and effective policymaking and classroom teaching can occur.

As Part 1 indicated, that has hardly been the case when it comes to monies spent on charter schools and classroom technologies then and now. Why is that?

Political and practical reasons, not research and evaluation, often guide policy decisions–or as two scholars put it: “evidence-based decision-making is sometimes framed as an antidote for ideology-driven decision-making [when] people make decisions precisely by drawing on what might be considered ideology … as a fundamental part of the decision-making process.”

Politically smart state and local policymakers believe–here is where ideology enters the picture–that buying new tablets loaded with software, deploying them to K-12 classrooms, and watching how the devices engage both teachers and students will work; it is considered “best practice” because, well, “we believe in it.” The theory is that student engagement with the device and software will dramatically alter classroom instruction and lead to improved  achievement. The problem, of course (you no doubt have guessed where I am going with this) — is that evidence of this electronic innovation transforming teaching and achievement growth is not only sparse but also unpersuasive even when some studies show a small “effect size.”

When the research pantry is nearly empty and evidence for raising student test scores or transforming teaching is sparse, how do  policymakers and administrators justify buying new devices and software?

Here are three reasons that I see spurring decision-makers to allocate scarce dollars for new technologies.

First, keeping up with the rest of the changing world. Call it “modernization” or recasting schools as less like museums and more like fast-paced companies using technology in daily work. No more jokes about educators being technological slow-pokes. Use of new technologies is considered modern, being with-it, even an unadulterated “good” that all children and youth in age-graded schools should embrace.

Second, because new technologies are highly valued in the culture, school boards and their superintendents feel strong pressures to keep up with other sectors–both public and private–undergoing technological changes. If those leaders do not act, they fear that taxpayers and voters will lose confidence in public schools. And public confidence is like money in the bank since tax-supported public schools are politically and fiscally dependent on the good will of taxpayers.

And there is a less obvious third reason for school leaders to purchase new technologies: increase efficiency in testing and scoring results. Schools have to have computers because eventually U.S. students will be taking state tests online. The Los Angeles Unified School District’s recent fiasco with iPads was triggered by demands to implement the standardized testing required by adoption of the Common Core standards.  Just as the move from quill pens to pencils to computer-adaptive-testing required no research studies but were done on grounds of cost-saving efficiency, so it was when the LAUSD School Board and Superintendent authorized buying iPads.

Note that the three reasons I offer are political–not in any negative sense–but ones that are practical and realistic in the world that policymakers inhabit. Research findings to support the promises that school leaders make for the “good” that high-tech purchases will achieve, are simply not there. And that pattern of pursuing innovations without much evidence or data to support the decisions that school boards and superintendents make is plain to see.

23 Comments

Filed under Reforming schools, technology use

23 responses to “The Lack of Evidence-Based Practice–The Case of Classroom Technology (Part 2)

  1. As a public school teacher surrounded by “data-driven decision making” as a seemingly endless refrain, I could feel deep down there was something rotten about it. Coming upon Cambell’s Law, I feel I’ve found what was bothering me:

    “The more any quantitative social indicator (or even some qualitative indicator) is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

    Donald T. Cambell

  2. It is indeed true that there is a 50-year history of “no significant difference” of effect for technology. However, as you have noted, there are numerous examples of specific applications by particular teachers that do show a worthwhile effect. Thus, the appropriate conclusion is that there is no *main* effect of technology on learning, but there are many worthwhile *interaction* effects. This is common among education innovations, as noted more than a decade ago by David Berliner. Stated differently, introduction of technology may be a necessary condition for a technology effect on learning, but it certainly is not a sufficient condition.

    It is also important not to minimize cost-efficiency as an argument for technology in education. If we can reduce cost (and time) of administrative or even instructional operations through use of technology, then perhaps more resources and time will be available to teachers. Use of technology to improve cost-efficiency has received far too little research

    • larrycuban

      Thanks for the comment, Rob. And the point about interaction effects. I do not minimize or neglect the imperative of efficiency which drives much of the technology surge in schools in the past decade, particularly in the enthusiasm for online instruction. I do note the word “perhaps” in your next-to-last sentence. That is and has been a remote possibility rather than a probability. I have yet to see the “perhaps” occur. If you know of instances where resources saved through efficiencies were made available to teachers, please tell me.

      • The scarcest resource in any classroom is the teacher’s time. Only fairly recently have technology applications begun to make claims of time savings, ranging from electronic gradebooks, to lesson planning tools and resources, to assessment systems. However, I also await sound research on the effects of uses of technology to save teacher time. I would agree with you that this question has been largely overlooked by the research community.

      • larrycuban

        Thanks for the comment, Rob.

  3. Pingback: The Lack of Evidence-Based Practice--The Case o...

  4. Reblogged this on From experience to meaning… and commented:
    This is the second part of an interesting double bill by Larry Cuban (read the first post here.

    Cuban describes 3 possible reasons why policy-makers and decision-makers spent money on new technologies:

    First, keeping up with the rest of the changing world. Call it “modernization” or recasting schools as less like museums and more like fast-paced companies using technology in daily work. No more jokes about educators being technological slow-pokes. Use of new technologies is considered modern, being with-it, even an unadulterated “good” that all children and youth in age-graded schools should embrace.

    Second, because new technologies are highly valued in the culture, school boards and their superintendents feel strong pressures to keep up with other sectors–both public and private–undergoing technological changes. If those leaders do not act, they fear that taxpayers and voters will lose confidence in public schools. And public confidence is like money in the bank since tax-supported public schools are politically and fiscally dependent on the good will of taxpayers.

    And there is a less obvious third reason for school leaders to purchase new technologies: increase efficiency in testing and scoring results. Schools have to have computers because eventually U.S. students will be taking state tests online. The Los Angeles Unified School District’s recent fiasco with iPads was triggered by demands to implement the standardized testing required by adoption of the Common Core standards. Just as the move from quill pens to pencils to computer-adaptive-testing required no research studies but were done on grounds of cost-saving efficiency, so it was when the LAUSD School Board and Superintendent authorized buying iPads.

  5. I agree that advocates of ed-tech need to face up to the lack of evidence for effect. A recent report written for UK government by the Education Technology Action Group (http://etag.report/) explicitly avoided doing this by declaring that “Evidence is a problematic concept when thinking about digital technology in education” (p.11). Brazen or what?

    There is one caveat worth bearing in mind about evidence, though, which is that one can only test empirically what has already happened. Evidence does not help so much with the process of innovation, which must at best be based on what is plausible, combined with imagination.

    Your explanation of the problem, Larry, is that the technology has not been not been used for “promoting the hoped-for student-centered approach”. But has that student-centred approach been grounded in empirical evidence? On the contrary, the evidence coming from PISA suggests that the more teacher-led approaches still used in Asia are more effective than the child-centred approaches in the West. There has recently been a spate of well-argued books suggesting that student-centred approaches are part of the problem and not part of the solution (Daisy Christodoulou, Robert Pearce, Tom Bennett in the UK).

    True virtue is likely to consist in a middle way, with good subject knowledge combined with good teacher feedback and plentiful opportunities for creativity and originality (just what traditional, elite education always provided). The rhetoric of revolution and transformation that often surrounds ed-tech, and often used for example by online gurus like Sir Ken Robinson and Sugara Mitra, does not help here. Much of this rhetoric concerns “independent learning” and “twenty-first century skills” which present themselves as supplanting traditional subject knowledge. This does not in my view come anywhere close to honing in on the middle way.

    In my view ed-tech has huge potential in address the resourcing problems brought about by trying to deliver universal education using a traditional delivery model which, though very effective on a small scale, is hugely labour intensive and therefore very difficult to scale. But the way that we will achieve the technical and pedagogical innovation that we need is by getting politicians out of the way, and setting up an institutional framework that rewards successful innovation and recognises failure early. Evidence will find its own place in such an ecosystem. I would call it a market, though it is a word that many seem to find uncomfortable.

    I agree that there is as yet no evidence for the effect of ed-tech – and that matters. But nor is there evidence for the effect of the underlying pedagogies on which many predicate new approaches to ed-tech. And ed-tech needs not only to be the subject of research, but also the means by which research can be more effectively conducted. It can provide a medium for encapsulating and replicating pedagogy of all types, and the means of collecting data on the scale that is required to make RCTs work in education.

    Thanks for the opportunity to comment. Crispin.

    • larrycuban

      Crispin, thank you for the information on the UK report which I had not seen. Your point about the lack of substantial evidence supporting student-centered instruction (or learning) is correct. My own research for over two decades has shown that most teachers employ a hybrid approach in their pedagogy. Linking pedagogy to student outcomes has indeed shown a tilt toward teacher-directed instruction (and learning). In the U.S. one of the strongest pushes for technology came from those committed to student-centered pedagogy and that ideological bent still prevails among many technophiles–the current passion for “personalized” or “customized” learning in the U.S. is the recent incarnation of that belief. I appreciated your comment.

      • Thanks for the reply Larry. I think you sum up the situation very succinctly, here in the UK as well as in the US.

        The prevalence of student-centred pedagogy amongst ed-technophiles strikes me as one of the key reasons why ed-tech has performed so poorly, when judged by traditional measures of outcome. I also think that the determination of the ETAG report to discount evidence of effectiveness when this is judged against traditional educational objectives means that the report will be effectively ignored by the government that commissioned it.

        It seems to me that we suffer from an unclear definition of “pedagogy”. I see the term as referring to the technology of teaching, the means by which we seek to attain our ends. I cannot imagine how those who do *not* “link pedagogy to student outcomes” understand it.

        Thanks again for the conversation, Crispin.

      • larrycuban

        Thank you, Crispin, for taking the time to comment.

  6. Hi Larry,

    thanks for part two, and for the clarity and insight.

    I’d add a fourth reason for policy making decisions about technology uptake. Entrepeneurs, companies, developers, investors, and others have invested significant effort, time and money in heavily marketing their technologies to schools, parents, the media and policymakers as educational, revolutionising, liberating and cost effective. As disruptive to the broken tradition of education. As cheap replacements, as new paradigms, as vectors for essential new curriculum shifts ( 21st century skills…), as panaceas to the uncertainty that the future is inevitably going to confound us with.

    Edtech propheteering, and profiteering is, maybe, a fourth reason ( though it’s tied in with some of the others you mention).

    I’d also make a point about evidence based policy making here. The medicine for the above is not ideology. It is evidence. Evidence based practice, and evidence based decision making can, at the absolute minimum, serve to protect scant resources from being misdirected. So, in SJSU, with the Udacity experiment, it was already known that the target demographic tended to do worse in online learning than in face to face – something that Udacity found out after SJSU hired them. The one laptop per child program has already provided fairly good evidence that one ipad per child is not going to have a major impact on outcomes. There’s lots of evidence about the inefficacy of minimal guidance problem based, enquiry based and discovery learning for novices. We know enough about the milits of human cognition to demonstrate why 21st century skills should not be a discrete subject, or taught as discrete skills.

    Evidence based practice has limits, and issues, The testing that we do to estimate outcomes can have issues, innovation must be allowed some space to experiment in that has freedom to grow, alter, evolve and adapt. But, as a process for deciding the allocation of finite resources in important spheres, it;s the least worst system we have. And it is the most effective prophylactic we have against the relatively expensive myths we have most recetly funded.

  7. jeffrey bowen

    For classroom teachers especially, time is a precious commodity. The time wasted when technology does not work is an unconscionable loss of learning engagement. Way too often the equipment or the electronic links just won’t cooperate. zGod knows what chaos systemic online testing will create. Have empirical investiations been conducted to see what depressive effects on achievement have been induced by balky technology?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s