Answering the Big Question on New Technology in Schools: Does It Work? (Part 1)

What drives many district and state technology leaders bonkers is being asked time and again by their school boards, superintendents, parents, and media: What does the research say about whether we should invest in iPads, tablets, and 1:1 laptops?  What they really want to know is: does the new technology work? Is it effective?

Of course, some district technology leaders and superintendents already know the answer; they forge ahead to buy iPads for kindergartners in Auburn (ME) or hand out over 6,000 to every high school student and teacher in Lexington  (SC). They do not need researchers to tell them that these devices “work.”  They believe and know that they will work.

For those technology leaders, however, who want to provide credible answers to the inevitable question that decision-makers ask about the effectiveness of new devices, they might consider a prior question. What is the pressing or important problem to which an iPad is the solution? Asking that question first uncovers the confused set of purposes that surround buying and using high-tech devices in classrooms.

Drawn from reports of superintendents, school board members, and technology champions, here is a short list of those problems that will get solved by 1:1 laptops and other devices.

*These devices will motivate students to work harder, gain more knowledge and skills, and be engaged in schooling.Engaged students will achieve higher grades. When the  Auburn (ME)  school board authorized the purchase of  iPads for kindergartners, their leaders assured them that reading scores would rise.

*Students will be prepared for an information-driven labor market. Or as one superintendent put it: “Students have to have digital competence, and to be competent, you have to have access. Using current-day technology should be a normal part of what we do. We need to close the gap between schools, education and the real world.”

*High-tech devices will erase the gap in access to knowledge that exists between poor and wealthy. The superintendent who bought 6,000 iPads said:  “It’s an equalizer. There’s no difference in learning advantage from the poorest to the most affluent.”

*Using laptops and tablets will transform traditional teaching.

There are, then, many immediate problems that high-tech devices and software could solve. Now the question of whether these hardware and software solutions “work” can be asked. These varied solutions attribute great powers to these devices and software to solve problems of  students disengagement,  lack of job preparation, the gap between the worlds that children and youth experience inside and outside school, socioeconomic differences in access to knowledge and skills, traditional classroom lessons, and, finally, the problem of low academic achievement of U.S. students.

That is  one big heap of purposes squeezed into the question of whether 1:1 tablets and laptops “work.” Especially since there is hardly any research evidence that these high-tech devices solve these complex problems.

Think for a moment about investments in schools that do have a solid basis in research evidence. Where research clearly shows that certain practices do, indeed, “work.” Take preschool education. Study after study done on three and four year-olds who were in preschool programs (e.g., Perry pre-schools, Abecedarian) and their progress through schools and into adulthood show short- and long-term gains in academic achievement, earnings, and other behaviors. Or consider the research on career-technical academies where students get prepared for both college and career.  Researchers have found over the past four decades solid results for students who have graduated from these programs.
When it comes to research supporting major purchases of laptops, tablets, and similar devices, such a cumulative body of evidence is missing-in-action.
So if the research pantry is nearly empty, why do districts buy iPads?
They want to use hardware and software to solve difficult problems. But school boards and superintendents also buy high-tech devices because they want to be seen as technologically innovative and ahead of other districts. In this culture, the value of technology is equal to social and economic progress. Because school boards are completely dependent upon the political support of their parents, taxpayers, and voters to fund annual budgets, being seen as ahead of the game in technology garners public support. Not to adopt new technologies, even when funds are short, means that district leaders are failing their students and against progress.
So the truth of the matter is that research studies that show positive effects of technology hardly matter.   Occasional studies that do show promising results for new technologies are dragged in to cover the near nakedness of research, much like a fig leaf, to justify the high costs of these new devices in the face of little evidence. The fact remains that no one knows for sure whether the new hardware and software appearing in schools work. They are all beta versions with glitches that teachers and students end up discovering.

94 Comments

Filed under leadership, school reform policies, technology use

94 responses to “Answering the Big Question on New Technology in Schools: Does It Work? (Part 1)

    • Nikkol Bauer

      We’re not ready. We are overwhelmed with high-stakes accountability, implementing new standards, closing gaps, etc. Until we start focusing education on gaining [21st century] skills as opposed to regurgitating facts and performing simple computations, technology won’t make much of a difference.

      • Agreed. Still, even if there’s something disorderly in the way technologies are implemented I believe some kind of a priority will come out eventually and whatever is useless will be discriminated from whatever else that has a certain value.

  1. jorge werthein

    agree!; we need more articles like these ones to keep the discussion alive in such trascendental topic.

  2. What astounds me truly Larry, is that the superintendents and others holding the purse strings, still keep falling for the “snake oil salesmen.” And I assure you, that latter phrase is one frequently used by colleagues I’ve worked with in the business itself!

    The one element in this picture I think you’re underestimating, is the power and influence of the techno-zealots and “gurus” in this field and how they have been so effective at exploiting new technology to subvert conventional academic practice and publicise their views. I could point you to the “uber guru” in this respect in the UK, who has a whole string of awards behind him, but if you could find one single, peer reviewed, academic paper by him, published in a respectable journal, I’d be delighted…because I can’t! You will find plenty of online material and journalism, self-marketing, but nothing any serious academic would regard as “research.”

    My favourite example of this is the “Shift Happens” video Carl Fisch created. Hugely influential (some of the world’s leading companies made their own versions) yet most of the stats it contains are pure, unverifiable fantasy.

    • larrycuban

      Thanks, Joe, for the comment, especially on “gurus”in the field. What’s puzzling to me in your comment is whether or not Carl Fisch has academic research papers published in peer-reviewed journals. I understand that it may be irritating to academics and others in the field that someone is popular (in the U.S. David McCullough is a popular writer of history and not an academic. He has been attacked for writing award-winning best sellers on the Revolutionary War, George Washington, etc. —see http://hnn.us/articles/12073.html–by historians who find his “history” of an event or person inaccurate). But being popular is not the same as influential; being popular means that the person is articulating something creatively or cleverly already believed by many people and its sells. Popularity is not equal to either truth or worthiness. Always been that way, always will be–at least it seems so to me.

      • Jerrid Kruse

        I love the notion that popularity & influence are not necessarily correlated. Here’s to being unpopular.

        A colleague & I are just embarking on research in the area of admin decision-making regarding tech. We presented the paper linked below at a conference last fall that may be of interest. It discusses what newly minted aspiring admins think about when considering tech adoption in schools.

        http://m.box.com/view_shared/603zd2t39oz9tz2an40n

      • larrycuban

        Thanks, Jerrid, for sending along the paper. And for your comment as well.

      • Larry, I don’t know anything about Carl Fisch’s academic credentials. I believe he was responsible for ICT at a high school when he produced “Shift Happens.” What is so significant is the way it was so rapidly taken up by all sorts of individuals and companies because they were so keen to hear its message. No one stopped to actually check if it had any value behind it. When I examined it, this is what I discovered. Forgive the huge quotation, but it’s needed to make the point.

        “Of the various claims in Shift Happens, most of the sources were web-based blogs, or other presentations, and are wholly unverifiable. In the source document which now accompanies the presentation on Fisch’s own website, the phrases ‘don’t know,’ ‘can’t verify,’ ‘couldn’t verify’
        and ‘presumably,’ occur repeatedly. The few references that refer to books make no reference to page numbers but send the user to Amazon’s main publicity page for the book concerned. Of the 22 URL citations to what Fisch refers to as ‘My Source,’ 11 no longer function and 10 of these
        are to the same presentation by one individual. Of the 13 URL citations Fisch refers to as ‘Original Source,’ three no longer function, two send the user to Amazon’s main publicity page for the book concerned, three are to online magazine articles and the rest to presentations or blogs.”

        It is a perfect example of how the technology itself subverts real research. The original presentation used the sound track to the film of ‘The Last of the Mohicans’ (until the copyright issue stopped Fisch using it) and a clever, rhythmical cutting. Far more appealing to the kinds of people who want to hear its message than reading a substantial paper in a serious journal.

      • larrycuban

        Joe,
        Now your analysis of sources for “Shift Happens” is damning, to say the least. Thanks for providing that background. I have not heard it cited often by U.S. folks when adopting the latest high-tech device but its popularity after it first appeared spiked. Its influence, always hard to track, is uncertain.

      • Joe,

        You make an important point about how technology can subvert serious research. Some friends wrote a piece (Olson & Clough, 2001) that I believe appeared in “the clearinghouse”. The piece noted how tech may also subvert serious study. You may be interested.

        Do you have a reference or link for the quote you used about Fisch’s lack of research? I’d be appreciative.

      • Larry, what I found disturbing about the “Shift Happens” example (at one point it was ubiquitous at every event I attended in the UK) is that the very people who are so keen to proffer “digital literacy” as a new part of the curriculum, showed not the least sign themselves of possessing any kind of “literacy” skill in reading Fisch’s presentation.

      • larrycuban

        Now that, Joe, is depressing.

      • Hi everyone. I thought it might be nice to share a few brief thoughts and include some links so that everyone could easily find the presentation/sources/person that you’re talking about.

        First I think we need to a bit more clear on purpose. “Did You Know/Shift Happens” was never intended to be an academic paper nor research on student achievement, so I find the argument that it fails to meet those standards somewhat unhelpful. It was simply meant to be a conversation starter at my school talking about changes in our world and how that might impact education – http://thefischbowl.blogspot.com/2006/08/did-you-know.html.

        Second, being compared to over the course of several paragraphs as a “snake oil salesman,” a “techno-zealot” and a “guru”, who is trying to “subvert” conventional academic practice seems a bit over the top and also not particularly helpful. We disagree on a few things, sure, but has name-calling ever really helped advance the discussion?

        Third, yes some of the sources were somewhat suspect, as I commented on here – http://thefischbowl.blogspot.com/2007/03/over-two-million-served.html. But, for the most part, I think it holds up fairly well for what it was intended for. Mr Nutt, I’m sorry that almost 6 years later some of the links don’t work, but they did work at the time and certainly there was no intent to deceive.

        Fourth, I’m a little confused about Mr Cuban’s comment about popularity and being influential. We should be clear (again) that this presentation was not designed to be popular or influential, nor to “sell.” But, beyond that, your statement that “popularity is not equal to either truth or worthiness” seems to imply that it can’t be equal (especially considering your following sentence) – I disagree. I’m not saying that this presentation meets that threshold, but that seems like a stretch to say that just because something is popular means it’s not worthwhile.

        Fifth, I don’t understand Mr Nutt’s comment about “why would a brand new school . . . win a design award if technology was as crucial to learning value as the industry and the zealots insist?” Really? That argument doesn’t even make sense. Now, you can certainly argue that it’s a great school and that perhaps technology isn’t necessary to make a great school, but you certainly haven’t made that case in the comment.

        Finally, I think the crux of the disagreement is most likely over what we feel is important for students. I agree with Mr. Cuban (and probably Mr. Nutt) that many of the increases in student achievement – as defined by test scores – that folks are promising due to these technology implementations are not supported by the research, and may or may not actually occur. (Certainly in my own district we have seen some increases due to our technology use – http://www.littletonpublicschools.net/DISTRICTINFORMATION/CommunitySupport/GetInvolved/LPSBlogs/tabid/656/EntryId/412/Warschauer-Report.aspx, but your mileage may vary.)

        But many of the folks advocating for these technologies – myself included – are not looking solely at what I guess could be described as “academic achievement.” I’m definitely not saying that’s bad, I’m just asking the question of whether that’s enough. Many of us feel that technology allows our students to interact with people and ideas in important and useful ways that were difficult to impossible before, and that there are some new literacies that they need that weren’t addressed in school in previous generations. Our schools – and our curricula – were designed for an age that was very different, all we’re asking is for folks to consider that perhaps we should be having a discussion about whether schools should look different as well.

      • larrycuban

        Thanks, Karl, for your comments and links. I share your concern about name-calling and how little it advances thoughtful discussion. I am glad that you did not let it impede your comments. Your last two paragraphs, in my opinion, get at the some crucial points about technology, its worth, and toward what ends it should be bent.Thank you.

      • Sorry, also meant to include this link for the sources – http://shifthappens.wikispaces.com/Sources

      • Nikkol Bauer

        As someone in the edtech world, I congratulate both camps that seek to promote more technology and those that caution us to tread carefully.

        Fisch’s original video and the subsequent adaptations of it were meant as “food for thought” they didn’t point (in my mind at least) to technology, they pointed to a need for what I can say best as “real thinking” among our students .. our youth.

        I’ve seen how ubitiqous access can fall flat on it’s face, mostly because we aren’t prepared. I’ve also seen how others (in professions different from education and a few ed folks) have leveraged this acccess to their advantage. Why aren’t we [education] doing the same?

      • larrycuban

        How would you answer your own question, Nikkol?

  3. When schools and districts invest so much money into technology, there are so many questions that need to be addressed: How much money and time are they investing in making sure that teachers know how to effectively incorporate the technology into their teaching? What infrastructure have they established to support the use of this technology? What type of support (if any support) is available for teachers? How are they monitoring the use of technology in the classroom? Without addressing these issues, schools and districts are just pouring money into a “technology black hole.”

    • larrycuban

      Those are critical questions to ask. Add at least one more: What is the total cost of operation in implementing the new technology. Thanks for the comment

      • Another question Larry?

        What is the opportunity cost of not implementing the new technology?

      • larrycuban

        Bob,
        First, secondary meta-analyses are not yet “powerful research” studies that can guide national or district policymakers in allocating funds to purchasing and deploying new technologies.

        Second,the opportunity costs of not implementing new technologies in U.S. schools is losing political support from parents and taxpayers. That is costly since in the U.S. you have to have state and local political support to fund and maintain schools. The positive value placed on new technologies runs high throughout the culture overwhelming any weak or even negative evidence so, in many cases, school districts in U.S. embrace new technologies to sustain political support.

  4. Pingback: Answering the Big Question on New Technology in Schools: Does It ... | Facilitated classroom | Scoop.it

  5. Larry, I know you will be surprised that I agree with your statement, well, almost. To say that there is no research is a stretch. So here I go. In the 80’s there was a line of research or a debate if you will, about the effects of technology on learning or achievement if you will. In my field, we refer to this line of research as “media effects” studies. It was a great debate almost to the degree that it is today. Bob Kozma from Stanford led one side that believed that the characteristics or features of a piece of technology could effectively result in enhancing or providing a learning experience unlike a classroom without technology (I’m doing some loose paraphrasing here). On the other side of the debate was Richard Clark. Clark used an analogy to refer to the effects of technology on learning and achievement. He proposed that media (read ipads, internet, mobile technologies) are like grocery trucks making deliveries. The trucks alone had no real effects. But when groceries were loaded on the truck to be delivered, learning and achievement is directly influenced. So now your question is, what are groceries an analogy for…well it boils down to the instructional practices that teachers use when using technology. If you are using an ipad to drill a kid on facts, you won’t find much in the way of achievement but if you use it to make things that represent their understanding or to problem solve then there are positive research results. Technology only offers a way to extend or to provide educational experiences that a learner might otherwise not have. But if there is a non-technology method that is just as good, then do it. The problem is that most administrators who are in the role of making these purchases, know nothing about technology. So they fail to train teachers, to provide proper updates to the equipment and buy what’s flashy and looks good to his/her constituency. Want to talk about waste of money? How about Whiteboards? They do nothing to make any difference in the instructional practices of a teacher. A few manage to get some good out of them by actually using them in an interactive way. But for the most part they serve the role as something administrators can point to but that teachers don’t know how to use and that quickly become obsolete. Want to see 4 years of data linking a 1:1 program with improved behavior and achievement, take a look at Mooresville, NC. I have been there and observed. Was every teacher using the technology with the best instructional practices, No. But, every teacher was using technology at some level, at some point on the continuum of instructional practices. Did the kids, teachers, community, and schools have the adequate support for the program. You bet they did. So I’ll take my leave here, but to say there is no research, sigh.

    • larrycuban

      Overall, Bob,I agree with pretty much of what you stated–except for the very ending. Superintendent of Mooresville (NC) is Mark Edwards. He ran the Henrico County (VA) schools where he installed 1:1 laptops. What happened there after a decade? Research and evaluation results, at best,are mixed. Too early and too narrow a position to take–as you do–to say that a 1:1 program in Mooresville linked to “improved behavior and achievement”is because of the 1:1 without looking at other changes made in the district and the tradeoffs entailed with the sharp focus on state test score items.

      • Focusing on Mooresville again, while you may be right, that there are many other variables going on with the change at Mooresville, I do know that they have been collecting multiple streams of data since I spoke to the company that is helping them to collect their data. They seem to be aggregating that data from multiple sources which “should” provide a pretty accurate picture. But I will note that their achievement related results are tied to substantially more of their graduates attending 2 and 4 years schools than every before. I also believe that they have credibility because they didn’t hide their teachers who were using technology in very superficial way. But I will say for every one that used the technology in that manner, they were balanced by another teacher who were using instructional practices where kids were creating knowledge. I’d also refer folks to Project Red’s white paper on technology integration.

      • larrycuban

        Thanks for the elaboration of your comment on Mooresville, Bob.

  6. Larry, as always, a good read for a Saturday morning cup of coffee. I think I follow Dr. Bob on the ‘groceries’ analogy. There are great things and meaningless things teachers and students can do with technology devices.

    I have personally developed an elearning application for students with disabilities that gets outstanding results. But, I will tell you, I don’t have the time to wait for the peer-reviewed research to get completed without pushing it like a ‘snake-oil salesman.’ Unfortunately, some folks do push technology solely for profits and this muddies up the water quite a bit for me and my colleagues who do our best to maintain complete integrity.

    Unfortunatley, the peer-reviewed process is long and grueling. However, I do have a doctoral student who is doing the research and has found astounding results by the ‘groceries’ being delivered to the students. If you’d like to see what I’m talking about, took at this video http://excent.com/?page_id=1315 Because the research has not been reviewed formally, does that mean the data does not mean anything? I think not. It is also a fact this application provides a great efficiency to the delivery of content to the teachers; thus the students.

    So, I believe there are many elearning curriculums and newly developed processes that don’t have formal research behind them because the people who developed them, like me, are not big $$$$$ publishers like Pearson.

    I do agree, though, going to 1:1 computing or buying iPads with no plan for delivering meaningful ‘goods’ to the student is just a big political showcase.

  7. Pingback: Answering the Big Question on New Technology in Schools: Does It Work? (Part 1) | TFM | Scoop.it

  8. Dear Larry
    As always appreciate your perspective and arguments.
    As someone who has taught with, through and around educational technologies in classrooms for over 25 years (Yes your 1986 book was influential) I feel that you have an important role to play highlighting limits, but we need to be debating on a much grander plane than just the areas pinpointed in your views.
    Yes the shallowness of bureaucratic leadership and thinking, and the inadequacy of educational technology research should be grounds for ongoing concern. But the fundamental question remains: how best to prepare our students for the learning, work and social challenges that they are facing and are yet to face. The unknown future has been with us from year dot and with it opportunity for unverifiable contentions and risk taking. The overriding question is whether we are facing the future by just looking backwards, jumping in without due thought, or trying to make the best out of at time competing, conflicting and contrasting needs. Technology in schools is part of this and brings with it constant challenges. To view as unsubstantiated and/or superficially inevitable is not helping us think at the level we all need to add value to what has been hitherto not experienced. (with all the opportunity, risk and challenges that go with such new dynamics). To me it’s a learning value question, not a technology value question. Hope we can find a way through all this.

    • larrycuban

      John,
      I do agree with you that “it’s a learning value question, not a technology value question.” As a writer and teacher, I have tried to deal with questions and issues at where the rhetoric and action is now. By that I mean the existing hyperbole about “technology” as “transforming” devices, not technology as a process or as a way that adults and children learn. Perhaps you can suggest another way that someone who writes and teaches can better do that.

      • Larry, Joe
        Appreciate the reply and overall discussion around and through your blog. Hope you appreciate I was trying to be reflective, not critical.
        First, in reply to your last statement, speaking as an in-school practitioner with only such research experience, perhaps what is needed is a balanced research-linked publication on the value of digital technologies for learning (and school learning – which are not the same things). As I read somewhere: Science Has Nature; what do we have? The decreasing influence of relevant, academic thinking for school decision-making and the rise of marketing-led decision-making (from both within and outside schools) has unfortunately filled any void created. I agree that digital technologies have tended to be oversold (or over-attacked), but maybe as well as being underused they have also been underconsidered. The commonality of views expressed by digital technology purveyors and school gatekeepers (with teachers deliberately absented) is of continuing concern. Whenever the word ‘revolution’ or ‘transformation’ are used I think of the dictionary definition for revolution as ‘transfer of power’ and wonder, from who to where?
        Overall “Does it work” is such a big, and I think critical question, that is it so difficult to be succinct and take in all relevant factors. I see three fundamental influences at play
        (1) belief driven perspectives. In this I as a teacher believe that computers can be used to engage, support and even enhance learning. My experiences back this up. The fundamental question that drives me is Do computers add value? If one asks the wrong or inadequate question one shouldn’t be surprised by the inadequacy of any answers (I have a very Socratic view on this). Not that I’m disagreeing with your question. Just reflecting whether it should be wider?
        (2) Further to (1) we have as yet not resolved what relationship digital technologies should have in educational thinking. Discontinuities between digital technologies and school structures are well documented. The changing and challenging nature of digital technologies are seen by some as opportunities, by others as threats. I see this as a cultural question.
        (3) research and cases can be found to back-up just about any logical contention (I see this in the Finland school example [Montessori could also be used] or on “improvement” or whatever). With propensity for defensiveness that one can just about assess outcomes by the name(s) attached. Perhaps a view post-modern but the Internet challenges what we see as truth.
        I use these three personal fundamentals when reviewing what others say, and try to understand school cultures when seeking how to add-value. I was reminded of Schoefield’s 1995 Computers and Classroom Cultures research when reflecting on this.
        To round off, I lament a lack marketing-free debate and deep thinking influencing action. I appreciate this discussion and the opportunity to test beliefs.

      • larrycuban

        Thanks, John, for your thoughtful comment. As I suggested in the post, policymakers, parents, and others often ask: “does it work?” but the answer doesn’t really matter since boards and superintendents (seldom involving teachers in decision-making)have already made up their minds to get the equipment and software regardless of what the research says or does not say about the effectiveness (read: test scores) of the technology. What gives researchers and technology champions among administrators and teachers a mighty case of heartburn is the simple fact that how teachers use devices and software in classrooms is part of their pedagogical repertoire and separating out the use of these tools to establish that the hardware/software made the difference is not only nearly impossible–because of so many teacher and teaching factors involved (you mention culture of a school and classroom as one factor that needs to be considered)–but also, yeah I have to say it, a fool’s errand.

    • John, like Larry, I’m listening, but aren’t all responses about unknown futures and employment just the most utilitarian of responses?

      To pose a learning value question in return. Why would a brand new secondary school I visited only a few months ago in Finland, containing only 35 PCs for 800 pupils and all staff, including admin staff, win a major international design award design award, if technology was as crucial to learning value as the industry and the zealots insist?

      Yet, in essence I think I’m with you on the search for learning value. Which is why the one thing I think has been crucially missing, is the scale of engagement with technology by an overwhelming majority of skilled, professional, subject expert teachers. Enthusiasts, techno-zealots, commerce and most damaging, self-seeking gurus were very quick to step Into the vacuum they left. Teaching, at all levels, seems to me an unavoidably altruistic activity. Not something that preoccupies the guru mindset.

      • One last thought Joe. All this reflection on researching and valuing educational technologies in school leads to my unfortunately ongoing view that we are but tinkering about the edges. How to measure this? Judge what would be the impact on schools if all the edtechs were removed. Until we develop a value system for school learning that goes beyond test scores edtech remain optional. Through such a prism the question of value becomes judgmental even if to many, myself included, a holy grail for philosophical reasons.

  9. Again not sure whether these people satisfy Jon Nutt’s “guru” criteria but they are trying to address to do something positive about teachers capability and confidence using technology

    http://www.agent4change.net/ict-policy/continuing-professional-development-cpd-/1508-teaching-school-leaders-greet-ict-for-schools-20.html

    A bit more constructive than denigrating “shift happens” which is rather old hat!!

    • larrycuban

      Hi Bob,
      Thanks for the comments on the post and sending along the Tamin, et. al.secondary meta-analysis of studies. Joe Nutt can speak for himself about what he means when he uses the word “guru” but in my informal dictionary two academics doing such a meta-analysis of meta-analyses does not fit my sense of the term. I, for one, do not get ecstatic about the “small to moderate positive” impact on learning (read: test scores) for a few reasons. The first is what they say about their analysis: “there has been no attempt to address the methodological quality of the included meta-analyses or explain the variance in effect sizes.” That is a big missing part to me. And then to draw a policy conclusion that investing in technology on the basis of this secondary meta-analysis is over-reaching for the following reason. A far more important question to ask is: Under what conditions and in what situations is technology (if defined as devices and not processes) effective in raising achievement. The Tamin et. al., analysis does not answer those questions.

      • Thanks Larry, If you read on you will see that your “missing bit” is part 2 of the study 🙂

        So plenty of material for another few posts?

        I am not over excited either at “small to moderate positive impact” but to continually deny there is no impact in the face of such powerful research is verrging on denial?

      • Larry wrote:
        A far more important question to ask is: Under what conditions and in what situations is technology (if defined as devices and not processes) effective in raising achievement.

        Dr.Bob wrote:
        Have we already forgotten my earlier post on media effects studies from the 80’s? I thought it pretty clear that technologies are Grocery Trucks depending on the instructional strategies (Groceries) employed by the teacher when using technology. So we are talking about when technology used for the right reasons and instructional practices that are enhanced that do improve achievement. Oh well, some choose denial over healthy skepticism or are we just going in circles.

      • larrycuban

        Good point, Bob. The “under what conditions and in what situations is technology …effective in raising achievement” question should arrive at an answer that the hardware and software are a part of the overall pedagogy, the strategies and tactics teaching lesson that teachers use in their classrooms.

  10. 40yrs of evidence analysed = “where computer technology was used to support teaching, it had a “small to moderate positive” impact on learning and attitude.
    Read more: http://www.montrealgazette.com/news/Computers+schools+money+well+spent+Concordia+University+study+says/6182416/story.html#ixzz1ow0LSJ67

  11. “There were many studies that said that it actually had a negative impact. But the preponderance of evidence suggests that it has a positive impact,”said Schmid

    Read more: http://www.montrealgazette.com/news/Computers+schools+money+well+spent+Concordia+University+study+says/6182416/story.html#ixzz1ow3FJPQJ

  12. Just one of the comments made by the World Bank after they had reviewed the international research evidence, which I use in the presentations I do on this issue.

    “Bias is a very real issue in most of the monitoring and evaluation work done of ICT in education issues across the board…Such biases are often introduced at the monitoring and evaluation design stage, and include a lack of relevant and appropriate control groups, biases on the part of ‘independent evaluators’ (who often have a stake in seeing positive outcomes), and biases on the part of those evaluated (who may understandably seek to show that they have made good use of investments in ICTs to benefit education). The opportunity for such biases (which are usually positive biases) are especially acute where there is a great reliance on self-reported data.”

  13. Lets be clear here….Are you saying the Concordia Study is “biased” Joe?

    It has just been published so can not have been considered by the World bank?

    • I would have to read the Concordia study in its entirety, to make any kind of professional judgement. I certainly wouldn’t make one on the basis of any newspaper article. The World Bank’s formal review was 2005, and they have maintained their activity in the field and their conclusions since. Nothing I’ve seen anywhere in the educational ICT world has convinced me their conclusions (and bias is just one) are any less secure than they were then.

      Just to give one example. I was asked last year by a leading Australian organisation, to look at the OLPC programmes and advise them on the basis of what I discovered. Not one of the existing programmes, had put any kind of measure in place to assess educational benefits or outcomes.

      • So a world bank(what do they know given recent world economics?) report that is 7yrs old?

        The Concordia study,which I have read in full,is a meta-analysis of 40yrs of research.

        I know which evidence I prefer!

  14. Pingback: Answering the Big Question on New Technology in Schools: Does It Work? (Part 1) | Learning in a Digital World | Scoop.it

  15. Pingback: Answering the Big Question on New Technology in Schools: Does It Work? (Part 1) « Ajarndonald's Blog

  16. From my post, Schools, technology, test scores, and the New York Times (which has active hyperlinks in it), http://bit.ly/mV5uI6
    —–

    Earlier this week the New York Times wondered whether investments in educational technology were worth it since most schools don’t see any concurrent improvement in students’ standardized test scores. That’s not exactly a new issue but it’s worth examining again. After all, we are talking about large sums of money here. I’ll start with some broad categories of pushback against the article…

    1. Striving for different, higher-level learning outcomes

    It’s hard to get at critical thinking, problem solving, effective communication and collaboration, complex synthesis and analysis, and other higher-order thinking skills with a bubble test. Many schools aren’t aiming at low-level factual recall and procedural knowledge with their technology initiatives.

    2. An appalling lack of support

    Most school districts ask their technology coordinator(s) to support computers and/or people at ratios that would absolutely horrify folks in the business world. Support ratios that are 3 to 10 times higher than in other sectors don’t result in meaningful, reliable technology usage. Also, many (most?) school districts still don’t have technology integration personnel on hand to work with teachers; they just have IT support folks.

    3. An appalling lack of training

    We shouldn’t expect test score gains when few teachers have been trained well to use digital technologies to improve learning outcomes. Instead, teachers usually are just given various technology tools and, if they’re lucky, some minimal training in how to access the various features. Deep, rich technology integration training that has the potential to change educators’ pedagogy is rare.

    4. We need more technology

    There’s not enough technology in schools to adequately judge the claim that they don’t impact test scores. The average student still uses digital technologies pretty infrequently. Ask the children in your extended family / circle of friends how many minutes per week they get to use technology to further their learning in school. Most likely will say very little…

    5. Technology at the periphery leads to replicative use

    Digital technologies have yet to significantly impact the day-to-day core work of learners and teachers. Instead, we have seen mainstream adoption and growth of replicative technologies (i.e., those that allow teachers to mirror traditional educational practices only with more bells and whistles). We still primarily see learning environments where teachers push out basic information to student recipients and then assess them on the kind of stuff that you can find on Google in 3 seconds. Also, when digital technologies are used, it’s primarily teachers using them, not students. Schools still mainly buy teacher-centric tools, not student-centric tools. We’re not actually seeing technology uses that would ‘change the game’ and thus maybe ‘change the scores.’

    6. It’s the future [actually, it’s the present]

    In case we haven’t noticed, it’s a digital world out there (and will be even more so in the future). What’s the alternative to putting learning technologies in the hands of students? Is there one? Knowledge workers in the real world (i.e., outside of school) use computers to do their work. Can educators really claim to be relevant to life outside of schools while simultaneously ignoring the technological transformations that surround them, as if digital technologies were a fad that were going to go away?

    So, let’s sum up…

    We have schools and classrooms that are still doing what they’ve always done, but with some additional infrequent and marginal uses of new learning tools. We have educators who don’t really know how to use the tools very well and who also have little access to those tools, reliable IT support, and/or regular integration assistance. For some reason we expect changes in certain learning outcomes to occur anyway, despite these environmental factors and despite the fact that those outcomes may not be what the schools were striving for in the first place. And, if we don’t see those outcomes, we’re going to claim it’s the fault of the technologies themselves rather than human and system factors and then we’re going to claim that traditional analog learning environments are just fine in a digital, global world.

    Does this make sense to anybody? Apparently it does, because plenty of people chimed in to support the slant of the New York Times article…

    Wrap-up

    This has been a long post so I’ll close with three thoughts:

    A. I think that George Siemens has it right:

    If it changes how information is created…
    If it changes how information is shared…
    If it changes how information is evaluated…
    If it changes how people connect…
    If it changes how people communicate…
    If it changes what people can do for themselves…
    Then it will change education, teaching, and learning.

    Digital technologies and the Web WILL change education, teaching, and learning. Maybe not yet, at least not in the ways that we hope (and definitely not in the ways that we think). Maybe not until we get our collective act together and actually get serious about these technologies and start recognizing their learning potential and begin doing the things we should be doing to realize their affordances. Maybe right now we’re still in that place where corporations were in the 1980s and 1990s when pundits bemoaned that productivity gains were yet to be realized from technology investments, the place where we have yet to change the human and system factors sufficiently to realize the desired goals. But change is coming (and for many of us it already has).

    B. I also think that Virginia Heffernan has it right (look, also at the New York Times!):

    we can’t keep preparing students for a world that doesn’t exist. We can’t keep ignoring the formidable cognitive skills [that students] are developing on their own. And above all, we must stop disparaging digital prowess just because some of us over 40 don’t happen to possess it.

    These I didn’t have technology when I was a kid and I turned out okay or technology makes kids dumber attitudes to which Heffernan refers are both rampant and unhelpful. Again, what are we supposed to do, go back to the quill or slate? I struggle particularly with folks like Larry Cuban, who somehow can internally reconcile his statements that digital technologies have no place in P-12 learning environments (“There is insufficient evidence to spend that kind of money. Period, period, period. There is no body of evidence that shows a trend line.”) with his own admission that he has learned greatly from using the very tools he criticizes (“Learning also has come from the surprises I have found in the 1300-plus comments readers have posted. From those comments, I have received ideas I had not considered, sources sending me off to explore other topics, and counter-arguments I had overlooked.”).

    C. And, as usual, David Warlick has it right:

    There are many barriers that prevent us from retooling our classrooms for 21st century teaching and learning. But at the core is the story of education that resides in our minds. Most adults base their knowledge of schooling on their education experiences from 20, 30, or 40 years ago. It is a story that is etched almost indelibly by years of being taught in isolated, assembly-line fashioned classrooms.

    How do we retell the story of education and fashion a new image of the classroom as a rich and comprehensive environment where students learn by asking questions, experimenting with a rich and diverse information environments, and interact with people around the world — in order to discover and build knowledge?

    Right now – as evidenced by the New York Times article and its many supporters – we educational technology advocates still aren’t telling ‘the story’ very well to many educators, parents, community and school board members, policymakers, and/or the news media. That’s something we all have to work on if we ever are to accomplish the goal of making our children’s learning environments relevant to the world in which they and we now live.

    • Scott, In all seriousness, because this failure as you put it, by technology advocates to tell the story well, has cost taxpayer’s billions and children…

      If as you say, “We can’t keep ignoring the formidable cognitive skills [that students] are developing on their own” can you explain why literally millions of teenagers have been so easily gulled, manipulated and deceived by the appalling Kony 2012 video? Shouldn’t all these “digital natives” have been superbly equipped to evaluate and understand it?

      Digital reality is that their attention spans have been eroded, their natural empathy undermined so deeply, their critical faculties stifled and their appetite for immediate gratification so well fuelled, that it takes the most shocking of images and evangelical of messages (take a look) to get a response.

      • I think I understand where Joe is coming from. It seems as though he’s not necessarily in denial, a conclusion one could draw in reading his responses and ignoring the research being presented. He’s in that camp that’s referred to as, “Is Google making us Dumber.” He’s an advocate for Nicholas Carr’s point of view who contends that technology reroutes our neural pathways preventing us from reading and thinking deeply. What I find ironic is that he ignores the technology research being presented but yet seems to embrace a point of view that is totally devoid of substantive research. Go figure!

  17. Pingback: Answering the Big Question on New Technology in Schools: Does It Work? (Part 1) | Web2.0 Education | Scoop.it

  18. Thanks Scott, Such a refreshing change to read an informed and evidence based analysis from a respected educator.

    You do realise you will now qualify as a Joe Nutt “guru”?

    A fine honour in my book 🙂

  19. Pingback: Shanker Blog » Technology In Education: An Answer In Search Of A Problem?

  20. Pingback: Answering the Big Question on New Technology in Schools: Does It Work? (Part 1) | Mediawijsheid in het VO | Scoop.it

  21. Karl, the frank and fair-minded response is much appreciated and apologies if you felt the name-calling were personally targeted. It wasn’t, I assure you. But equally I’d be being disingenuous if I pretended that many people I’ve worked closely with in the business, aren’t deeply sceptical themselves about the promises and visions they frequently find themselves selling along with the kit. One reason you hear them use the terms I used. I didn’t invent them for rhetorical effect.

    I don’t doubt your honest, localised motives for creating the presentation and I’m on record elsewhere as being openly in awe of its skilful construction and rhythm… (Eisenstein would have been impressed.) My issue is with the way Shift Happens was eagerly adopted by people working in strategic or educational advisory roles, who should have known better, but who used it precisely as though it did have some kind of research value or weight. You will be aware yourself of just how many companies actually remade it because they immediately grasped its marketing value.

    I think it actually exposes a serious cultural question, one which coincidentally I’ve just written about in connection with the Kony 2012 video which has had an even more dramatic flowering. If digital literacy had any meaning at all, then the teachers who saw your slides originally would have been demanding to know what your sources were, their academic provenance and value. My guess is few, if any, did. In just the same digitally illiterate way, millions of teenagers have watched the Kony 2012 video without exhibiting anything like the critical faculties one would expect from the hordes of digitally literate, digital natives who populate the presentations, literature and fantasies of the techno-zealots and gurus I do have a serious issue with.

    As educators and teachers we have a responsibility to be honest. A lot of new technology seems to have the propensity to foster some of the least attractive of human characteristics and not enough educators have either noticed this, or stood up to say so. Take this little test and see what it tells you. Look objectively at your adult Facebook friends and pick out the ones who are obviously the most active, prolific users. What do they have in common as personalities?

  22. Dr. Cuban and others,
    Let’s assume that you are right.
    What then should be the minimum level of technology that teachers use to instruct students, and that students use to learn?
    By technology, I mean both analog and electronic.
    Thank you in advance for your thoughtful responses.
    M. Walker

    • larrycuban

      In my opinion, Michael, teachers should have access to a full array of new and old technologies (including an infrastructure of technical and professional support)and determine which hardware/software they can use to best reach the goals they set for their students in lessons and units.

      • Can you be more specific?
        Does this mean an Interactive White Board, Blackboard with chalk, an electronic device per student, a phone for the teacher? All of the tools I mention here have been part of initiatives in the past that have been derided as ineffective. I think if you are going to be critical of efforts that have failed, or have have not been shown to raise student achievement in a peer reviewed study, you should make recommendations on what we should be doing differently. Thank you again in advance for your responses.

      • larrycuban

        Michael,
        Old and new devices in of themselves are neither magical nor miraculous in what they do for teachers or students. It is what is in teachers’ heads, their expertise and experience, their beliefs, and how they reach the goals they set for student learning that matters–not the devices. Teachers who want students to collaborate can use Google docs or other software that helps students work together–you get my drift.

      • Larry
        What looks straight-forward can be shrouded in complexity. Case in point, today we are being called to respond (if not just support) the media-rich limitations that schools have to work under (see http://www.networkworld.com/news/2012/031312-bandwidth-study-257222.html for what we educators are facing). And this is only one tip of the iceberg.

  23. I agree with Larry’s response to your first question John. This is about teachers being in charge of their professional environment but I would add this. Unfortunately, skilled, professional teachers, the ones who really do the job well, have rarely taken up the challenge that new technologies have created at the highest cultural and intellectual levels. They should have been the ones asking the hard social and psychological questions about these issues. Instead, enthusiasts have stepped in to fill the vacuum and the guru mindset has exploited their enthusiasm for less altruistic reasons.

    So to answer your specific question about recommendations. (This was precisely the question I’ve just been asked here in the UK, by someone running a chain of 29 schools.) Apart from providing the same reliable service levels and infrastructure for any school, which a small business would require, what’s needed is a budgeting exercise and strategic policy combined, which encourages creative, innovative teachers to experiment with new technologies, but ONLY under strict conditions where appropriate measures are agreed to capture the benefits or value. You can expect to have 3 flops for every success, but what you will get in the end will be sustainable and most crucially, educationally beneficial.

    Last year I was asked by an major Australian organisation to advise them on funding and look at a large number of OLPC programmes running. Not one had any kind of measure or assessment process in place to evaluate the benefits or outcomes.

    • Thanks Joe
      Looks like we have a similar mindset in supporting teachers to trial and innovate first; then assess; then scale up as warranted. A big issue is trust and I in more recent times have reevaluated (upwards) the level of trust we need to give to our teachers. We can do more to acknowledge, support, connect and communicate the efforts of those teachers you identified, who often work in quite demanding structures. We have the technology to facilitate this but do we have the will? Agree also that OLPC approaches can be their own worst shortcoming; but that doesn’t mean they are in error. But I come back to the observation that we first need to reassess what we truly value as good learning and from there make appropriate judgements (including the taking of risks) And test scores as a focus is an insufficient lens.

    • Joe you really excel yourself with this one:

      “Instead, enthusiasts have stepped in to fill the vacuum and the guru mindset has exploited their enthusiasm for less altruistic reasons”

      An outrageous allegation! What evidence do you have to support this?

      You must have had some bad experiences in your brief time at RM?

      • Bob, I worked for RM for 5 years… and had a great time. Perhaps this articulate and frank comment from a visitor to my own blog, demonstrates that I am not quite alone in my “outrageous” views.

        “When ICT was introduced as a subject schools needed large numbers of teachers to teach this new compulsory subject. These teachers mostly came from other subjects and had an enthusiasm for using computers but not necessarily any understanding of the underlying technology. These converts to the new subject needed courses to teach; courses that could be delivered by enthusiasts without formal training in computer science or software engineering. Therefore ICT is incredibly lightweight, superficial and ill-defined, favouring woolly concepts and gimmicks over actual science. As someone trying to keep Computer Science alive in my own school it has been a difficult last 10 years and I will not be disappointed to see the end of ICT and hopefully its replacement by something with greater ‘disciplinary coherence’.”

      • Joe ,I thought you would be interested in the BCS announcement about Computer Science following Michael Gove speech at BETT?

        http://academy.bcs.org/category/16655

  24. Larry, I’d like to throw another thought into this discussion. I’ve already pointed out about the importance of instructional strategies to technology and I’ve noted the line of research on media effects which seems to have been lost in this discussion. So let me try again. There is a new model of expertise being researched that addresses a teacher’s expertise in integrating technology, their knowledge of pedagogy, content etc. It comes complete with an emerging line of research and instrumentation to measure. It’s called TPACK. Here’s one link:
    http://mkoehler.educ.msu.edu/tpack/

    • larrycuban

      Thanks, Bob, for the link to TPACK. I am familiar with it since it builds on Lee Shulman’s work on pedigogical content knowledge. It is useful insofar as pointing out the complexity of the cognitive work teachers must do in combining knowledge and skills in subject matter, knowledge of kids, how to teach, and connecting all of the foregoing to available hardware and software for particular lessons.

      • Bob (and all),

        We have used TPACK in our district and have found it helpful. Not a silver bullet (nothing is), but it does help us think through the entire process more deeply.

  25. “Does it work?” Why aren’t school leaders asking the same question about the pedagogical effectiveness and student learning impact of textbooks, science lab materials, math manipulatives, the type of pen/pencil and paper provided for student writing, the ergonomics of the chairs students sit in, the lighting of the classroom, the quality of the acoustics of the classroom, the distance between student seating and teacher presentation medium, the temperature of the room while learning, the food they serve during lunch, etc. – I could go on and on. The reason that most of us do not ask the same question of the aforementioned tools and resources is that we continue to think of them (sans technology) holistically and historically as just part of the classroom and the teaching and learning process. They’ve been there so long that we no longer have to question or measure their impact. Meanwhile, technology continues to be considered (now 30 years or more later) an outlier, new layer, add-on or luxury for students and teachers and therefore must be measured to determine the schools return on their investment (ROI) in them. I agree. But it must be measured as an integrated tool for teaching and learning that works in tandem with all of the others that already exist in the classroom even if it replaces one or two of them. Only then can we have a true depiction of whether or not the technology works regardless of the original purpose of investing in it Unfortunately, without the requisite but ultimately expensive, professional development and support needed for the technology to be integrated meaningfully, appropriately and effectively (let alone in a unique and compelling manner) throughout the teaching and learning process, we will never have an accurate measure of a particular technology’s impact on student learning. In the meantime, students continue to get returns on their (or their parents) investment in technology as tool and resource that enables them to learn “accidentally” or informally despite schools inability to tap the same power (see: http://digitalyouth.ischool.berkeley.edu/ ). All that being said, we are about to see a transformation of how we are able to measure the impact of a technology as a learning tool as schools begin to implement adaptive learning systems. Adaptive learning systems (http://en.wikipedia.org/wiki/Adaptive_learning) by their nature provide real-time data on student learning as a result of their use. To reference Fisch’s Shift Happens presentation – the pace of change is “exponential” and debates such as these are critical to the success of education as it moves through the process of what Clay Christensen terms disruptive innovation (http://en.wikipedia.org/wiki/Disruptive_technology). Thanks for continuing to stir the discourse Mr. Cuban with such provocative writing.

    • larrycuban

      Thanks, Gregg, for your extensive comments on the importance of integrating technology into the learning process and your strong belief in adaptive learning systems. I cannot peek around the corner to see if such systems will be adopted quickly and see “a transformation of how we are able to measure the impact on student learning.” Thanks for joining the discussion.

    • lrzuniga

      Adaptive systems are not new…neither is the concept of using real-time data driving decisions. In the corporate world it is know as Business Intelligence and now it’s bigger brother as “Big Data”.
      Can it make it’s way into education on any significant level? Hard to say as the educational world adopts change slowly and (arguably) less efficiently than other sectors (read:business). This is probably a function of the difficulty in quantifying the success of implementing new strategies, programs and/or technologies.
      In my opinion, some basics have to be put in place before automated systems can bear fruit.
      1. What is our goal? To increase test scores? Increase the frequency students use technology? Annual spend of a school or district of technology? Bandwidth consumed? Number of “Apple Distinguished Educators” on staff?
      2. What do we track and measure (in real-time of course) to ensure we are on track to meeting our goals? – this is a toughy given the myriad systems, standards, programs, databases, SIS’s, LMS’s,etc. used by schools. There is a saying in IT; “If you cant measure it, you cant manage it.”
      3. Are our administrators and curriculum folk ready to lead this? Truth be told most aren’t tech savvy enough to ward off “Snake Oil Salesmen” from tech companies nor from their own Tech Integration staff that paint a very dark picture for students if significant money isn’t spent.
      All in all, it’s a fascinating time to be in education and technology!

      ~Luis

  26. Pingback: Unfiled | Pearltrees

  27. Pingback: En respuesta a la gran pregunta en la nueva tecnología en las escuelas: ¿Funciona? | Educacion, ecologia y TIC | Scoop.it

  28. Pingback: Research & Publications | Pearltrees

  29. Pingback: 1:1 IPAD Initiative – Yes or No

  30. I am somewhat disappointed at the discussion of popular books on education. Has Jonathan Kozol been influential? Has Ted Sizer been influential? The most influential thing there is is the zeitgeist. It is the water we swim in. It is the context of all influence. Think, for example, of the issue of gay marriage. A decade ago it was something of a contradiction. Today anyone against it is labeled a fascist.

    The key point is that academic research articles and popular books engage different audiences with different concerns. It is nothing but a category mistake to think that popular books live in the same epistemic world as peer-reviewed articles. Different genre with different criteria of merit. Both do important work bnut not the same work.

    • larrycuban

      Good point, Leonard, about the different criteria and audiences for popular books and research articles. Zeitgeist, however, is an all-purpose causal answer that ignores the influences, indirect to be sure, of ideas conveyed in both popular media (including books) and, even–note the even–research when that research is directed toward crucial national goals. I am thinking of the the scientific research and development of radar, the atomic bomb, the use of computers for artillery in World War II. Shifting attitudes and behavior toward smoking for example, over the past quarter-century, was anchored in medical research. The example of gay marriage and other cultural shifts in attitude is more than the generic explanation of zeitgeist. Thanks for commenting,Leonard.

      • Always great to exchange points of view with you, Larry. My book on Ed Tech is finished and will appear later this year. Big Surprise: You are still a central fopcus of my arguments. Hope to have further exchanges this year.

      • larrycuban

        I look forward to your book on technology, Leonard. Congratulations on its forthcoming publication.

  31. Pingback: Thoughts on the late MMSD budget recommendations | AMPS

  32. Pingback: Ipad 1 Questions And Answers | Get Your iPad Now

  33. Pingback: Els grans interrogants que avui planteja Internet al Sistema Educatiu. – BLOC D'UNA ESTUDIANT D'EDUCACIÓ

  34. Pingback: Hilft der Computer beim Lernen? Ja oder Ja? « Ampelmaennchen und Todesschuesse

Leave a comment