Image via Wikipedia
After all those clinical trials in medicine to conquer depression, Alzheimers, and cancer, we are told now that Prozac and other anti-depressants work no better than placebos and doing crossword puzzles, exercising, and taking fish oil does not prevent Alzheimer’s disease. Moreover, studies have found that mammograms, colonoscopies, and PSA tests are far less useful to detect cancer than we had been told. The truth is that even with the toughest standards used by medical researchers to demonstrate one drug, one therapy, one test is better than other treatments it is very difficult to prove anything scientifically or even trust that those studies are accurate. Consider cancer.
President Richard Nixon declared war on cancer in 1971. Since then medical entrepreneurs have promised that surgery, radiation, and chemotherapy will cure the scourge. Molecular biologists and genome specialists have identified cell mechanisms and genes that trigger particular cancers. Yet after billions of dollars have been spent in cancer research, the disease with its remarkable heterogeneity remains at the top of the list of leading causes of death in the U.S. In short, politics, policy, and research have failed to cure cancer. But wait, there is even worse news.
One medical researcher and statistician, John Ioannidis, claims that 90 percent of published medical research that doctors rely on is flawed. Ioannidis and others raise serious issues about biases in research designs and the directions researchers pursue. The big unasked question remains: Can medical research studies be trusted?
Now, if doubts are raised about medical research where the highest statistical and ethical standards are used in studies on human subjects, what about educational research?
The sorry state of policy and practice research in education has been mentioned numerous times. Every important (and unimportant) issue in schooling children and youth has studies that say one thing and studies that say the opposite. Biases of researchers–as in medical investigations–often taint designs and methodologies. Efforts to make educational research more scientific and relevant to policy and practice have occupied scholars in both education and the social sciences for decades (see PDF EDUCATIONAL RESEARCHER-2002-Feuer-4-14).
To raise research standards, the federal government has launched a “What Works Clearinghouse” (WWC) to evaluate studies investigating educational policy and practice. Using tough standards of evidence, WWC determines the degree of effectiveness of curriculum, software, and intervention programs across 11 areas (e.g., Adolescent Literacy, Character Education, Elementary Math, English Language Learners). The stark results are eye-openers. Of nearly 75 Elementary Math programs, for example, six programs had mixed outcomes and only one had positive result for a single outcome (“Everyday Mathematics”). The other interventions, including software programs, had neither a report available nor had studies been done.
Stuck with results like this, educational policymakers, practitioners, and parents, ask: Can we trust research to point the right direction for practice?
As a high school teacher and district administrator for nearly 25 years and a researcher for another 25 years, part of me says yes and part of me says no.
In general, I value research. I have asked questions investigating the history of teaching, curriculum, and school reform (including technology). I have designed studies, and, using different methodologies, collected evidence and published my findings. I know that truth is elusive and that biases, including mine, can taint even the best designed study. Still, a careful, rigorous, and honest search for truth in different venues from preschool through the university is essential, I believe, for improved teaching and student learning.
Then the practitioner part of me kicks in and says that so much educational research fails to ask, much less answer, puzzling questions that teachers, principals, and superintendents face daily. Instead,to get answers to these questions, hardworking professionals have to rely on their experiences and the wisdom of peers, as I had done.
I have worked in both worlds and find it tempting to agree with those studies that support my biases while rejecting those that challenge those very same biases. And when research findings are mixed, I am tempted to ignore the findings. So I am torn by conflicting values. In truth, what I often end up doing–the compromise I have worked out–is to rely upon my experiences in classrooms and schools while keeping an eye peeled for rigorous, high-caliber studies.
In the face of growing disenchantment with medical research studies and the disarray of educational research, I have learned to trust my experiences. Though I am an avid consumer of educational research, I remain skeptical of much of what I read.
Related Articles
- You: Why Almost Everything You Hear About Medicine Is Wrong (newsweek.com)
- “Prominent scientist says almost all published medical research is flawed” and related posts (depthreporting.com)
- Human Farming & the Limits of Medical Research (thehealthcareblog.com)
- Lies, Damned Lies and Medical Science (kir.com)
- Right Diagnosis, Wrong Prescription: New Medical Research Center (innovationandgrowth.wordpress.com)
Larry Cuban on School Reform and Classroom Practice 
Pingback: Tweets that mention Medical and Educational Research: What To Believe? | Larry Cuban on School Reform and Classroom Practice -- Topsy.com
I think that education is generally not like medicine; it is easier to make subjective judgements of the effectiveness of interventions and research serves to inform our experiences and experimentation in the classroom rather than to provide the final word on an issue.
The trouble I have with education research is not unconscious, unintentional bias but blatant, pre-meditated bias and obviously invalid methodologies in studies that nevertheless get into print because they serve one ideological agenda or another.
The other thing I have noticed since I started reading more about education research is how often people running schools, or just commenting on educational issues, simply lie about what the research shows. I think a lot of teachers have developed an instinctive scepticism about any claim that rests on educational research because of this, certainly I never cease to be amazed at the responses I get when I ask for an academic reference for a claim about what the research shows.
Probably worth having a look at this discussion:
http://www.badscience.net/2010/07/boris-johnson-and-his-innovative-trial-methodology/
Dear Joe and TeachingBattleground,
Joe: I agree that researchers continue to investigate different forms of teaching (e.g., project-based teaching, direct instruction, interactive whiteboards) and the biases often show at the seams and in the stitching. TeachingBattleground: Thanks for the reference to the badscience website and the goofy solution to the phonics debate in reading.
I would absolutely endorse Teaching Battleground’s irritation at, “blatant, pre-meditated bias and obviously invalid methodologies in studies that nevertheless get into print because they serve one ideological agenda or another.” But the subtleties of language are such that even those claiming a kind of pristine objectivity (Ben Goldacre’s Bad Science) come with just as much ideological baggage as anyone else.
In the Boris Johnson article he claims, “we make no attempt to cheaply and systematically assess the teaching profession’s various education methods,” yet I have just spent many days, wading through a morass of research material which does just that.
Hmmm, judging by the responses I should probably point out that I was suggesting people read the debate on that Bad Science blog, as opposed to endorsing the content of the post.
In 2005 I attended the AAAS annual meeting and enjoyed the diverse community of interdisciplinary symposia, workshops, and talks. The education sessions were informative but because it was an interdisciplinary meeting, there was much to be learned from other fields. I learned among other things the Administration had deliberately starved the NSF Education directorate of research funding and shifted it to the Department of Education. The presumed motive for this was lack of enforcement of ordinary scientific study standards in the D of Ed. allowing them to push research that was opinion-driven. This pronouncement was made by a panel that included the Baltimore superintendent and a Nobel laureate, people not given to rash statements.
During the intervening years, I have seen ample evidence of the truth of the allegations. On the good side, the research industry has been able to scrape the fallout together and go through the evidence in order to shine light into the dark corners of education reform.
JoeN：Are you saying you are examining research that uses randomized trials and control subjects? I suggested it to my my wife when she was doing graduate education research and her professor informed her that “she had not reached the correct conclusion.” Goldacre is merely saying it is not common in the least. Much of education research is entertaining as it involves irreproducible results.
I’ve been thinking that some of this disconnect between actual classroom practice and experiential knowledge, and education research occurs because teachers don’t tend to typically be involved in academic research. If teachers could more effectively combine their classroom findings and have this action research considered viable, this disconnect could be better addressed. So for example, I was considering how the open source model of software engineering could be applied to classroom research. Teachers could collaborate on research using the open source model to collate their findings into a useful, readily applicable product in other classrooms. In other words, turn active everyday teachers into researchers and utilize their data more effectively, rather than seeking to conduct studies under closed and rigid conditions that would never exist in a classroom environment.
The “action research” effort initially begun in UK and spread to U.S. academics and teachers has yet to reach a critical mass among teachers for many reasons–the most obvious is lack of time, given the daily crunch of activities and emotional energy expended. your suggestion of open source as another tool for teacher/academic collaboration makes sense. But the time factor remains a major barrier.
Yes, time is always an issue. I think the key to utilizing open source in the context of action research would be to build or utilize a free platform that enables teacher collaboration on action research projects to be rendered efficient and easy. Once that is established, however, it could enable teachers to share gathered data across a diversity of classroom environments, providing a greater wealth of data, though it would be messier data, of course. But if teachers begin to establish a consensus based on their own data, they’ll be more apt to take action based on that information. Not sure how this could happen, but the need is certainly there!
Is anyone else out there skeptical of the value of action research? I see two major problems: the teacher is too deeply involved in the project, and has too much stake in it, to assess the research with sufficient objectivity (a loaded term, “objectivity,” I know), and the research becomes an agenda of its own, which could interfere with the agenda of teaching. So both the research and the teaching can get shortchanged.
Here’s an example. A teacher has a hypothesis that if students write about their personal lives, they will do better on tests. She has them write about their personal lives, and then gives them a test. They do better, on the whole, than they did the last time. How is the teacher to know that this was because of the writing assignment? She could try to have a control group, but would that control group have a different kind of writing assignment? None at all? With such a small-scale study, it is very hard to draw conclusions.
Now, suppose the teacher wouldn’t otherwise have asked students to write about their personal lives. Suppose she had to complete an action research project, and this is what came to mind. The students have now been asked to write about themselves when they wouldn’t otherwise have been–not necessarily for the good of the instruction, but for the action research. For some, this may seem intrusive. They may not understand why they are being asked to do this. Is it fair to have students carry out assignments for a teacher’s experiment?
I have heard counterarguments to these concerns, but they haven’t been enough to convince me that action research is helpful and viable. I am much more in favor of alert, discerning teaching, which isn’t the same thing.
Diana, I think your points here go back to your original observation that before we can conduct research, we have to look at our values, which is an important point to make. If a teacher is conducting research simply for the sake of research (like for a grade in a class), then the research is more likely to be inauthentic. If, however, the teacher is genuinely exploring an avenue that will aid in reflective practice, and from which they would learn valuable new information, then the research could have a lot of potential. When I conducted some action research this past year, I started from a place that was authentic to who I am, what situation I was in, and to how I wanted to develop as a teacher. I learned a lot from the research, and it was due to the fact that I had great stake in it. For me, I don’t know that objectivity is as much the issue as authenticity.
Hi Manderson and Diana,
“Action research,” like “academic research,” covers so much territory that a strong streak of skepticism is necessary about both kinds of inquiry. Of Diana’s two problems, the first about objectivity applies to both teachers and higher ed researchers. I do know some teachers engaged in “action research” who are just as first-rate in asking tough questions and understanding threats to fair and balanced inquiry as the top academic researchers I know. Whether authenticity is more important than objectivity, I cannot say until I have a clear idea of what the former means. What distinguishes top “action researchers” from “academic” ones are the questions they ask and whether one of their priorities is to get published–not objectivity or authenticity.
The second problem that Diana mentions: public school teaching and research acts conflict because of competing agendas– masks some important differences between “action research’ and “academic research.” Those public school teachers engaged in classroom and school research lack the in-school time and paid time to make sense of what are overlapping, not conflicting agendas (depending, of course, on the questions both teachers and researchers ask). That scarcity of precious time to do what many sharp university researchers do (who have to teach 4 course each school year–or four hours a week each semester) which is have the research enrich their teaching. So the available time, the questions asked, and seeking or avoiding publication, in my opinion, distinguish the very best of the “action” and “academic” researchers.
Larry, thanks for clarifying some of those points out of the gobbledygook of what I was trying to get at. “Authenticity” wasn’t a very clear way to put it. What I meant was essentially what you said about what researchers who don’t have a lot of time do–which is to “have the research enrich their teaching.” If the teacher doing the research isn’t doing that, isn’t centering their research around something that is deeply of value to them within their immediate context, it’s not authentic nor integral to their professional development.
One question about something you had mentioned as fundamental to the best research: is the desire to be published an obstacle to good (action or academic) research or it is beneficial? I wasn’t clear on that point. Thanks for your time!
Manderson,
Most top academic journals have anonymous reviewers familiar with the topic evaluate submissions for publication. Peer review is considered the gold standard among academics in publications across the social and natural sciences, medical sciences, the law, and education. Action researchers as well as academics submitting manuscripts to journals often find out whether peers find their work suitable for publication. In my experience, such reviews have helped me revise and strengthen what I have written. I have found the review process often frustrating but ultimately beneficial in having better articles and books. Other writers, teachers and academics, may have had different experiences. Writing in the blogosphere, of course, means no prior review by peers (except from comments after you have clicked the “publish” button).
Pingback: Remainders: Bloomberg: 15,000 teacher layoffs possible | GothamSchools
Dr. Cuban,
As one of the few education writers who helped me to define and shape my dissertation research on dilemmas in teaching, I applaud the sentiments expressed here and offer my own personal compromise to this conundrum. In addition to relying on experience and rigorous research, I look for great thinkers, who write persuasively about the big questions in education and freely admit to their own uncertainty, who don’t offer “best practice” or “what works” solutions, respecting the idea that context and culture matter. I also continue to remind myself that all research based on test scores is as limited as the tests, which still tell us remarkably little about what students know, think, and can do.
I cringe when I see “what works”. Going back to the analogy to medicine, it is equivalent to anecdotal evidence.
Research can be useful if read carefully–for its caveats and hidden observations as well as its conclusions. When education research gets summarized, in white papers and so forth, an awful lot gets left out. Time after time I have looked into “research has shown” statements to find that the research in question doesn’t show quite what people claim it shows.
Now there is great pressure on teachers to cite research in support of instructional decisions they make (the teaching standards for New York State include this expectation). This, to me, simply encourages “research has shown” statements.
Another problem with research in education is that much of policy inevitably rests on values. You can’t prove or disprove values. You can’t prove the worth of a classical education, for instance, though you may have good grounds for believing in its importance. The same is true to some degree in medicine–how you wish to approach your health will influence which doctors and methods you seek out.
So, before even looking at research, one must first ask: what kind of education are we hoping for in the first place? That will help determine what sort of research is applicable. And then, when we do look into research, we must take care to distinguish between what it actually says and what people say it says.
Thanks, Diana, for your comments. I wish I had included the points you made about values–“you cannot prove or disprove values”–and inspecting carefully the common refrain from policymakers, administrators, and practitioners that “research shows” which it seldom does.
Pingback: The NYS Teaching Standards: Too Many, Too Broad? | GothamSchools
Pingback: Quick Hits (1.28.2011)