After all those clinical trials in medicine to conquer depression, Alzheimers, and cancer, we are told now that Prozac and other anti-depressants work no better than placebos and doing crossword puzzles, exercising, and taking fish oil does not prevent Alzheimer’s disease. Moreover, studies have found that mammograms, colonoscopies, and PSA tests are far less useful to detect cancer than we had been told. The truth is that even with the toughest standards used by medical researchers to demonstrate one drug, one therapy, one test is better than other treatments it is very difficult to prove anything scientifically or even trust that those studies are accurate. Consider cancer.
President Richard Nixon declared war on cancer in 1971. Since then medical entrepreneurs have promised that surgery, radiation, and chemotherapy will cure the scourge. Molecular biologists and genome specialists have identified cell mechanisms and genes that trigger particular cancers. Yet after billions of dollars have been spent in cancer research, the disease with its remarkable heterogeneity remains at the top of the list of leading causes of death in the U.S. In short, politics, policy, and research have failed to cure cancer. But wait, there is even worse news.
One medical researcher and statistician, John Ioannidis, claims that 90 percent of published medical research that doctors rely on is flawed. Ioannidis and others raise serious issues about biases in research designs and the directions researchers pursue. The big unasked question remains: Can medical research studies be trusted?
Now, if doubts are raised about medical research where the highest statistical and ethical standards are used in studies on human subjects, what about educational research?
The sorry state of policy and practice research in education has been mentioned numerous times. Every important (and unimportant) issue in schooling children and youth has studies that say one thing and studies that say the opposite. Biases of researchers–as in medical investigations–often taint designs and methodologies. Efforts to make educational research more scientific and relevant to policy and practice have occupied scholars in both education and the social sciences for decades (see PDF EDUCATIONAL RESEARCHER-2002-Feuer-4-14).
To raise research standards, the federal government has launched a “What Works Clearinghouse” (WWC) to evaluate studies investigating educational policy and practice. Using tough standards of evidence, WWC determines the degree of effectiveness of curriculum, software, and intervention programs across 11 areas (e.g., Adolescent Literacy, Character Education, Elementary Math, English Language Learners). The stark results are eye-openers. Of nearly 75 Elementary Math programs, for example, six programs had mixed outcomes and only one had positive result for a single outcome (“Everyday Mathematics”). The other interventions, including software programs, had neither a report available nor had studies been done.
Stuck with results like this, educational policymakers, practitioners, and parents, ask: Can we trust research to point the right direction for practice?
As a high school teacher and district administrator for nearly 25 years and a researcher for another 25 years, part of me says yes and part of me says no.
In general, I value research. I have asked questions investigating the history of teaching, curriculum, and school reform (including technology). I have designed studies, and, using different methodologies, collected evidence and published my findings. I know that truth is elusive and that biases, including mine, can taint even the best designed study. Still, a careful, rigorous, and honest search for truth in different venues from preschool through the university is essential, I believe, for improved teaching and student learning.
Then the practitioner part of me kicks in and says that so much educational research fails to ask, much less answer, puzzling questions that teachers, principals, and superintendents face daily. Instead,to get answers to these questions, hardworking professionals have to rely on their experiences and the wisdom of peers, as I had done.
I have worked in both worlds and find it tempting to agree with those studies that support my biases while rejecting those that challenge those very same biases. And when research findings are mixed, I am tempted to ignore the findings. So I am torn by conflicting values. In truth, what I often end up doing–the compromise I have worked out–is to rely upon my experiences in classrooms and schools while keeping an eye peeled for rigorous, high-caliber studies.
In the face of growing disenchantment with medical research studies and the disarray of educational research, I have learned to trust my experiences. Though I am an avid consumer of educational research, I remain skeptical of much of what I read.
- You: Why Almost Everything You Hear About Medicine Is Wrong (newsweek.com)
- “Prominent scientist says almost all published medical research is flawed” and related posts (depthreporting.com)
- Human Farming & the Limits of Medical Research (thehealthcareblog.com)
- Lies, Damned Lies and Medical Science (kir.com)
- Right Diagnosis, Wrong Prescription: New Medical Research Center (innovationandgrowth.wordpress.com)