Beliefs, opinions, and politics matter more in making policy decisions than applying research findings to schools and classrooms. A recent Canadian study (1905-8719-1-PB ) confirmed what amounts to a fact in U.S. policymaking. Candadian researchers looked at provincial policy elites (in U.S., they would be state-level decision-makers) and district officials (school board members and superintendents) and found across Canada what is very evident in U.S. districts as well: politics and beliefs trump use of research in adopting policies aimed at improving practice (see here and here).
As one would expect in academic circles, the language of applying research findings to educational policies has expanded. Canadian researchers Gerald Galway and Bruce Sheppard note that new phrases have entered the vocabulary: “knowledge transfer, evidence-informed policy, data-driven decision-making and knowledge brokering, to name a few. Knowledge mobilization (KM) has been touted as a useful all-encompassing term because it conveys the notion of direction instead of random interaction and it ’embodies the idea that the use of knowledge is a social process, not just an intellectual task’ ” (p. 9). Whatever phrase is used in Canada or the U.S., the pattern of applying research findings to forming, adopting, and implementing policies remains similar on both sides of the border.
Galway and Sheppard studied “senior bureaucrats” in provincial ministries and local board trustees and administrators at two points in time, 2006 and 2012. They used surveys, interviews,and focus groups to generate the data they analyzed.
A typical response from “senior bureaucrats” in provincial ministries about research and policymaking was (M refers to “ministry” official):
M1: I guess that what I find with university research is that those who are doing it
appear to be somewhat removed from the immediacy of it all; it’s not engaged, as it
were, and [university researchers are] somewhat reluctant to engage those who are
involved in a particular issue on a daily basis. That’s being pretty general; that’s not
the case with all university research.
M7: Well, I haven’t had a chance to look at anything [university researchers] are
doing so I really don’t know, but I do find that sometimes, just from past
experiences, that the research coming out of (names university) is not always; well
not the research, but some of the conclusions they draw from their research is not
always applicable….
M2: Well, I don’t really know about it, to be truthful. As I was saying earlier that’s
one of the things that is most surprising to me; that with the amount of research that
is done at the university, how little discussion or impact it’s had on my work as the Minister in this portfolio.
When the researchers turned to trustees–district school board members–survey results and focus group interviews in the 2012 study produced a list of factors that influenced their making of policy.
[W]e presented trustees with 20 potential factors/influences and asked them to
indicate on a seven- point scale, the extent to which each has had an effect on specific decisions or
recommendations that they have made as trustees…. The principal six factors are as follows:
(1) personal or professional beliefs and values,
(2) potential to directly influence student
outcomes/student learning,
(3) advice of district staff and/or colleagues,
(4) the school board’s strategic plan
(5) past experience.
Trustees and superintendents rated the following six factors as having the least influence on decision-making (my emphasis):
(1) representations of business/private sector,
(2) pressure from special interest or lobby groups,
(3) a situation or event someone told you about,
(4) pressure from government (ministry/department of education),
(5) public opinion/avoidance of negative media attention
(6) university-based research.
Analysis of the data by province showed very little variance in the factors and evidence that influence school board decision-making across jurisdictions.
The Canadian researchers conclude:
We conceptualize use of low research impact partly in terms of trust vs. risk avoidance. In
both policy contexts – provincial departments of education and school boards – we conclude that
decision-making is ambiguous and risky…. [I]n uncertain times, decision-makers value knowledge that is familiar and emerges from their
own community…. This information – now repackaged in the form of staff advice – becomes privileged
and trusted by ministers and senior bureaucrats as authentic knowledge….
readjustment.
Very little found north of the border, then, differs greatly from what U.S. researchers have found in the 50 states (see here, here, and here).
So what?
Keep in mind that none of the above critiques of limited influence of research on policy is restricted to public schooling. Making policy in systems of criminal justice, environmental improvement–think climate change–and health improvement and clinical medicine–think of TV ads for drugs–are subject to similar political factors and personal beliefs rather than what research has found. Calls for more collaboration between university researchers and policymakers have also been heard and ignored for decades. Critics have pointed out many times that the academic culture and its rewards overlap little with the world that decision-makers face every week.
I wish I had a neat answer to the “so what” question and offer a package of do’s and don’ts for those who want evidence-based policymaking. I am on the lookout for such advice but thus far I have come up empty. The fact is that beliefs, opinions, and politics matter more in making policy decisions than applying research findings to schools and classrooms.
Larry Cuban on School Reform and Classroom Practice 
Thanks Larry for this. It is spot on.
“offer a package of do’s and don’ts for those whose who want evidence-based policymaking. I am on the lookout for such advice but thus far I have come up empty”.
I’ve spent over a decade developing a website (1,000 plus best practices, research validated deep learning posts, 1,000’s more links) that validates evidence based policy making. I’ve promoted it heavily within education, but rarely have I heard wow, this is just what I was looking for:
http://mcdonaldsalesandmarketing.biz/
The available evidence and research is overwhelming, but individuals choose to say it doesn’t exist, due to institution/individual paradigm paralysis:
http://mcdonaldsalesandmarketing.biz/36704/paradigm-paralysis/
The sad reality is that traditional educators are at most subject matter experts. They rarely are pedagogy experts (21st century learning), nor are they education reform experts. BUT, we listen to them, assuming they are experts beyond their subject matter expertise, thinking that their desire to preserve the ineffective and inefficient one size fits all paradigm, is in the students best interest. When in reality, the current outdated one size fits all methodology, best serves the institution/Staff/faculty, at the direct expense of advancing individual students success outcomes…go figure
The other reality, that is all present, is educators broadcasting the proper approach, but not implementing it: ( “I’m in it for the students”, “the ideal is one teacher per student”, “yes differentiated learning does make a difference in advancing individual student adaptive learning skills” …BUT when you look at how they teach its one size fits all
These are ALL reasons, among many, why education/educators is/are failing our students in droves and why the value of an education is being questioned.
Who really cares about research and education reform in an education environment? The provost, its their job. Can the provost impact the whole organization positively if the leadership has paradigm paralysis? Of course not. So they promote ineffective and inefficient learning under the guise of research validated, education reform. Leaving everyone thinking that they are the experts, have adequately researched what works and are implementing that with the goal of advanced student success outcomes, when in reality little changes.
Then you have outright deceit and deception, from those that know better.
Stanford, Harvard, MIT all knew that the one size fits all MOOC would at best empower some participants with initial understanding, but would never empower participants with adaptive learning skills that would advance individual performance improvement. They rolled MOOCs out as innovative, we drank the cool aid and history has shown that MOOCs are probably less effective than big classroom lecture, which IS NOT effective for individual learning. Oops…I guess Stanford, Harvard, MIT’s credibility in learning methodology has now been eroded.
Especially in publicly funded organizations we should be outraged at this, but we seem to let it slide.
We keep thinking that educators are professionals like mechanical, etc. engineers are professionals and will embrace positive change as part of their profession. The reality is they do not (look at paradigm paralysis) embrace positive change to advance individual student success outcomes…the ONLY reason education exists.
Long story short students = revenue. Take away ALL the students and see what everyones reality is that remains
You want to know, in one place what the research says? Look at these links:
http://mcdonaldsalesandmarketing.biz/category/learning-strategies-in-2012/
http://mcdonaldsalesandmarketing.biz/student-success-1-of-5a-2/
http://mcdonaldsalesandmarketing.biz/education-reform2of2/
http://mcdonaldsalesandmarketing.biz/education-reform-part-30f3/
http://mcdonaldsalesandmarketing.biz/education-reform-part-4-of-5/
http://mcdonaldsalesandmarketing.biz/education-reform-part-5of-5/
You want to know what the research says we should be doing? Look here:
http://mcdonaldsalesandmarketing.biz/21658/sales-performance-2/
You can no longer say you don’t have access to the information. BUT you can still nay say it, criticize the messenger and pretend you are the expert to maintain a failed incumbent methodology
Thanks for your comment, Tom.
I am not sure what it’s like in other public service arenas, but in education there is the bias that the research is not very rigorous and has nothing to offer. I had a similar bias before I entered a education doctoral program. Coming from an engineering background, education research is very very different. A lot of the conclusions are much more tentative as they need to be given the contextual nature of education. This can be interpreted as a sign of weakness instead of a sign of strength and acknowledging the limitations of the study.
The research is also not very accessible to practitioners and district level decision-makers. They do not always have the time or the resources to study with the full depth of the research in a certain topic, certainly when most of the research is behind a pay wall for the journals.
So we need a little bit better communication on both sides of the research issue.
But I’m not sure that will really make a difference because as this paper and other’s have found, beliefs triumph all.
Thanks, Alice, for your comments on educational research since you do come from a different background.
“The fact is that beliefs, opinions, and politics matter more in making policy decisions than applying research findings to schools and classrooms.”
This statement certainly rings true in the US, especially for the last several decades. USDE policies that purport to be evidence-based are proving to be large-scale experiments with little or no evidence in support of them.
One of my favorite examples is this: In a belated recognition that something may be amiss in federal policy bearing on teacher evaluation, USDE decided to commission a study for the purpose of getting “rigorous” evidence on whether the evaluation systems called for in federal policy have their intended effects on teacher and leader performance and student achievement (American Institutes for Research, 2012, February). This five-year, $16 million study of Teacher and Leader Evaluation Systems will be completed in 2017, long after teachers and principals in almost every state have endured the requirements of evaluation systems known to be unreliable and ineffective as means to improve educational outcomes.
Or consider the influence of the American Legislative Exchange Council on state policies. ALEC’s A+ Literacy Act, model legislation called for a basketful of evaluation measures calculated to increase school and district failures while expanding the market for charter operators and creating incentives for those operators. Ohio legislators created a variant of this rating scheme and now almost every large metro district in Ohio is at risk of being “100% charterized,” just like Youngstown.
References: American Institutes of Research. (2012, February 23). AIR selected to conduct study measuring the impact of teacher and leader evaluation systems on student learning and performance. Press Release. Retrieved from http://www.air.org/news/index.cfm?fa=viewContent&content_id=1755
American Legislative Exchange Council. (2011, January). A-Plus literacy act, Model legislation: Chapter 1. School and district report cards and grades. http://www.alec.org/model-legislation/the-a-plus-literacy-act/
Thanks for the examples, Laura. They do reinforce the points that Canadian and U.S. policy researchers have made.
Three experience-based suggestions: (1) sandwich research between stories and anecdotes; (2) provide at least two different perspectives on issues, both relying on applied research, then invite the reader to choose either or both, and to ask themselves why; (3) avoid using the term research as it scares or intimidates or simply bores too many people.
Yes you are spot on Larry! Yes it does come absolutely down to beliefs, opinions, and politics, and politics especially. And I would like to add, that because of the peculiar nature of politics it is for this simple reason that policy making continues to widen the achievement gap, as very little decisions are based on educational research. School Districts are now the largest employers in many urban centers/cites in the United States and rampant abuses of power can be found when it comes to school policy regarding education policy. Therefore schools continue to reproduce and replicate the existing inequality in American society divided along lines of social class and race. And it makes no difference who controls the levers of power whether African American, Caucasian, or Latino. It still holds true that there is no regard for research concerning implementation of educational policy.
Thanks, Wilson, for the comment.
In 2011 the UK government did try to respond to this problem by creating the Education Endowment Foundation. One of their key actions has been to create a Teaching and Learning Toolkit which lets teachers and others see, quickly and easily, what the educational impact and costs are of a whole raft of choices and actions, based on what credible research says. The links below are to the toolkit itself, and for anyone interested, an account of its methodology.
https://educationendowmentfoundation.org.uk/toolkit/toolkit-a-z/
https://educationendowmentfoundation.org.uk/toolkit/toolkit-a-z/about-the-toolkit/
In my work I do see it being increasingly referenced, although only ever as shorthand. High quality educational research is after all, like any other kind of “difficult” text… not something one can “browse.”
Didn’t know this. Thanks, Joe.
I was always wondering who is supposed to find and read this research? Most school board members (the policy creators) have neither the time to find or the background to understand the research. The teachers (policy implementors) simply do not have the time. It is not just the reading of the research article, but the finding of relevant research that that takes so much time. The only time I have seriously looked at Ed research was when I was working on a EdD. The average classroom teacher has neither the time nor inclination to read the stuff. Pre-service teachers will see a little of it if their instructors have them dig but even then it will be minimal.
Again, Garth, thanks for taking the time to comment.
Reblogged this on David R. Taylor-Thoughts on Education.
Pingback: Research and practice again | bloghaunter
Pingback: How Small a Part Research Plays in Making and Implementing Educational Policy | Computing Education Blog