After Atlanta (GA) school administrators and teachers went to trial andwere convicted and sentenced to jail for cheating and before that the El Paso (TX) superintendent convicted of the same charge and in prison, the generally accepted idea that district superintendents can pump up student achievement has taken a serious hit. Cheating scandals across the country have turned the belief in superintendents raising test scores into something tawdry.
For decades, many superintendents have been touted as earnest instructional leaders, expert managers, and superb politicians who can mobilize communities and teacher corps to improve schools and show gains in students’ test scores. From Arlene Ackerman in Philadelphia to Joel Klein in New York City to Kaya Henderson in Washington, D.C., big city superintendents are at the top rung of those who can turn around failing districts.
Surely the Atlanta cheating scandal and others around the country have tarnished the image of dynamic superintendents taking urban schools from being in dumpsters to $1 million Broad Prize winners. A tainted image, however, will not weaken the Velcro belief that smart district superintendents will lead districts to higher student achievement. Just look at contracts that school boards and mayors sign with new superintendents. Contract clauses call for student test scores, graduation rates, and other academic measures to increase during the school chief’s tenure (see here and here).
Then along comes a study that asks whether superintendents are “vital or irrelevant.” Drawing on state student achievement data from North Carolina and Florida for the years 1998-2009, researchers sought to find out how much of a relationship existed between the arrival of new superintendents, how long they served, and student achievement in districts (see PDF SuperintendentsBrown Center9314 ).
Here is what the researchers found:
- School district superintendent is largely a short-term job. The typical superintendent has been in the job for three to four years.
- Student achievement does not improve with longevity of superintendent service within their districts.
- Hiring a new superintendent is not associated with higher student achievement.
- Superintendents account for a small fraction of a percent (0.3 percent) of student differences in achievement. This effect, while statistically significant, is orders of magnitude smaller than that associated with any other major component of the education system, including: measured and unmeasured student characteristics; teachers; schools; and districts.
- Individual superintendents who have an exceptional impact on student achievement cannot be reliably identified.
Results, of course, are from only one study and must be handled with care. The familiar cautions about the limits of the data and methodology are there. What is remarkable, however, is that the iron-clad belief that superintendents make a difference in student outcomes held by the American Association of School Administrators, school boards, and superintendents themselves has seldom undergone careful scrutiny. Yes, the above study is correlational. It does not get into the black box of exactly how and what superintendents do improves student achievement.
Ask superintendents how they get scores or graduation rates to go up. The question is often answered with a wink or a shrug of the shoulders. Among most researchers and administrators who write and grapple with this question of whether superintendents can improve test scores, there is no explicit model of effectiveness. That is correct, there is no theory of change, no theory of action.
How exactly does a school chief who is completely dependent on an elected school board, district office staff, a cadre of principals whom he or she may see monthly, and teachers who shut their doors once class begins–raise test scores, decrease dropouts, and increase college attendance? Without some theory by which a superintendent can be shown to have causal effects, test scores going up or down remain a mystery or a matter of luck that the results occurred during that school chief’s tenure (I exclude cheating episodes where superintendents have been directly involved because they have been rare).
Many school chiefs, of course, believe–a belief is a covert theory–that they can improve student achievement. They hold dear the Rambo model of superintending. Strong leader + clear reform plan + swift reorganization + urgent mandates + crisp incentives and penalties = desired student outcomes. Think former New York City Chancellor Joel Klein, ex-Miami-Dade Superintendent Rudy Crew, ex-Chancellor of Washington D.C.and ex-school chief Alan Bersin in San Diego. Don’t forget John Deasy in Los Angeles Unified School District. And now, Pedro Martinez in San Antonio Independent School District
There are, of course, other less heroic models or theories of action that mirror more accurately the complex, entangled world of moving school board policy to classroom practice. One model, for example, depicts stable, ongoing, indirect influence where superintendents slowly shape a district culture of improvement, work on curriculum and instruction, insure that principals run schools consistent with district goals, support and prod teachers to take on new classroom challenges, and communicate often with parents about what’s happening. Think ex-superintendents Carl Cohn in Long Beach (CA), Tom Payzant in Boston (MA) and Laura Schwalm in Garden Grove (CA). Such an indirect approach is less heroic, takes a decade or more, and ratchets down the expectation that superintendents be Supermen or Wonder Women.
Whether school chiefs or their boards have a Rambo model, one of indirect influences, or other models, some theory exists to explain how they go about improving student performance. Without some compelling explanation for how they influence district office administrators, principals, teachers, and students to perform better than they have, most school chiefs have to figure out their own personal cause-effect model, rely upon chance, or even in those rare occasions, cheat.
What is needed are GPS navigation systems imprinted in school board members’ and superintendents’ heads that contain the following:
*A map of the political, managerial, and instructional roles superintendents perform, public schools’ competing purposes, and the constant political responsiveness of school boards to constituencies that inevitably create persistent conflicts.
*a clear cause-effect model of how superintendents directly influence principals and teachers to do better as in creating incentives and sanctions, a culture of trust that encourages both risk-taking and willingness to learn.
*a practical and public definition of what constitutes success for school boards, superintendents, principals,teachers, and students.
Such a navigation system and map are steps in the right direction of answering the question of whether superintendents can raise test scores.
Larry Cuban on School Reform and Classroom Practice 
Reblogged this on David R. Taylor-Thoughts on Texas Education.
thanks again, David, for re-blogging post on superintendents and test scores.
While I resist defining student achievement on the basis of standardized test score alone, I do believe that rigorously controlled meta analyses that rely on thoughtful, research guided analytical frameworks can generate a more insightful set of conclusions than your combination of reliance on results from North Carolina and Florida and anecdotal observations about urban superintendents. No question that factors other than superintendents bear more directly on student performance, but the indirect links are surprising and potent. For a comprehensive theoretical model and solid meta analytical yield, See the work of Bob Marzano and colleagues. Your comments seem oblivious to Marzano’s superb body of theory and research.
Thank you, Jeff, for your comment. Yes, I am familiar with the Waters and Marzano meta-analysis of studies on superintendent influence on student test scores. Like the one study I profile in this post, Waters and Marzano also did a correlational study of superintendent effects which show the associations between superintendent actions and student outcomes. The listing of what these actions are, like so many previous studies of effective schools, detail features correlated with outcomes, not a causal model of what superintendents do to achieve those effects. An eager superintendent could do each and every one of the behaviors Waters and Marzano list and still have little effect on achievement, however measured.
Larry, thank you for taking on the very complex topic of school superintendents’ impact on student achievement. Any one of the points you bring up, or which were addressed by the Brown report you provide a link to, could require pages of type or long conversations to adequately address. I don’t agree. So, if this is open for discussion:
1. I suspect that if the Brown researchers applied similar criteria to NFL coaches, U.S. navy officers, or corporate CEOs, their leading questions would be compelled to draw the same inferences: that leadership doesn’t matter, or isn’t measurable, or is almost inevitably overridden by systemic factors. I reject that conclusion out of hand. A conclusion that leadership doesn’t matter ought to prompt them to dig deeper.
2. The Brown study represents a very “American” slant toward educational research. This is germane, if not seminal to the subject the researchers are tackling. Whether we’re talking about Japan, Finland or a well-run private or public school, entities that are getting it right in education aren’t publishing reports implying that their leadership doesn’t matter. The point here is that Americans don’t Value their educational leaders. This makes the job of superintendent unattractive. In America, the way we Show value is to pay for it. And we don’t pay superintendents very much. They top out at around $500,000 in the largest, most challenging districts. More commonly, superintendents clear somewhere around $150,000 – for a described skill set and day-to-day responsibilities that typically earn seven figures or more in the private sector.
3. The valuation and economic factors described above leave us with a very (very) limited hiring pool from which to recruit. The single largest pool of potential superintendents comes from failed classroom teachers: those teachers who got into the education profession and a) flat out disliked teaching, b) were bad at teaching, and c) were wed to teaching because that’s where they’d invested their college education and had neither the skills to find other work nor the capital to go back to college and pursue a different career path.
Our educational system Encourages these failures to “move up the chain.” Current administrators typically provide little to no accountability for bad teachers, so even the worst teachers can count on “acceptable” evaluations. And that’s all they need in order to get into the weak administrative programs colleges offer where these failed teachers take simplistic classes. Tuition is collected, everyone gets an ‘A’ or a ‘B’, and lo and behold, a new administration is born.
Attempts to recruit and hire superintendents from non-education backgrounds have mostly failed for one simple reason: Anyone with the skill set to run a school district is already making many times over the money in the private sector.
4. The Brown study itself is so flawed it’s hard to know where to begin.
a. The researchers talk about tracking longevity of service of superintendents and correlating that with… With what? In most districts, superintendents hang around because an indifferent school board and a cowed group of parents allows them to. End of story. Most of what these superintendents do is focused on their retirement goals. So, no, these superintendents won’t have much (or any) impact on student achievement.
b. So the board decides to replace the superintendent. To what end, and with whom? On the one hand, most school boards are comprised of individuals who got elected on a one-issue, axe-to-grind campaign and have no idea what they’re doing. On the other hand, a poorly run school district is pretty much guaranteed to be limited to a hiring pool of incompetent superintendents who were run out of other districts. No gain, no loss, just as the Brown study shows. But the underpinnings of this dynamic do not justify the conclusions the Ivy league researchers put into print.
c. At the other end of town, a school board is faced with the task of replacing a competent superintendent who has served long and well. This superintendent came into a well-run district, kept the ball rolling, and left. No gain, no loss. The school board will have a pool of the very best superintendents available to choose from. Most likely, the ball will continue rolling. No gain, no loss, just as the Brown study shows. But to conclude from this that that leadership doesn’t matter… is foolishness.
Over a span of four decades, the Pittsburgh Steelers won six Super Bowls under three different coaches.
To win a Super Bowl it takes great athletes. (Analogous to teachers.)
It takes a great fan base. (Analogous to parents, and like the fan base, shown an effectively run organization, they can be brought into the fold.)
It takes a great front office. (School board).
It takes great assistant coaches. (Analogous to building principals and other administrative leaders.)
And it takes a great head coach. (Analogous to a superintendent.)
Americans recognize the vital role leadership plays in every institution they value, from football teams to publicly traded corporations to police departments to military units. When it comes to education, they turn logic on its end and produce “studies” to show that leadership doesn’t matter. Why? You can be sure Finland isn’t producing such studies.
I am very interested in what you think about all of this, Larry.
Thanks for the extended comment on leadership. The Brown study, flawed as it may be, does not say that leadership is irrelevant or does not matter. It focuses on the superintendent and its data set and methodology comes up empty on the impact of superintendent leadership on test scores. Other studies do show influence of the superintendent. Moreover, research on principals’ influence on student outcomes does show indirect and significant impact on students’ test scores. So the issue is not whether leadership matters but how does it matter–what do superintendents do that can result directly or indirectly on improving students achievement. While the research thus far, flawed as it may be, showed, at best, mixed results–the Brookings study being one example–argues for looking at the superintendency as a managerial and political post rather than one that provides instructional leadership to principals and teachers.
The Brown study minimizes the role of superintendent leadership, and uses bad research to do so. So, there’s an agenda at play here… Hardly the first time we’ve gotten flawed interpretations of and prescriptions for public education form the Browns, Harvards and Stanfords of our world – under the moniker of “research” of a kind no scientist would vett. Rush Limbaugh or Bill O’Reilly blathering their opinions are no less and no more useful.
Your reference to a Brookings study intrigues me though.
Superintendent as a political post (and I would add financial overseer). Yeah…
No.
Maybe.
I’ve thought this model through end to end, Larry. At present, I’m teaching in a system where this works perfectly. Our director, analogous to a CEO or a superintendent, has virtually no input into how we teach or classes. Here’s the beauty of it.
He doesn’t have to.
There’s an overlying structure so soundly developed that principles and teachers can run the system on their own. There are parameters, but there’s also elbow room.
Meanwhile, in the background, there are significant issues which include the financing of the school, recruitment of students, big picture end results of the education we provide (as in how many students get into good colleges) and expansion of school property and facilities.
It’s an interesting and effective system. One based on competence. A competent superintendent allows competent administrators to allow competent teachers to do their jobs.
Larry, I trust that you understand that in most schools in American, this chain of command is broken. “Competence” at various levels is the operative word.
In most school districts in America, we Need the superintendent to be an instructional leader. Because the school board doesn’t provide that vision, and principals are lazy. Yes, Lazy. And no group of teachers I’ve ever been associated with will long rise above the level of leadership expectations.
What I’m presenting is the day-to-day reality of most schools in most districts in America… As a country, we’re frustrated with our education system. You can allow Brown, Harvard and Stanford to pass the pipe to you, or you can consider an alternative view.
Sincerely,
Jack Donachy
To this — “The familiar cautions about the limits of the data and methodology are there” — I think a caution about the limits of test scores as a measure of learning and school quality should be added. As Board Member in a district with a Carl Cohn protege as a Superintendent, I also think that the distinctions between the two models are not as clear in practice as they are from a distance.
A fair point. Thanks for commenting, Thomas.
The Brookings report fails to model the longitudinal effects of schools and districts on students. It assumes that a superintendent effect is independent year to year, which is obviously not the case, and exactly what the contracts linked to are calling for. A very nice critique of exactly these methods can be found in Coelli & Green (2012) in which they show at the school level, if you allow a principal’s effect to be dynamic and grow over time (as it does in the real world) principals can account for a very large percent of the variance in student achievement, perhaps even the majority of the variance ( http://www.sciencedirect.com/science/article/pii/S0272775711001488 ).
Also, Brookings does not engage in peer review (which would give the authors a chance to catch the issue above), so caution should always be taken in hand as it’s basically their opinion and has not been vetted by the field (understanding that peer review is like the old saying about representative democracy, not a great system, but the best we have).
Thanks for commenting. You are correct that the Brookings study is not longitudinal. Their data were lacking on that point. I did not know that Brookings papers were not peer reviewed.And you are correct that correlational studies on the principalship do show linkages between principal leadership and student outcomes. There are, however, a few causal models that a some researchers have developed over time. Phil Hallinger and Ron Heck have shown how principals influence student academic achievement indirectly in a series of studies.
I find this wildly interesting in light of my current studies on teacher pay-for-performance and Value-Added Model (VAM) related educator assessment. I will be following your blog closely, Mr. Cuban. I was fascinated by your thoughts shared in the film we watched for our Intro to Education class at St. Petersburg College. After viewing the film, I decided to look you up to see if I could find more information on your writings. I’m delighted to have come across this blog. Thank you for all you do for the cause of education in our nation.
Thank you for commenting, Lisa.
I wrote a dissertation on the topic – specifically as it relates to the qualities that school boards look for. It used the 5 leadership qualities identified in Marzanno’s book, District Leadership that Works and asked if school boards seek these traits at all.
You can find the abstract at: http://gradworks.umi.com/35/11/3511344.html
I could certainly share more if you are interested about what was found.
Thanks for the post, and more generally for the blog, Larry. The results you summarize here also align with the work of Kenneth Meier (Texas A&M) and Laurence O’Toole (Univ. of Georgia) who have had a long-running research program on public management and performance, with Texas school districts as their key venue for analysis. The work has consistently found that within school districts, higher level actors trying to assert control have much less power over results than those closer to practice (e.g., teachers). See https://jhupbooks.press.jhu.edu/content/bureaucracy-democratic-state and http://teep.tamu.edu/.
Thanks, Paul, for your comment and citation.
Hello Folks
We have for more than a decade used CST test scores in California to label school populations as failures mostly. Not true. Here is something that is true. If I were a high school student today, I would laugh in your face about trying to get me to waste my time taking a test that means nothing to me. I would not even show up to school on those wasted days. My behavior, and the behavior of the 60% of high school students who think this way, ruin any study about teacher, principal or superintendent effectiveness. It is widely ignored that the producers of the tests tell us that using them to label anything is inappropriate. The assessments were not designed to be used to label people. The true effectiveness of leadership at any level is not measurable. But the people exposed to it know it matters. They see the effect in real time, and then it is gone and we begin anew. Truth is, it doesn’t matter whether we think someone did a good job or not. They did the job they did while they had the job. Most of them did the best they knew how. Now the next one will do the job the way he or she does the job. Meet the new boss, same as the old boss. It’s a human endeavor and it will never be 100% at anything. Stop trying to measure it and try describing what you really see as you spend time there on the job with the boots on the ground. Believe me, if you spend time at a high school helping the principal, you will stop all of this nonsense. It can’t be measured or quantified. It is stories upon stories. Read and enjoy.
Thank you, Randy, for your comment.