Golf and Measuring School Reform

LB+-+Mayer+with+Thalberg+playing+golfLouis B. Mayer, a Russian immigrant who went into the film business, founded Metro-Goldwyn-Mayer (MGM) in 1924. He produced the blockbuster film of its day, Ben Hur, and wanted to join high-class society. Friends told him that upper-class men played golf. In his gruff way, he began playing golf but he never got the hang of the sport, failing to understand that you scored the game in strokes. He thought golf was a race and that finishing in a shorter time meant you were getting better. He hired two caddies with the first one posted down the fairway to find the ball after he hit it. Then, the second caddy would run ahead and put himself in position for Mayer’s second shot. This continued for eighteen holes. After the game was over, Mayer would look at his watch: “We made it in an hour and seven minutes! Three minutes better than yesterday!”

Judging success often depends upon whether those playing the game understand the basic logic of the game and whether the metric used to determine success fits that logic.

Mayer’s beliefs about golf and the metric he used were mistaken. It is equally tempting to use wrong measures to assess a school reform. Metrics that have little to do conceptually with the “game” of education, especially when complicated reforms such as mayoral control, charter schools, and small high schools are put into practice–may appeal to policy makers and even satisfy parents and taxpayers but are unconnected to the reform policy’s assumptions. Judging the record of such initiatives requires getting the basic idea of the reform right, that is, realizing that it’s golf strokes that matter, not the speed.

Consider small high schools. The basic policy logic of the reform or theory of action—is to take large comprehensive high schools and create new structures of small learning communities and advisories that would personalize schooling for students sufficiently to transform teaching and engage students sufficiently for them to perform well on tests and graduate ready for college or enter jobs in a rapidly developing information-driven economy. Converting big schools into small learning communities, then, will transform teaching and learning thereby producing desired policy outcomes.

The metrics most often used to determine the success of small high schools in big-city districts are largely test scores, achievement gaps between minorities and whites, dropouts, high school graduation rates, and college admissions. Research studies of small high schools, however, are, at best, unclear as to whether these policies produce the desired outcomes across the different measures.

Fitting the logic of the policy logic to outcomes also depends on when the assessment occurs. Thus, judging a reform too early in its unfolding often yields inconclusive data—showing who’s ahead at the end of nine holes of golf rather than after eighteen.

Determining policy success, then, depends on timing but of equal importance, whether the assumptions contained in the reform’s logic have indeed been put into practice, and the fit between the metrics used and the expected outcomes.

Few big cities judge their reforms in this manner. Yet when test scores go up, mayors, school boards and superintendents are quick to take credit for the gains, even though it is unclear which of the implemented reforms caused the gains. When test scores dip, reasons spill forth people for the disappointing results and blame gets dispersed. Few, however, can go beyond guesses as to why scores declined.

So to avert both premature and mistaken judgments, I propose four questions to judge whether the logic of the policy (the game of golf) and metrics (strokes) are in sync.

1. Did reform structures and programs aimed at improving student achievement get implemented?
Without complete implementation, no determination of policy success or its validity as a theory of change can be made.

2. Have teaching practices changed in the intended direction? Without altering typical classroom routines consistent with policy assumptions and strategies, chances of students learning more, faster, and better are slim.

3. Did changed classroom practices account for what students learned? Without systematic inquiry into which classroom practices have changed, it is impossible to determine whether student learning, even if only measured by test scores, is due to a close match between what teachers did daily (e.g., deeper probing into subject-matter content, focusing on test preparation) or to some other factors.

4. Did what students learn achieve policy makers’ desired outcomes? Do higher test scores for students translate into a reduced achievement gap between minorities and whites, fewer students dropping out of high schools, and more graduating seniors going on to college?

Few policymakers ask these questions and then seek answers to them. As a result, they end up like Louis Mayer: playing golf by the wrong metrics.

2 Comments

Filed under school reform policies

2 responses to “Golf and Measuring School Reform

  1. The Washington Post editorial staff should read this posting. Last week they cited DC’s rising NAEP scores (good news, to be sure) as evidence that Chancellor Rhee’s proposed reforms are the right path for the district: “Those seeking to block the changes being pursued by Ms. Rhee…might want to think again.”

    That statement irked me for several reasons. First, the upward trend on NAEP began before Rhee’s tenure, which suggests that her predecessor’s work to strengthen standards and curriculum may have changed instructional habits and improved instruction. Rhee’s subsequent reforms to personnel MAY have sustained the trend (though it’s certainly hard to tell on both counts.) But Rhee’s signature reform–teacher merit pay–hasn’t been implemented yet, so it seems premature to use NAEP gains as a justification to go full steam ahead on that one.

    Pay for performance is a worthy idea, but teachers are right to be concerned about the many, many technical difficulties it still presents. And before people crow about gains–or even anticipated gains (!)–they should pose your four questions.

    In doing so, they would commit themselves to a robust research agenda.

    • larrycuban

      Claus,
      Taking credit for gains and distributing blame for losses is common among CEOs, U.S. Presidents, and district superintendents. Such behavior has occurred often. Herbert Hoover blamed for the Depression when earlier Presidents and Congresses made fiscal and monetary policies that produced the crash in 1929 and subsequent bank failures. More recently, Bill Clinton took credit for the economic boom of the 1990s when economic growth of those years came from earlier and multiple influences that economists still try to figure out. So why shouldn’t Michelle Rhee take credit?

      The reason officials take credit for gains is that few policymakers and researchers (and especially media folks) know what causes scores to go up, plateau, and sink with any degree of confidence. This is another way of saying that few superintendent policies within a 2-3 years ever directly change teachers’ lessons. Therefore, student learning, even inadequately measured by state and national tests, cannot be attributed to any single superintendent’s actions within a few years. Now, were that superintendent to have built over a decade curricular and instructional structures for teachers to use in classrooms, help teachers stretch their knowledge and skill to use those structures, and create a district-wide climate of collaboration to increase student learning then maybe, just maybe, a superintendent could take credit. Not yet for D.C.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s