More Data Needed for Hard-To-Measure Student Learning and Teacher Quality (Guest blogger Stephen Lane)

Cover of "The Quants: How a New Breed of ...

Cover via Amazon

Stephen Lane is a high school teacher with 10 years’ experience at a suburban high school outside of Boston, MA. He teaches history and economics and also coaches cross-country and track & field.

We are living in an age of the quants. In the social sciences, sports, and of course education, the number-crunchers rule. Information that can’t be reduced to a numerical essence is suspect, whether the subject is basketball or economics. In education, quantifiable data can be useful, but the current infatuation with numbers betrays muddled thinking, misallocates teaching resources, and rewards behavior we probably don’t want to see rewarded in teaching.

Now that I’m done preaching to the choir, I’d like to think about how to reverse this trend. Railing against the misapplication of numbers (in particular, standardized test results, and misguided comparisons of American students to their foreign counterparts) is important, but we also ought marshal the tools of the quants themselves – data – to support a more nuanced look at student achievement and quality teaching.

My aunt, a former school committee chair (don’t hold it against her), and uncle, a retired principal, both avow they knew good teaching after 5 minutes of classroom observation. A former student of mine, whom we’ll call Hermione (the brightest witch of her age), is an ed consultant on value-added models (again, don’t judge). She feels confident their models can measure a teacher’s impact – given 10 years of data in relatively unchanging circumstances. Neither a 5-minute observation nor 10 years of data seem like reasonable metrics, nor do they address the conditions underlying the performance.

Quants assume that teachers work primarily in a vacuum – that a teacher’s success or failure is primarily the product of individual ability, effort, or lack thereof. My experience differs. My growth as a teacher is primarily due to the culture within my department, and the guidance and example of my colleagues. I am not unusual: Interviews conducted with active and retired teachers in our district with at least 20 years of experience overwhelmingly identified fellow teachers as the primary influence on teachers’ development and improvement.

In a paper on compensation methods in manufacturing, economists Susan Helper, Morris Kleiner, and Yingchun Wang noted that incentivizing certain individual outcomes (e.g., production rate) tends to crowd out other worker activities, such as process and quality improvement, which require greater teamwork.[i]

Professor Helper was kind enough to respond to an email about her work, and noted:

“Individual incentive pay is … problematic when individual contributions to performance are hard to measure… this is especially the case when good performance is hard to define and/or has many competing objectives as is often the case in a lot of modern activities, teaching included.  In these cases, group incentives, or even NO incentive (hourly pay) is better than individual incentives, because otherwise it’s hard to get individuals to engage in hard-to-measure activity (mentoring co-workers…), if the reward is for an easy to measure task that may be less important…”

Standardized test results are an easy measure of student achievement and teacher performance. But the trade-offs are unfortunate. Teachers are people too; people respond to incentives. If standardized tests are used to measure performance, teachers will respond accordingly: More test prep, less collegial collaboration. Over-quantification not only assumes teaching in a vacuum, it makes it more likely.

Further, teachers will set up incentives for students so that test prep will crowd out activities which hone hard-to-measure skills such as problem solving, creativity, decision-making, and teamwork. Over-quantification not only assumes that success is about attaining a certain score, it increases the likelihood that attainment of that score will be the only success.

Again, not a new argument. But I’d like to repackage the argument in the language of the quants. Can hard data make a case for the nuanced view of teacher and student success? Can it shape a message that breaks through the noise, reverses current trends, and changes the discussion? Let me ask several questions:

1 – What kinds of teamwork are significant and measurable in the teaching profession? Can we measure the degree to which teachers collaborate positively in a school?

2 – Is it possible to draw correlations between degree of teacher collaboration and student achievement? How exactly does teacher teamwork improve student performance?

3 – How to measure student achievement on hard-to-measure activities?

Producing data that helps teachers, policymakers, and the public understand what constitutes good teaching and real student achievement is a start towards knocking down the edifice the quants have built. Their numbers are simple because they ignore complexity. If we can’t make the complexity understandable, the quants’ simplistic view of education will reign unchallenged.

Granted, larger sociopolitical trends are at work, but trends are trends. They reverse. The tide will turn eventually, but we can do more to help the process along. What kind of data can be gathered? And how can we shape it into a more compelling message?

[i] Helper, Kleiner, Wang, Analyzing Compensation Methods in Manufacturing: Piece Rates, Time Rates, or Gain-Sharing. 2010, NBER Working Paper Series



Filed under how teachers teach

8 responses to “More Data Needed for Hard-To-Measure Student Learning and Teacher Quality (Guest blogger Stephen Lane)

  1. Cal

    I agree that a five minute check won’t tell you if a teacher is good or not. It assumes a homogenous classroom, for one thing.

    But I disagree that colleagues are a big part of teaching. That’s either feel-good happy talk or determined souls who have the ability to force everyone else to work with them. I’m at my third school, and only one of them (Sequoia) had any collaboration at all, and it was “light” collaboration (vry much to my comfort level). The other two schools have none. That’s very normal.

    I’m fine with value-added assessment, which is a good way to see if high school teachers are teaching the standards. But we need to accept that most kids will come in below grade level in many schools, so the tests need to evaluate lower level abilities at a much more granular level.

    • Stephen Lane

      Thanks for your comments. I agree that there is a wide variety of collaboration levels in different districts. All I can speak to is what interview data in my district showed – I don’t have data for a wider population (though I’d be very interested to see such data). I’d argue that a lack of collaboration is indeed one aspect of teachers’ influence on their colleagues – albeit probably a negative influence.
      On value added: I also think it could have its place, but the question on value-added assessment is the question of how one defines ‘value’. That’s the crux of the hard-to-measure / easy-to-measure issue. Test scores are easy to measure, but not necessarily the best metric for measuring a student’s growth and development. So do we use bad / incomplete metrics just because they are easy?

      • Cal

        See, I do think that test scores are the best assessment of a student’s academic development, and that’s what we should be looking for.

        I teach math (although I have credentials for English and history as well), and the standards issue isn’t much debated there. We math teachers might gripe that there are too many standards, given the low level abilities of many of our students, but we don’t dispute the standards themselves.

        In English and history, teachers feel quite differently. I know a lot of US History teachers who turn up their noses at teaching”facts” in favor of critical thinking. But those pesky standards expect the kids to know facts. When a teacher argues that the test isn’t capturing the student’s development because it tests facts, I am deeply skeptical. First, content is essential. Second, that’s what the teacher is committed to do, whether he or she likes it or not. English teachers have similar issues with standards. (Not sure about science).

        I don’t know if that’s what you mean when you argue that the test doesn’t capture what is important. If it isn’t, then I would still argue that standards and academic improvement are really the only important measures that society (as the funders) should care about.

        When we could take academic improvement for granted, we could get squishy.

        All that said, I think the ideal conclusion to value added testing should be the conclusion that our goals are too ambitious for many of our students. That way, we could back down on the academic content for all but the best students and teachers could go back to teaching “what is important”.

  2. Paul Muench

    Collect data on private schools. What are teachers doing in private schools?

    • Stephen Lane

      The comparisons might be interesting to draw; what would you expect them to show, and why do you think data from private schools would be particularly helpful?

      • Paul Muench

        Yes, comparing different environments is the main motivation. What do teachers, schools, and parent communities do differently when they have a different sets of resources, requirements, and motivations.

  3. megganj

    I am a qualitative researcher, about to graduate with a Phd in sociology. I’ve worked with my fair share of “quants,” who want to use my expertise to convert qualitative statements into quantitative item measures. Let’s just say it doesn’t work out the way they planned. Despite their frustration with the nuances of day-to-day human interaction, that doesn’t mean that we shouldn’t try to come up with more complex quantitative measures for learning.

    I would love to work in a policy/research position that tinkers with these ideas and tries to make them fit. Anyone know how to get their foot in the door for a job that would attempt such a thing?

  4. Pingback: More Data Needed for Hard-To-Measure Student Learning and Teacher Quality (Guest blogger Stephen Lane) | For Parents: About Teacher Effectiveness Policy |

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s