Bias toward Numbers in Judging Teaching

Image via Wikipedia

In the U.S. people—yes, I include myself here—making decisions about important issues such as buying a home, picking a school for a five year-old or deciding on a college often give more weight to those features carrying numbers with them rather than qualitative features without numbers. Say, focusing on the square footage in the house vs. the feel of roominess. Or a teacher-student ratio in a kindergarten vs. sense of family that children and teacher communicate. From unemployment figures to batting averages and pass interceptions to calories, numbers carry far more weight with Americans than those variables that are harder to measure. Jonah Lehrer makes this point in one of his postings.

“Buying a car is a hard decision. There are just so many variables to think about. We’ve got to inspect the interior and analyze the engine, and research the reliability of the brand. And then, once we’ve amassed all these facts, we’ve got to compare different models.

How do we sift through this excess of information? When consumers are debating car alternatives, studies show that they tend to focus on variables they can quantify, such as horsepower and fuel economy…. We do this for predictable reasons. The amount of horsepower directly reflects the output of the engine, and the engine seems like something that should matter. (Nobody wants an underpowered car.) We also don’t want to spend all our money at the gas station, which is why we get obsessed with very slight differences in miles per gallon ratings.

Furthermore, these numerical attributes are easy to compare across cars: All we have to do is glance at the digits and see which model performs the best. And so a difficult choice becomes a simple math problem.

Unfortunately, this obsession with horsepower and fuel economy turns out to be a big mistake. The explanation is simple: The variables don’t matter nearly as much as we think. Just look at horsepower: When a team of economists analyzed the features that are closely related to lifetime car satisfaction, the power of the engine was near the bottom of the list. (Fuel economy was only slightly higher.) That’s because the typical driver rarely requires 300 horses or a turbocharged V-8. Although we like to imagine ourselves as Steve McQueen, accelerating into the curves, we actually spend most of our driving time stuck in traffic, idling at an intersection on the way to the supermarket. This is why, according to surveys of car owners, the factors that are most important turn out to be things like the soundness of the car frame, the comfort of the front seats and the aesthetics of the dashboard. These variables are harder to quantify, of course. But that doesn’t mean they don’t matter.”

Switch channels from buying cars to determining teacher effectiveness. Judging teacher effectiveness now means using multifactor algorithms with quantifiable variables including test scores and observers’ ratings while avoiding qualitative judgments about teacher practices that are hard to quantify. Examples: interviewing students after a teacher has praised their effort and persistence and seeing them glow. Or listening to students who remember teachers who applauded their self-control in difficult classroom situations. Or watching students in a class struggle with a problem that the teacher gave them that had no right answer to it.  Or see students who honored their favorite teachers by emulating  them as adults. Teacher blogger Stephen Lane  makes a similar point about the lack of metrics for things that really matter.

I end with Jonah Lehrer’s example that makes the same point vividly.

“When asked by David Remnick, in a 2000 New Yorker profile, how he felt about a cramped literary interpretation of one of his novels, Roth busted out a sports analogy. He imagined going to a baseball game with a little boy for the very first time. The kid doesn’t understand what’s happening on the field, and so his dad tells him to watch the scoreboard, to keep track of all the changing numbers. When the boy gets home someone asks him if he had fun at the game:

‘It was great!” he says. ‘The scoreboard changed thirty-two times and Daddy said last game it changed only fourteen times and the home team last time changed more times than the other team. It was really great! We had hot dogs and we stood up at one point to stretch and we went home.’ ”

But, of course, the boy would have missed the point of baseball.

And all the complex algorithms used in current plans to judge teacher performance too often ignore the hard-to-quantify variables that students, teachers, and parents value and remember years later.