Bias toward Numbers in Judging Teaching

New fuel economy label in 2008 shows estimated...

Image via Wikipedia

In the U.S. people—yes, I include myself here—making decisions about important issues such as buying a home, picking a school for a five year-old or deciding on a college often give more weight to those features carrying numbers with them rather than qualitative features without numbers. Say, focusing on the square footage in the house vs. the feel of roominess. Or a teacher-student ratio in a kindergarten vs. sense of family that children and teacher communicate. From unemployment figures to batting averages and pass interceptions to calories, numbers carry far more weight with Americans than those variables that are harder to measure. Jonah Lehrer makes this point in one of his postings.

“Buying a car is a hard decision. There are just so many variables to think about. We’ve got to inspect the interior and analyze the engine, and research the reliability of the brand. And then, once we’ve amassed all these facts, we’ve got to compare different models.

How do we sift through this excess of information? When consumers are debating car alternatives, studies show that they tend to focus on variables they can quantify, such as horsepower and fuel economy…. We do this for predictable reasons. The amount of horsepower directly reflects the output of the engine, and the engine seems like something that should matter. (Nobody wants an underpowered car.) We also don’t want to spend all our money at the gas station, which is why we get obsessed with very slight differences in miles per gallon ratings.

Furthermore, these numerical attributes are easy to compare across cars: All we have to do is glance at the digits and see which model performs the best. And so a difficult choice becomes a simple math problem.

Unfortunately, this obsession with horsepower and fuel economy turns out to be a big mistake. The explanation is simple: The variables don’t matter nearly as much as we think. Just look at horsepower: When a team of economists analyzed the features that are closely related to lifetime car satisfaction, the power of the engine was near the bottom of the list. (Fuel economy was only slightly higher.) That’s because the typical driver rarely requires 300 horses or a turbocharged V-8. Although we like to imagine ourselves as Steve McQueen, accelerating into the curves, we actually spend most of our driving time stuck in traffic, idling at an intersection on the way to the supermarket. This is why, according to surveys of car owners, the factors that are most important turn out to be things like the soundness of the car frame, the comfort of the front seats and the aesthetics of the dashboard. These variables are harder to quantify, of course. But that doesn’t mean they don’t matter.”

Switch channels from buying cars to determining teacher effectiveness. Judging teacher effectiveness now means using multifactor algorithms with quantifiable variables including test scores and observers’ ratings while avoiding qualitative judgments about teacher practices that are hard to quantify. Examples: interviewing students after a teacher has praised their effort and persistence and seeing them glow. Or listening to students who remember teachers who applauded their self-control in difficult classroom situations. Or watching students in a class struggle with a problem that the teacher gave them that had no right answer to it.  Or see students who honored their favorite teachers by emulating  them as adults. Teacher blogger Stephen Lane  makes a similar point about the lack of metrics for things that really matter.

I end with Jonah Lehrer’s example that makes the same point vividly.

“When asked by David Remnick, in a 2000 New Yorker profile, how he felt about a cramped literary interpretation of one of his novels, Roth busted out a sports analogy. He imagined going to a baseball game with a little boy for the very first time. The kid doesn’t understand what’s happening on the field, and so his dad tells him to watch the scoreboard, to keep track of all the changing numbers. When the boy gets home someone asks him if he had fun at the game:

‘It was great!” he says. ‘The scoreboard changed thirty-two times and Daddy said last game it changed only fourteen times and the home team last time changed more times than the other team. It was really great! We had hot dogs and we stood up at one point to stretch and we went home.’ ”

But, of course, the boy would have missed the point of baseball.

And all the complex algorithms used in current plans to judge teacher performance too often ignore the hard-to-quantify variables that students, teachers, and parents value and remember years later.



Filed under how teachers teach

9 responses to “Bias toward Numbers in Judging Teaching

  1. Bob Calder

    If we choose the wrong inputs to change education, then use those inputs to measure teacher effectiveness, what happens?

    I don’t think that’s a question for economists, though they can answer it. I think it’s a question for philosophy.

  2. If we compare the value-added scores for 100 teachers who praise persistence and effort (Dweck), focus on self-control (Bandura), and help students develop resilience in the face of challenges (Seligman) while teaching the content for which they are responsible with 100 teachers who praise results and intelligence, diminish self control, and teach learned helplessness, what would be the results? In other words, assuming a teacher knows the subject matter and actually teaches it, won’t these factors that you suggest show up in value-added scores?

    And if we teach the second set of teachers how to focus on strengths, build resilience, and strengthen relationships, isn’t it likely their value-added scores will go up, assuming they know and are teaching the subject? In fact, isn’t it likely we would see lingering effects on those students for years after they leave the teacher’s classroom? In fact, we do see such effects from teachers with high value-added scores – could this be the reason?

    Actually, these questions are relatively easy to answer, and Tennessee should have answered them years ago. Our bad.

  3. The strangest thing to me is the data focuses on 20-century skills using that same old paper and pencil testing. The truth is any child entering school today we graduate into a literate world that is not anything like the one we are testing them for. Schools tend to teach towards the test, and our children will suffer for it. Another issue that concerns me is no one is talking to students about how all of this is impacting the motivation to learn.
    In medical research nothing goes patient input. What is happening in education would be unethical in medical research.
    Thank you Dr. Cuban for another thought provoking blog.
    Children Are More Than Test Scores,
    Jesse Turner

  4. tkilbourn

    What we know and how we teach often depends on the questions we ask and the authors we read. I taught for forty-one years, retiring this past June, and as I moved toward the closure (How I detest that Madeline Hunter term!), of my last set of classes, I continued to be aware of how insipid the numbers game is regarding the authentic doings of teaching and learning. The essential paradox of psychology, an investigation into why people do what they do, is how it has confused quantitative study with qualitative in-seeing. More, James Hillman is correct in saying the mistake psychology made in the 20th century is the adoption of a medical model. (See his Re-Visioning Psychology) Sounds like a non-sequitur, but it is not.
    For consideration, here is a long quotation from the introduction to Erich Heller’s The Disinherited Mind. Consider the ramifications:

    “It is true that his [the teacher of literature] devotion to literature is capable of purging his affections of too narrowly subjective and emotional elements; yet his comprehension will remain largely determined by his own character, his spontaneous sympathies or antipathies, the happiness he has enjoyed or the disasters that have befallen him. And this, he will see, is no shortcoming of his own discipline, to be conquered in scientific campaigns or disguised by scientific masquerades, but is in fact its distinctive virtue. For the ultimate concern of his subject is neither facts nor classifications, neither patterns of cause and effect nor technical complexities. Of course, strict honesty in the face of facts and a certain mastery in dealing with their manifold interconnections are the indispensable qualifications of the literary scholar. In the end, however, he is concerned with the communication of a sense of quality rather than measurable quantity, and of meaning rather than explanation.

    Thus he would be ill-advised to concentrate exclusively on those aspects of his discipline which allow the calm neutrality of what is indisputably factual and ‘objective’. His business is, I think, not the avoidance of subjectivity, but its purification; not the shunning of what is disputable, but the cleansing and deepening of the dispute. As a teacher he is involved in a task which would appear impossible by the standards of the scientific laboratory: to teach what, strictly speaking, cannot be taught, but only ‘caught’, like a passion, a vice or a virtue. This ‘impossiblity’ is the inspiration of his work.”

    Jerome Bruner, one of the founders of the cognitive revolution with its emphasis on quantification, says in his book, Acts of Meaning, that the adoption of a quantification metaphor was an utter mistake.

    Alas, how to shift the metaphor? How to change the base narrative of equating learning with “how much”?

    Changing semantic habits is no easy matter!!

    Tom Kilbourn

    • larrycuban

      Hi Tom,
      Your comments and quotes are apt. The mix of subjectivity and objectivity in teaching kindergarten and high school seniors, physics and literature shifts with subject matter, student’s age, and other contextual factors, as I see it. The quantification metaphor in teaching and learning, as you know, is not as recent as The Nation at Risk report in 1983; it extends back to educators embracing Frederick Taylor’s “scientific management” in the decade before World War I. I, for one, want more balance to qualitative and quantitative metrics than exist now. I do not want to eliminate numbers from teaching and learning. I just want a more informed conversation between educators and non-educators over what gets measured, how it is measured, and the hard-to-measure–but significant–factors that you refer to.

  5. Pingback: Larry Ferlazzo’s Websites of the Day… - This Week’s “Round-Up” Of Good School Reform Posts & Articles.

  6. Pingback: The Best Articles & Posts On Education Policy In 2011 — Part Two | Larry Ferlazzo’s Websites of the Day…

  7. Pingback: The Six Biggest Problems with Data-Driven Instruction | gadflyonthewallblog

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s