Measuring the Worth of Schooling and Medical Therapies: How Objective Are The Metrics?

Choosing the right metric to measure a medical therapy or school effectiveness is not a fact-filled, objective decision. It is subjective and packed with trade-offs.

Consider the metrics used since 1971 when President Richard Nixon signed legislation declaring a War on Cancer.*

In 1985 one researcher used the measure of how many lives were saved with chemotherapy and exams that detected the onset of cancer. He used the national cancer registry to determine how many Americans were diagnosed with cancer annually (about one million in 1985) and how many died of cancer in the same year (about 500,000). He then estimated how many cancer patients were “cured” (five years of remission) after heavy doses of chemotherapy alone (about 5,000) a year. He then estimated lives saved annually from chemotherapy given after surgery (35,000 to 40,000). Total: Less than one out of 20 people diagnosed with cancer in a given year and less than one in ten of the total patients who died of cancer had benefited from screening and chemotherapy (pp. 227-229). According to this metric, the War on Cancer had hardly dented the problem. Other measures were used subsequently such as “age-adjusted mortality” that showed even worse numbers than “lives saved.”  To medical researchers and practitioners, the nation was losing the War on Cancer.

Another medical researcher, however,  showed that if the measure was changed from “cured” to “years of life saved,” cancer was being beaten. He calculated that if chemotherapy cured a 5 year-old child of leukemia, that child lived at least 65 more years (assuming age 70 as life expectancy); then if chemotherapy cured a 65 year-old man, he would have five more years, given the typical life span of 70 years. Other metrics could not detect any differences in the two cases. But using the measure of “years of life saved,” the researcher showed that therapies and detection had, indeed, made substantial, progress in fighting cancer.

What’s the point?

Even with a shower of statistics, measuring success in fighting a serious illness remains subjective. The counting of deaths–an objective process–could be done in at least three different ways leading to opposite conclusions on the effectiveness of the overall effort to curb cancer.   As some readers may have noted, I have said nothing about what has historically reduced the incidence of killer diseases (e.g., tuberculosis, cholera, typhus, scurvy, pellagra, lung cancer).  No “magic bullet” pill or therapy reduced deaths but better sanitation, sewage,  housing, nutrition and public health campaigns such as anti-smoking did. In effect, prevention reduced deaths from disease.

Turn now to schools where objective standardized tests are the chief metrics used to judge national, state, district, school, principal, teacher, and, yes, individual student effectiveness. International test results provoke media frenzies and blogosphere blather over declining quality of U.S. schools compared to Asian nations. Test results–supposedly precise numbers–are used to judge effectiveness and, simultaneously, hold educators and children accountable for results.

As with the War on Cancer, subjective choices are made constantly by testing experts in reading, math, and academic subjects about which items to include in tests (e.g.,multiple choice, open-ended questions, writing samples). Policymakers decide which tests to use (e.g., NAEP, state tests, teacher-made ones) and where to set cut-off scores to determine student passing or advanced standing. Except for insiders in the testing industry, these subjective choices seldom leak out to the public or practitioners, much less to parents and students. And with a national climate for accountability dominant since the early 1980s, the  quest for better teaching and learning has come to wholly rely upon a surrogate for quality–the standardized achievement test.With pressure to use student-test scores–“value-added measures”–for evaluating teachers and pay-4-performance plans, the illusion of objectivity has driven underground the subjective choices that top officials make.

After all, a test is a proxy for quality, not the real thing–it is the sizzle, not the steak.  While the numbers produced from the tests have the patina of objectivity, the biases contained within the choices that psychometricians and policymakers make are less  obvious but are nonetheless there (see Koretz_JHR).

Like the ever-continuing search for a medical “magic bullet” to cure cancer, Parkinsons, and obesity, prevention, which has saved more lives in the past than any pill or therapy, is often a footnote rather than the text.  Early interventions at preschool age by health and education authorities have clearly established (also see here and here) that the ill effects of poor health, poverty, and neglect can be reduced substantially. These preventive actions have far more payoff now and in the future than the restless search for a knock-’em-dead innovation or later interventions in high schools and colleges.

While tests, like medical metrics, are worthwhile as signals for intervention, they remain subjective measures of quality. Change the metric, the outcome changes.

_______________

*The content on metrics (and page numbers) used to determine success/failure of the War on Cancer come from Siddartha Mukherjee, The Emperor of  All Maladies: A Biography of Cancer (2010)

Related Articles

• Measuring the rights things! (therunninglibrarian.co.uk)
• The Emperor of All Maladies: A Biography of Cancer by Siddhartha Mukherjee – review (guardian.co.uk)