Tag Archives: research and practice

Evidence Based Education Policy and Practice: A Conversation (Francis Schrag)

 

This fictitious exchange between two passionate educators over making educational policy and influencing classroom practice through careful scrutiny of evidence–such as has occurred in medicine and the natural sciences–as opposed to relying on professional judgment anchored in expertise gathered in schools brings out a fundamental difference among educators and the public that has marked public debate over the past three decades. The center of gravity in making educational policy in the U.S. has shifted from counting resources that go into schooling and relying on professional judgment to counting outcomes students derive from their years in schools and what the numbers say.

That shift can be dated from the Elementary and Secondary Education Act of 1965 but gained sufficient traction after the Nation at Risk report (1983) to dominate debate over innovation, policy, and practice. Although this is one of the longest guest posts I have published, I found it useful (and hope that viewers will as well) in making sense of a central conflict that exist today within and among school reformers, researchers, teachers, policymakers and parents.

Francis Schrag is professor emeritus in the philosophy of education at the University of Wisconsin, Madison. This article appeared in Teachers College Record, March 14, 2014.

A dialogue between a proponent and opponent of Evidence Based Education Policy. Each position is stated forcefully and each reader must decide who has the best of the argument.

Danielle, a professor of educational psychology and Leo, a school board member and former elementary school teacher and principal, visit a middle-school classroom in Portland Maine where students are deeply engaged in building robots out of Lego materials, robots that will be pitted against other robots in contests of strength and agility.  The project requires them to make use of concepts they’ve learned in math and physics.  Everything suggests that the students are deeply absorbed in what is surely a challenging activity, barely glancing around to see who has entered their classroom.

Leo:  Now this is exciting education. This is what we should be moving towards.  I wish all teachers could see this classroom in action.

Danielle:  Not so fast.  I’ll withhold judgment till I have some data.  Let’s see how their math and science scores at the end of the year compare with those of the conventional classroom we visited this morning.  Granted that one didn’t look too out of the ordinary, but the teacher was really working to get the kids to master the material.

Leo:  I don’t see why you need to wait.  Can’t you see the difference in level of engagement in the two classrooms?  Don’t you think the students will remember this experience long after they’ve forgotten the formula for angular momentum? Your hesitation reminds me of a satirical article a friend showed me; I think it came from a British medical journal.  As I recall the headline went: “Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomized controlled trials.”

Danielle:  Very cute, but let’s get serious.  Spontaneous reactions can be misleading; things aren’t always what they appear to be, as I’m sure you’ll agree.  I grant you that it looks as if the kids in this room are engaged, but we don’t know whether they’re engaged in the prescribed tasks and we don’t know what they’re actually learning, do we?  We’ll have a much better idea when we see the comparative scores on the test.  The problem with educators is that they get taken in with what looks like it works, they go with hunches, and what’s in fashion, but haven’t learned to consult data to see what actually does work.  If physicians hadn’t learned to consult data before prescribing, bloodletting would still be a popular treatment.

Suppose you and I agreed on the need for students to study math and physics.  And suppose that it turned out that the kids in the more conventional classroom learned a lot more math and physics, on average, as measured on tests, than the kids in the robotics classroom.  Would you feel a need to change your mind about what we’ve just seen?  And, if not, shouldn’t you?  Physicians are now on board with Evidence Based Medicine (EBM) in general, and randomized controlled trials (RCTs) in particular, as the best sources of evidence.  Why are teachers so allergic to the scientific method?  It’s the best approach we have to determine educational policy.

Leo:  Slow down Danielle.  You may recall that a sophisticated RCT convincingly showed the benefits of smaller class sizes in elementary schools in Tennessee, but these results were not replicated when California reduced its elementary school class size, because there was neither room in the schools for additional classrooms nor enough highly skilled teachers to staff them.  This example is used by Nancy Cartwright and Jeremy Hardie in their book on evidence-based policy to show that the effectiveness of a policy depends, not simply on the causal properties of the policy itself, but on what they call a “team” of support factors (2012, p. 25).  If any one of these factors were present in the setting where the trial was conducted but is lacking in the new setting, the beneficial results will not be produced.  This lack of generalizability, by the way, afflicts RCTs in medicine too.  For instance, the populations enrolled in teaching hospital RCTs are often different from those visiting their primary care physician.

Danielle:  I have to agree that educators often extrapolate from RCTs in a way that’s unwarranted, but aren’t you, in effect, calling for the collection of more and better evidence, rather than urging the abandonment of the scientific approach.  After all, the Cartwright and Hardie book wasn’t written to urge policy makers to throw out the scientific approach and go back to so-called expert or professional judgment, which may be no more than prejudice or illicit extrapolation based on anecdotal evidence.

Leo:  You seem to be willing to trust the data more than the judgment of seasoned professionals.  Don’t you think the many hours of observing and teaching in actual classrooms counts for anything?

Danielle: If your district has to decide which program to run, the robotics or the traditional, do you really want to base your decision on the judgment of individual teachers or principals, to say nothing of parents and interested citizens?  In medicine and other fields, meta-analyses have repeatedly shown that individual clinical judgment is more prone to error than decisions based on statistical evidence (Howick, 2011, Chap. 11). And, as I already mentioned, many of the accepted therapies of earlier periods, from bloodletting to hormone replacement therapy, turned out to be worse for the patients than doing nothing at all.

Now why should education be different?  How many teachers have “known” that the so-called whole-word method was the best approach to teaching reading, and years later found out from well-designed studies that this is simply untrue?  How many have “known” that children learn more in smaller classes?  No, even if RCTs aren’t always the way to go, I don’t think we can leave these things to individual educator judgment; it’s too fallible.

And you may not need to run a new study on the question at issue.  There may already be relevant, rigorous studies out there, testing more exploratory classrooms against more traditional ones in the science and math area for middle-schoolers.  I recommend you look at the federal government What Works website, which keeps track of trial results you can rely on.

Leo:  I’ve looked at many of these studies, and I have two problems with them.  They typically use test score gains as their indicator of durable educational value, but these can be very misleading.  Incidentally, there’s a parallel criticism of the use of “surrogate end points” like blood levels in medical trials.  Moreover, according to Goodhart’s Law—he was a British economist—once a measure becomes a target, it ceases to be a good indicator.  This is precisely what happens in education: the more intensely we focus on raising a test score by means of increasing test preparation to say nothing of cheating—everything from making sure the weakest, students don’t take the test to outright changing students’ answers—the less it tells us about what kids can do or will do outside the test situation.

Danielle:  Of course we need to be careful about an exclusive reliance on test scores.  But you can’t indict an entire approach because it has been misused on occasion.

Leo: I said there was a second problem, as well.  You recall that what impressed us about the robotics classroom was the level of involvement of the kids.  When you go into a traditional classroom, the kids will always look at the door to see who’s coming in.  That’s because they’re bored and looking for a bit of distraction.  Now ask yourself, what does that involvement betoken. It means that they’re learning that science is more than memorizing a bunch of facts, that math is more than solving problems that have no meaning or salience in the real world, that using knowledge and engaging in hard thinking in support of a goal you’ve invested in is one of life’s great satisfactions.  Most kids hate math and the American public is one of the most scientifically illiterate in the developed world.  Why is that?  Perhaps it’s because kids have rarely used the knowledge they are acquiring to do anything besides solve problems set by the teacher or textbook.

I’m sure you recall from your studies in philosophy of education the way John Dewey called our attention in Experience and Education to what he called, the greatest pedagogical fallacy, “the notion that a person learns only the particular thing he is studying at the time” (Dewey, 1938, p. 48).  Dewey went on to say that what he called “collateral learning,” the formation of “enduring attitudes” was often much more important than the particular lesson, and he cited the desire to go on learning as the most important attitude of all.  Now when I look at that robotics classroom, I can see that those students are not just learning a particular lesson, they’re experiencing the excitement that can lead to a lifetime of interest in science or engineering even if they don’t select a STEM field to specialize in.

Danielle:  I understand what Dewey is saying about “collateral learning.”  In medicine as you know, side effects are never ignored, and I don’t deny that we in education are well behind our medical colleagues in that respect.  Still, I’m not sure I agree with you and Dewey about what’s most important, but suppose I do.  Why are you so sure that the kids’ obvious involvement in the robotics activity will generate the continuing motivation to keep on learning?  Isn’t it possible that a stronger mastery of subject matter will have the very impact you seek?  How can we tell?  We’d need to first find a way to measure that “collateral learning,” then preferably conduct a randomized, controlled trial, to determine which of us is right.

Leo:  I just don’t see how you can measure something like the desire to go on learning, yet, and here I agree with Dewey, it may be the most important educational outcome of all.

Danielle:  This is a measurement challenge to be sure, but not an insurmountable one.  Here’s one idea: let’s track student choices subsequent to particular experiences.  For example, in a clinical trial comparing our robotics class with a conventional middle school math and science curriculum, we could track student choices of math and science courses in high school.  Examination of their high school transcripts could supply needed data.  Or we could ask whether students taking the robotics class in middle school were more likely (than peers not selected for the program) to take math courses in high school, to major in math or science in college, etc.  Randomized, longitudinal designs are the most valid, but I admit they are costly and take time.

Leo: I’d rather all that money went into the kids and classrooms.

Danielle:  I’d agree with you if we knew how to spend it to improve education.  But we don’t, and if you’re representative of people involved in making policy at the school district level, to say nothing of teachers brainwashed in the Deweyian approach by teacher educators, we never will.

Leo:  That’s a low blow, Danielle, but I haven’t even articulated my most fundamental disagreement with your whole approach, your obsession with measurement and quantification, at the expense of children and education.

Danielle:  I’m not sure I want to hear this, but I did promise to hear you out.  Go ahead.

Leo:  We’ve had about a dozen years since the passage of the No Child Left Behind Act to see what an obsessive focus on test scores looks like and it’s not pretty.  More and more time is taken up with test-prep, especially strategies for selecting right answers to multiple-choice questions.  Not a few teachers and principals succumb to the temptation to cheat, as I’m sure you’ve read.  Teachers are getting more demoralized each year, and the most creative novice teachers are finding jobs in private schools or simply not entering the profession.  Meanwhile administrators try to game the system and spin the results.  But even they have lost power to the statisticians and other quantitatively oriented scholars, who are the only ones who can understand and interpret the test results.  Have you seen the articles in measurement journals, the arcane vocabulary and esoteric formulas on nearly every page?

And do I have to add that greedy entrepreneurs with a constant eye on their bottom lines persuade the public schools to outsource more and more of their functions, including teaching itself.  This weakens our democracy and our sense of community.  And even after all those enormous social costs, the results on the National Assessment of Educational Progress are basically flat and the gap between black and white academic achievement—the impetus for passing NCLB in the first place—is as great as it ever was.

Danielle:  I agree that it’s a dismal spectacle.  You talk as if educators had been adhering to Evidence Based Policy for the last dozen years, but I’m here to tell you they haven’t and that’s the main reason, I’d contend, that we’re in the hole that we are.  If educators were less resistant to the scientific approach, we’d be in better shape today.  Physicians have learned to deal with quantitative data, why can’t teachers, or are you telling me they’re not smart enough?  Anyhow, I hope you feel better now that you’ve unloaded that tirade of criticisms.

Leo:  Actually, I’m not through, because I don’t think we’ve gotten to the heart of the matter yet.

Danielle:  I’m all ears.

Leo:  No need to be sarcastic, Danielle.  Does the name Michel Foucault mean anything to you?  He was a French historian and philosopher.

Danielle:  Sure, I’ve heard of him.  A few of my colleagues in the school of education, though not in my department, are very enthusiastic about his work.  I tried reading him, but I found it tough going.  Looked like a lot of speculation with little data to back it up.  How is his work relevant?

Leo:   In Discipline and Punish, Foucault described the way knowledge and power are intertwined, especially in the human sciences, and he used the history of the school examination as a way of illustrating his thesis (1975/1995, pp. 184-194).  Examinations provide a way of discovering “facts” about individual students, and a way of placing every student on the continuum of test-takers.  At the same time, the examination provides the examiners, scorers and those who make use of the scores ways to exercise power over kids’ futures.  Think of the Scholastic Assessment Tests (SATs) for example.  Every kid’s score can be represented by a number and kids can be ranked from those scoring a low of 600 to those with perfect scores of 2400.  Your score is a big determinant of what colleges will even consider you for admission.  But that’s not all: Foucault argued that these attempts to quantify human attributes create new categories of young people and thereby determine how they view themselves.  If you get a perfect SAT score, or earn “straight As” on your report card, that becomes a big part of the way others see you and how you see yourself.  And likewise for the mediocre scorers, the “C” students, or the low scorers who not only have many futures closed to them, but may see themselves as “losers,” “failures,” “screw-ups.”  A minority may, of course resist and rebel against their placement on the scale—consider themselves to be “cool”, unlike the “nerds” who study, but that won’t change their position on the continuum or their opportunities.  Indeed, it may limit them further as they come to be labeled “misfits” “ teens at-risk,” “gang-bangers” and the like. But, and here’s my main point, this entire system is only possible due to our willingness to represent the capabilities and limitations of children and young people by numerical quantities.  It’s nothing but scientism, the delusive attempt to force the qualitative, quirky, amazingly variegated human world into a sterile quantitative straight-jacket.  You recall the statement that has been attributed to Einstein, don’t you, “Not everything that can be counted counts, and not everything that counts can be counted.” I just don’t understand your refusal to grasp that basic point; it drives me mad.

Danielle:  Calm down, Leo.  I don’t disagree that reducing individuals to numbers can be a problem; every technology has a dark side, I’ll grant you that, but think it through.  Do you really want to go back to a time when college admissions folks used “qualitative” judgments to determine admissions?  When interviewers could tell from meeting a candidate or receiving a letter of recommendation if he were a member of “our crowd,” would know how to conduct himself at a football game, cocktail party, or chapel service, spoke without an accent, wasn’t a grubby Jew or worse, a “primitive” black man or foreign-born anarchist or communist.  You noticed I used the masculine pronoun:  Women, remember, were known to be incapable of serious intellectual work, no data were needed, the evidence was right there in plain sight.  Your Foucault is not much of a historian, I think.

Leo:  We have some pretty basic disagreements here.  I know we each believe we’re right.  Is there any way to settle the disagreement?

Danielle:  I can imagine a comprehensive, longitudinal experiment in a variety of communities, some of which would carry out EBEP and control communities that would eschew all use of quantification.  After a long enough time, maybe twenty years, we’d take a look at which communities were advancing, which were regressing.  Of course, this is just an idea; no one would pay to actually have it done.

Leo:  But even if we conducted such an experiment, how would we know which approach was successful?

Danielle:  We shouldn’t depend on a single measure, of course.  I suggest we use a variety of measures, high school graduation rate, college attendance, scores on the National Assessment of Educational Progress, SATs, state achievement tests, annual income in mid-career, and so on.  And, of course, we could analyze the scores by subgroups within communities to see just what was going on.

Leo:  Danielle, I can’t believe it.  You haven’t listened to a word I’ve said.

Danielle:  What do you mean?

Leo:   If my favored policy is to eschew quantitative evidence altogether, wouldn’t I be inconsistent if I permitted the experiment to be decided by quantitative evidence, such as NAEP scores or worse, annual incomes?  Don’t you recall that I reject your fundamental assumption—that durable, significant consequences of educational experiences can be represented as quantities?

Danielle:  Now I’m the one that’s about to scream.  Perhaps you could assess a single student’s progress by looking at her portfolio at the beginning and end of the school year.  How, in the absence of quantification, though, can you evaluate an educational policy that affects many thousands of students?  Even if you had a portfolio for each student, you’d still need some way to aggregate them in order to be in a position to make a judgment about the policy or program that generated those portfolios.  You gave me that Einstein quote to clinch your argument.  Well, let me rebut that with a quotation by another famous and original thinker, the Marquis de Condorcet, an eighteenth century French philosopher and social theorist.  Here’s what he said:  “if this evidence cannot be weighted and measured, and if these effects cannot be subjected to precise measurement, then we cannot know exactly how much good or evil they contain” (Condorcet, 2012, p.138).  The point remains true, whether in education or medicine.  If you can’t accept it, I regret to say, we’ve reached the end of the conversation.

References

Cartwright, N & Hardie, J. (2012). Evidence-based policy:  A practical guide to doing it better.  Oxford and New York: Oxford University Press.

Condorcet, M. (2012). The sketch. In S. Lukes, and N. Urbinati (Eds.), Political Writings (pp. 1-147). Cambridge: Cambridge University Press.

Dewey, J. (1938/1973). Experience and education.  New York: Collier Macmillan Publishers.

Foucault, M. (1995).  Discipline and punish: The birth of the prison. (A. Sheridan, Trans.) New York: Vintage Books. (Original work published in 1975)

Howick, J. (2011). The Philosophy of evidence-based medicine. Oxford: Blackwell Publishing.

21 Comments

Filed under comparing medicine and education, school reform policies

What’s The Evidence on School Devices and Software Improving Student Learning?

The historical record is rich in evidence that research findings have played a subordinate role in making educational policy. Often, policy choices were (and are) political decisions. There was no research, for example, that found establishing tax-supported public schools in the early 19th century was better than educating youth through private academies. No studies persuaded late-19th century educators to import the kindergarten into public schools. Ditto for bringing computers into schools a century later.

So it is hardly surprising, then, that many others, including myself, have been skeptical of the popular idea that evidence-based policymaking and evidence-based instruction can drive teaching practice. Those doubts have grown larger when one notes what has occurred in clinical medicine with its frequent U-turns in evidence-based “best practices.”

Consider, for example, how new studies have often reversed prior “evidence-based” medical procedures.

*Hormone therapy for post-menopausal women to reduce heart attacks was found to be more harmful than no intervention at all.

*Getting a PSA test to determine whether the prostate gland showed signs of cancer for men over the age of 50 was “best practice” until 2012 when advisory panels of doctors recommended that no one under 55 should be tested and those older  might be tested if they had family histories of prostate cancer.

And then there are new studies that recommend women to have annual mammograms, not at age  50 as recommended for decades, but at age 40. Or research syntheses (sometimes called “meta-analyses”) that showed anti-depressant pills worked no better than placebos.

These large studies done with randomized clinical trials–the current gold standard for producing evidence-based medical practice–have, over time, produced reversals in practice. Such turnarounds, when popularized in the press (although media attention does not mean that practitioners actually change what they do with patients) often diminished faith in medical research leaving most of us–and I include myself–stuck as to which healthy practices we should continue and which we should drop.

Should I, for example, eat butter or margarine to prevent a heart attack? In the 1980s, the answer was: Don’t eat butter, cheese, beef, and similar high-saturated fat products. Yet a recent meta-analysis of those and subsequent studies reached an opposite conclusion.

Figuring out what to do is hard because I, as a researcher, teacher, and person who wants to maintain good health has to sort out what studies say and  how those studies were done from what the media report, and then how all of that applies to me. Should I take a PSA test? Should I switch from margarine to butter?

If research into clinical medicine produces doubt about evidence-based practice, consider the difficulties of educational research–already playing a secondary role in making policy and practice decisions–when findings from long-term studies of innovation conflict with current practices. Look, for example, at computer use to transform teaching and improve student achievement.

Politically smart state and local policymakers believe that buying new tablets loaded with new software, deploying them to K-12 classrooms, and watching how the devices engage both teachers and students is a “best practice.” The theory is that student engagement through the device and software will dramatically alter classroom instruction and lead to improved  achievement. The problem, of course–sure, you already guessed where I was going with this example–is that evidence of this electronic innovation transforming teaching and achievement growth is not only sparse but also unpersuasive even when some studies show a small “effect size.”

Turn now to the work of John Hattie, a Professor at the University of Auckland (NZ), who has synthesized the research on different factors that influence student achievement and measured their impact on learning. For example, over the last two decades, Hattie has examined over 180,000 studies accumulating 200, 000 “effect sizes”  measuring the influence of teaching practices on student learning. All of these studies represent over 50 million students.

He established which factors influenced student learning–the “effect size–by ranking each from 0.1 (hardly any influence) to 1.0 or a full standard deviation–almost a year’s growth in student learning. He found that the “typical” effect size of an innovation was 0.4.

To compare different classroom approaches shaped student learning, Hattie used the “typical” effect size (0.4) to mean that a practice reached the threshold of influence on student learning (p. 5). From his meta-analyses, he then found that class size had a .20 effect (slide 15) while direct instruction had a .59 effect (slide 21). Again and again, he found that teacher feedback had an effect size of .72 (slide 32). Moreover, teacher-directed strategies of increasing student verbalization (.67) and teaching meta-cognition strategies (.67) had substantial effects (slide 32).

What about student use of computers (p. 7)? Hattie included many “effect sizes” of computer use from distance education (.09), multimedia methods (.15), programmed instruction (.24), and computer-assisted instruction (.37). Except for “hypermedia instruction” (.41), all fell below the “typical ” effect size (.40) of innovations improving student learning (slides 14-18). Across all studies of computers, then, Hattie found an overall effect size of .31 (p. 4).

According to Hattie’s meta-analyses, then, introducing computers to students will  fall well below other instructional strategies that teachers can and do use. Will Hattie’s findings convince educational policymakers to focus more on teaching? Not as long as political choices trump research findings.

Even if politics were removed from the decision-making equation, there would still remain the major limitation of  most educational and medical research. Few studies  answer the question: under what conditions and with which students and patients does a treatment work? That question seldom appears in randomized clinical trials. And that is regrettable.

 

 

26 Comments

Filed under comparing medicine and education, how teachers teach, technology use

Don’t Help Your Kids With Their Homework (Dana Goldstein)

Dana Goldstein is a Brooklyn-based journalist, a Schwartz Fellow at the New America Foundation, and a Puffin Fellow at the Nation Institute. This article appeared March 19, 2014 in Atlantic Online

One of the central tenets of raising kids in America is that parents should be actively involved in their children’s education: meeting with teachers, volunteering at school, helping with homework, and doing a hundred other things that few working parents have time for. These obligations are so baked into American values that few parents stop to ask whether they’re worth the effort.

Until this January, few researchers did, either. In the largest-ever study of how parental involvement affects academic achievement, Keith Robinson, a sociology professor at the University of Texas at Austin, and Angel L. Harris, a sociology professor at Duke, mostly found that it doesn’t. The researchers combed through nearly three decades’ worth of longitudinal surveys of American parents and tracked 63 different measures of parental participation in kids’ academic lives, from helping them with homework, to talking with them about college plans, to volunteering at their schools. In an attempt to show whether the kids of more-involved parents improved over time, the researchers indexed these measures to children’s academic performance, including test scores in reading and math.

What they found surprised them. Most measurable forms of parental involvement seem to yield few academic dividends for kids, or even to backfire—regardless of a parent’s race, class, or level of education.

Do you review your daughter’s homework every night? Robinson and Harris’s data, published in The Broken Compass: Parental Involvement With Children’s Education, show that this won’t help her score higher on standardized tests. Once kids enter middle school, parental help with homework can actually bring test scores down, an effect Robinson says could be caused by the fact that many parents may have forgotten, or never truly understood, the material their children learn in school.

Similarly, students whose parents frequently meet with teachers and principals don’t seem to improve faster than academically comparable peers whose parents are less present at school. Other essentially useless parenting interventions: observing a kid’s class; helping a teenager choose high-school courses; and, especially, disciplinary measures such as punishing kids for getting bad grades or instituting strict rules about when and how homework gets done. This kind of meddling could leave children more anxious than enthusiastic about school, Robinson speculates. “Ask them ‘Do you want to see me volunteering more?

Going to school social functions? Is it helpful if I help you with homework?’ ” he told me. “We think about informing parents and schools what they need to do, but too often we leave the child out of the conversation.”

One of the reasons parental involvement in schools has become dogma is that the government actively incentivizes it. Since the late 1960s, the federal government has spent hundreds of millions of dollars on programs that seek to engage parents—especially low-income parents—with their children’s schools. In 2001, No Child Left Behind required schools to establish parent committees and communicate with parents in their native languages. The theory was that more active and invested mothers and fathers could help close the test-score gap between middle-class and poor students. Yet until the new study, nobody had used the available data to test the assumption that close relationships between parents and schools improve student achievement.

While Robinson and Harris largely disproved that assumption, they did find a handful of habits that make a difference, such as reading aloud to young kids (fewer than half of whom are read to daily) and talking with teenagers about college plans. But these interventions don’t take place at school or in the presence of teachers, where policy makers exert the most influence—they take place at home.

What’s more, although conventional wisdom holds that poor children do badly in school because their parents don’t care about education, the opposite is true. Across race, class, and education level, the vast majority of American parents report that they speak with their kids about the importance of good grades and hope that they will attend college. Asian American kids may perform inordinately well on tests, for example, but their parents are not much more involved at school than Hispanic parents are—not surprising, given that both groups experience language barriers. So why are some parents more effective at helping their children translate these shared values into achievement?

Robinson and Harris posit that greater financial and educational resources allow some parents to embed their children in neighborhoods and social settings in which they meet many college-educated adults with interesting careers. Upper-middle-class kids aren’t just told a good education will help them succeed in life. They are surrounded by family and friends who work as doctors, lawyers, and engineers and who reminisce about their college years around the dinner table. Asian parents are an interesting exception; even when they are poor and unable to provide these types of social settings, they seem to be able to communicate the value and appeal of education in a similarly effective manner.

As part of his research, Robinson conducted informal focus groups with his undergraduate statistics students at the University of Texas, asking them about how their parents contributed to their achievements. He found that most had few or no memories of their parents pushing or prodding them or getting involved at school in formal ways. Instead, students described mothers and fathers who set high expectations and then stepped back. “These kids made it!,” Robinson told me. “You’d expect they’d have the type of parental involvement we’re promoting at the national level. But they hardly had any of that. It really blew me away.”

Robinson and Harris’s findings add to what we know from previous research by the sociologist Annette Lareau, who observed conversations in homes between parents and kids during the 1990s. Lareau found that in poor and working-class households, children were urged to stay quiet and show deference to adult authority figures such as teachers. In middle-class households, kids learned to ask critical questions and to advocate for themselves—behaviors that served them well in the classroom.

Robinson and Harris chose not to address a few potentially powerful types of parental involvement, from hiring tutors or therapists for kids who are struggling, to opening college savings accounts. And there’s the fact that, regardless of socioeconomic status, some parents go to great lengths to seek out effective schools for their children, while others accept the status quo at the school around the corner.

Although Robinson and Harris didn’t look at school choice, they did find that one of the few ways parents can improve their kids’ academic performance—by as much as eight points on a reading or math test—is by getting them placed in the classroom of a teacher with a good reputation. This is one example for which race did seem to matter: white parents are at least twice as likely as black and Latino parents to request a specific teacher. Given that the best teachers have been shown to raise students’ lifetime earnings and to decrease the likelihood of teen pregnancy, this is no small intervention.

All in all, these findings should relieve anxious parents struggling to make time to volunteer at the PTA bake sale. But valuing parental involvement via test scores alone misses one of the ways in which parents most impact schools. Pesky parents are often effective, especially in public schools, at securing better textbooks, new playgrounds, and all the “extras” that make an educational community come to life, like art, music, theater, and after-school clubs. This kind of parental engagement may not directly affect test scores, but it can make school a more positive place for all kids, regardless of what their parents do or don’t do at home. Getting involved in your children’s schools is not just a way to give them a leg up—it could also be good citizenship.

 

11 Comments

Filed under raising children

The Seductive Lure of Big Data: Practitioners Beware

Big Data beckons policymakers, administrators and teachers with eye-popping analytics and snazzy graphics. Here is Darrell West of the Brookings Institition laying out the case for teachers and administrators to use Big Data:

Twelve-year-old Susan took a course designed to improve her reading skills. She read short stories and the teacher would give her and her fellow students a written test every other week measuring vocabulary and reading comprehension. A few days later, Susan’s instructor graded the paper and returned her exam. The test showed that she did well on vocabulary, but needed to work on retaining key concepts.

In the future, her younger brother Richard is likely to learn reading through a computerized software program. As he goes through each story, the computer will collect data on how long it takes him to master the material. After each assignment, a quiz will pop up on his screen and ask questions concerning vocabulary and reading comprehension. As he answers each item, Richard will get instant feedback showing whether his answer is correct and how his performance compares to classmates and students across the country. For items that are difficult, the computer will send him links to websites that explain words and concepts in greater detail. At the end of the session, his teacher will receive an automated readout on Richard and the other students in the class summarizing their reading time, vocabulary knowledge, reading comprehension, and use of supplemental electronic resources.

In comparing these two learning environments, it is apparent that current school evaluations suffer from several limitations. Many of the typical pedagogies provide little immediate feedback to students, require teachers to spend hours grading routine assignments, aren’t very proactive about showing students how to improve comprehension, and fail to take advantage of digital resources that can improve the learning process. This is unfortunate because data-driven approaches make it possible to study learning in real-time and offer systematic feedback to students and teachers (education technology west-1).

West sees teachers and administrators as data scientists mining information, tracking individual student and teacher performance and making subsequent changes based on the data. Unfortunately, so much of the hype for using Big Data ignores time, place, and people.

Context matters.

Consider what occurred when Nick Bilton, a New York University journalist and adjunct professor designed a project for his graduate students in a course called “Telling Stories with Data, Sensors, and Humans.” Could sensors, Bilton and students asked, be reporters, collect information, and tell what happened?

The students built small electronic machines with sensors that could detect motion, light, and sound. They then asked the straightforward question whether students in the high-rise classroom building used the elevators more than the stairs  and whether they shifted from one to the other during the day. They set the device in some elevators and stairwells. Instead of a human counting students, a machine did.

Bilton and his graduate students were delighted with the results. They found that students seemed to use the elevators in the morning “perhaps because they were tired from staying up late, and switch to the stairs at night, when they became energized.”

That night when Bilton was leaving the building, the security guard who watched students set up the devices in elevators asked him what happened with the experiment. Bilton said that the sensors had captured students taking elevators in morning and stairs at night. The security guard laughed and told Bilton: “One of the elevators broke down a few evenings last week, so they had no choice but to use the stairs.”

Context matters.

In mining data, using analytics, and reading dashboards (see DreamBox) for classrooms and schools, the setting, time, and the quality of adult-student relationships count also. For Darrell West and others who see teachers and students profiting from instantaneous feedback from computers, context is absent. They fail to consider that the age-graded school is required to do far more than stuff information into students. They fail to reckon with the age-old wisdom (and research to support it) that effective student learning beyond test scores resides in the relationship between student and teacher.

And when it comes to evaluating individual teachers on the basis of student test scores, the  context of teaching–as complex an endeavor as can be imagined, one that is only partially mapped by researchers–trumps Big Data even when it is amply funded by Big Donors.

Big Data, of course, will be (and is) used by policymakers and administrators for tracking school and district performance and accountability. But the seductive lure of mining data and creating glossy dashboards will entice many educators to grab numbers to shape lessons and judge individual students and teachers. If they do succumb to the seduction without considering the complex context of teaching and learning, they risk making mistakes that will harm both teachers and students.

19 Comments

Filed under school reform policies

Evidence: The Case of the Common Core Standards

I have admired Rodin’s statue of “The Thinker” for many years.

Yet the statue is not a man of action.

Too much thinking, too little action is a recipe for fecklessness. Yet too much action, too little thought are ingredients for a potential disaster.*

And this is where the Common Core standards enter the picture.

Exactly how much evidence did policymakers have to justify the crafting and adoption of national standards?  Of that evidence supporting the policy, what part, if any, did research play in making policy? Since evidence never speaks for itself–it has to be interpreted–these are fair questions to ask of any policy but especially one with high-stakes consequences for how teachers teach to the standards, what children and youth study in classrooms lessons, and tests used to measure how much of the standards students have learned.

There have been two major justifications for Common Core standards: (1) raising academic standards across U.S. schools will grow the economy and make the nation globally competitive; (2) higher standards will improve students’ academic achievement. After parsing these reasons for the Common Core standards, I then turn to the evidence used by policymakers and practitioners and where research studies fit (or do not fit) into the policymaking process.

1. What evidence is there that common standards will increase a nation’s global competitiveness?

Answer: None. Zip. Nada.

See here and here.

2. What evidence is there that national standards will improve student achievement on domestic and international tests?

Answer: None. Zip. Nada.

See here, here and even here.

So how can a public policy that has heavy consequences for students, teachers, and public schools have an appalling lack of evidence?

The answer is in what top decision-makers consider as evidence when they determine policy. Or the answer is in the simple fact that policies get made for many reasons, only one of which may be evidence, including research studies. I take up each of these explanations.

First, what do policymakers consider to be evidence? Generally, school boards, state and federal officials, and practitioners–teachers and principals–have a broader definition of evidence than do researchers who rely upon the results of randomized control studies, rigorously conducted case studies, and carefully constructed interventions in schools (see Tseng-Social-Policy-Report-2012-1).

In an ongoing study of school boards’ decision-making, for example, Robert Asen and colleagues found that local policymakers drew from many sources for “evidence.” They relied on first-hand experiences,   systematically collected data on conditions, testimony of authoritative individuals and groups, specific examples that illustrated the policy issue being discussed, and, yes, they used empirical findings culled from researchers. What constitutes evidence to school board members was a broad array of experience-produced and research-produced knowledge, some carrying more weight than others in each policymaker’s mind. Few decision-makers, however, say that research findings guided their actions (coburnhonigsteinfinal-1).

Second, what drives policymaker decisions? Many reasons propel policy and evidence is only one of those reasons. Consider that financial and political pressures push policy without any reference to “what the research says.” When drug abuse or teenage pregnancies rise in a community, parents and politicians lobby school boards to initiate or revamp drug and sex education programs–regardless of what research studies say about the effectiveness of such programs. Or to cite another example, when a program becomes controversial such as “Man: A Course of Study” in the 1970s, studies of its effectiveness are disregarded as pressure groups got school boards to dump the program.

Or consider evaluating teachers on the basis of student test scores–one of the public reasons given for Chicago teachers striking this month. U.S. Secretary of Education Arne Duncan had, at best, contested   research findings a few years ago when he required performance evaluations to be included in state proposals for Race To The Top funds. Or even now. Political considerations mattered, not the amount or quality of evidence.

In an economic recession when state revenues shrink, districts cut staff and programs without checking research studies to determine which programs or staff were effective.

So local, state, and federal policymakers have a broader view of what constitutes evidence–practitioners even more so–than researchers. Research studies play a minor role, if at all, in making most significant policy decisions.

Which, of course, brings me around full circle to Common Core standards which is a train carrying few research studies that has left the station on its way into the nation’s classrooms. Believe me, that train is not carrying statues of Rodin’s “The Thinker.”

____________

*Thanks to Joel Westheimer for sending me the “Thinker and Doer” cartoon

59 Comments

Filed under school reform policies

Yet Again: Principals as Instructional Leaders

The constant chatter that principals should be innovative and tough-minded instructional leaders, on-top-of-everything CEOs, and smooth political tacticians reminds me of a photo* sent to me by a fellow blogger in Turkey.

I have written numerous times on the DNA of principaling and how  three roles–managing, instructing, and politicking–are essential to the daily work of principals. Researchers have observed elementary and secondary principals over the past century and documented time and again that most of their daily activities (at least half) are spent in administrative tasks. Managing a building, staff, children and youth, parents, central office officials, external agencies and companies doing business with the school consumes big chunks of time. And that is just to keep the place working and on course for teachers to teach and students to learn.

Principals reading the last paragraph would probably nod in agreement and could add activities that I omitted.

Of course, facts have little to do with ideology and the latest reform. For the past few decades, but especially since the federal law, No Child Left Behind, was passed, reform-minded academics and principal associations have advocated that the instructional leader is the primary role that principals  have to perform if schools are to do well academically–especially in urban districts where poor performance is pervasive. The key to  registering higher test scores, promoters of instructional leadership claim, is for the principal to lead teachers in designing the instructional program, coach teachers, do drop-in visits daily to classrooms, teach an occasional lesson, and evaluate how well (or poorly) teachers do over the 180 days of instruction. But as the photo of the rocket strapped to the Basset Hound says: “not everything new and shiny works.”

A recent report ( Shadow Study Miami-Dade Principals) of what 65 principals did each day during one week in 2008 in Miami-Dade county (FLA) shows that even under NCLB pressures for academic achievement and the widely accepted (and constantly spouted) ideology of instructional leadership, Miami-Dade principals spend most of their day in managerial tasks that influence the climate of the school but may or may not affect daily instruction. What’s more, those principals who spend the most time on organizing and managing the instructional program have test scores and teacher and parental satisfaction results  that are higher than those principals who spend time coaching teachers and popping into classroom lessons.

The researchers shadowed these elementary and secondary principals and categorized their activities minute-by-minute through self-reports, interviews, and daily logs kept by the principals.

In the academic language of the study:

The authors find that time spent on Organization Management activities is associated with positive school outcomes, such as student test score gains and positive teacher and parent assessments of the instructional climate, whereas Day-to-Day Instruction activities are marginally or not at all related to improvements in student performance and often have a negative relationship with teacher and parent assessments. This paper suggests that a single-minded focus on principals as instructional leaders operationalized through direct contact with teachers may be detrimental if it forsakes the important role of principals as organizational leaders (p. iv)

Two things jump out of this study for me. First, the results of shadowing principals in 2008 mirror patterns in principal work that researchers have found since the 1920s although the methodologies of time-and-motion studies have changed. Second, there is an association–a correlation, by no means a cause-effect relationship–between principals who spend more time managing the organization and climate of the school than those principals who spend time in direct contact with teachers in classrooms.

One study, of course, will not lower the volume or temper the rhetoric of principal-as-instructional-leader. But that study does bring into perspective that putting goggles and a rocket on a Basset Hound won’t make it fly any more than hyping the role of instructional leadership will make principals better at their jobs.

____________________

*Tony Gurr a blogger who is an educational consultant in Ankara, Turkey, sent me a range of graphics that included this photo. No source was provided.

25 Comments

Filed under Reforming schools, school leaders

“Why Do Good Policy Makers Use Bad Indicators?”*

Test scores are the coin of the educational realm in the U.S.. In No Child Left Behind, they are used to reward and punish districts, schools, and teachers for how well or poorly students score on state tests. In pursuit of federal dollars, The Race To The Top competition has shoved state after state into legislating that teacher evaluations include student test scores as part of judging teacher effectiveness.

Numbers glued to high stakes consequences, however, corrupt performance. Since the mid-1970s, social scientists have documented the untoward results of attaching high stakes to quantitative indicators not only for education but also across numerous institutions. They have pointed out that those who implement policies using specific quantitative measures will change their practices to insure better numbers.

The work of social scientist Donald T. Campbell and others about the perverse outcomes of incentives was available and known to many but went ignored. In Assessing the Impact of Planned Social Change, Campbell wrote:

“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor” (p. 49).

Campbell drew instances of distorted behavior when police officials used clearance rates in solving crimes, the Soviets set numerical goals for farming and industry, and when the U.S military used “body counts” in Vietnam as evidence of winning the war.

That was nearly forty years ago. In the past decade, medical researchers have found similar patterns when health insurers and Medicare have used quantitative indicators to measure physician performance. For example, Medicare requires—as a quality measure—that doctors administer antibiotics to a pneumonia patient within six hours of arriving at the hospital. As one physician said: “The trouble is that doctors often cannot diagnose pneumonia that quickly. You have to talk to and examine the patient and wait for blood tests, chest X-rays and so on.” So what happens is that “more and more antibiotics are being used in emergency rooms today, despite all-too-evident dangers like antibiotic-resistant bacteria and antibiotic-associated infections.” He and other doctors also know that surgeons have been known to pick reasonably healthy patients for heart bypass operations and ignore elderly ones who have 3-5 chronic ailments to insure that results look good.

More examples.

TV stations charge for advertising on the basis of how many viewers they have during  “sweep” months (November, February, May, and July). Nielsen company has boxes in two million homes (representative of the nation’s viewership) that register whether the TV is on and what families are watching during those months. They also have viewers fill out diaries. Nielsen assumes that what the station shows in those months represents programming for the entire year (see 2011-2012-Sweeps-Dates). Nope. What TV networks and cable companies do is that during those “sweeps” they program new shows, films, extravaganzas, and sports that will draw viewers so they can charge higher advertising rates. They game the system and corrupt the measure (see p. 80).

And just this week, ripped from the headlines of the daily paper, online vendors secretly ask purchasers  of their products to write reviews and rate it with five stars in exchange for a kickback of the price the customer paid. Another corrupted measure.

Of course, educational researchers also have documented the link between standardized test scores and narrowed instruction to prepare students for test items, instances of state policymakers fiddling with cut-off scores on tests, increased dropouts, and straight out cheating by a few administrators. (see Dan Koretz, Measuring Up).

What Donald Campbell had said in 1976 about “highly corruptible indicators” applies not only in education but also to many different institutions.

So why do good policy makers use bad indicators? The answer is that numbers are highly prized in the culture because they are easy to grasp and use in making decisions.The simpler the number–wins/losses, products sold, profits made, test scores– the easier to judge worth. When numbers have high stakes attached to them, they then become incentives (either as a carrot or a stick) to make the numbers look good. And that is where  indicators turn bad as sour milk whose expiration date has long passed.

The best policymakers, not merely good ones, know that multiple measures for a worthy goal reduce the possibility of reporting false performance.


*Steven Glazerman and Liz Potamites, False Performance Gains: A Critique of Successive Cohort Indicators,” Working Paper, Mathematica Policy Research, December 2011, p. 13.

8 Comments

Filed under Reforming schools

Bias toward Numbers in Judging Teaching

New fuel economy label in 2008 shows estimated...

Image via Wikipedia

In the U.S. people—yes, I include myself here—making decisions about important issues such as buying a home, picking a school for a five year-old or deciding on a college often give more weight to those features carrying numbers with them rather than qualitative features without numbers. Say, focusing on the square footage in the house vs. the feel of roominess. Or a teacher-student ratio in a kindergarten vs. sense of family that children and teacher communicate. From unemployment figures to batting averages and pass interceptions to calories, numbers carry far more weight with Americans than those variables that are harder to measure. Jonah Lehrer makes this point in one of his postings.

“Buying a car is a hard decision. There are just so many variables to think about. We’ve got to inspect the interior and analyze the engine, and research the reliability of the brand. And then, once we’ve amassed all these facts, we’ve got to compare different models.

How do we sift through this excess of information? When consumers are debating car alternatives, studies show that they tend to focus on variables they can quantify, such as horsepower and fuel economy…. We do this for predictable reasons. The amount of horsepower directly reflects the output of the engine, and the engine seems like something that should matter. (Nobody wants an underpowered car.) We also don’t want to spend all our money at the gas station, which is why we get obsessed with very slight differences in miles per gallon ratings.

Furthermore, these numerical attributes are easy to compare across cars: All we have to do is glance at the digits and see which model performs the best. And so a difficult choice becomes a simple math problem.

Unfortunately, this obsession with horsepower and fuel economy turns out to be a big mistake. The explanation is simple: The variables don’t matter nearly as much as we think. Just look at horsepower: When a team of economists analyzed the features that are closely related to lifetime car satisfaction, the power of the engine was near the bottom of the list. (Fuel economy was only slightly higher.) That’s because the typical driver rarely requires 300 horses or a turbocharged V-8. Although we like to imagine ourselves as Steve McQueen, accelerating into the curves, we actually spend most of our driving time stuck in traffic, idling at an intersection on the way to the supermarket. This is why, according to surveys of car owners, the factors that are most important turn out to be things like the soundness of the car frame, the comfort of the front seats and the aesthetics of the dashboard. These variables are harder to quantify, of course. But that doesn’t mean they don’t matter.”

Switch channels from buying cars to determining teacher effectiveness. Judging teacher effectiveness now means using multifactor algorithms with quantifiable variables including test scores and observers’ ratings while avoiding qualitative judgments about teacher practices that are hard to quantify. Examples: interviewing students after a teacher has praised their effort and persistence and seeing them glow. Or listening to students who remember teachers who applauded their self-control in difficult classroom situations. Or watching students in a class struggle with a problem that the teacher gave them that had no right answer to it.  Or see students who honored their favorite teachers by emulating  them as adults. Teacher blogger Stephen Lane  makes a similar point about the lack of metrics for things that really matter.

I end with Jonah Lehrer’s example that makes the same point vividly.

“When asked by David Remnick, in a 2000 New Yorker profile, how he felt about a cramped literary interpretation of one of his novels, Roth busted out a sports analogy. He imagined going to a baseball game with a little boy for the very first time. The kid doesn’t understand what’s happening on the field, and so his dad tells him to watch the scoreboard, to keep track of all the changing numbers. When the boy gets home someone asks him if he had fun at the game:

‘It was great!” he says. ‘The scoreboard changed thirty-two times and Daddy said last game it changed only fourteen times and the home team last time changed more times than the other team. It was really great! We had hot dogs and we stood up at one point to stretch and we went home.’ “

But, of course, the boy would have missed the point of baseball.

And all the complex algorithms used in current plans to judge teacher performance too often ignore the hard-to-quantify variables that students, teachers, and parents value and remember years later.

8 Comments

Filed under how teachers teach

Reforming the Science Curriculum Yet Again (Part 1)

“Creating new curriculum standards for science doesn’t reform teaching and learning any more than standing in a garage makes you a car or a truck.”*

The top research body in the U.S., the National Research Council, recently released its Framework for K-12 Science Education. An 18-member committee of top scientists and educational experts drawn from the National Academy of Sciences identified key concepts, scientific practices, and ideas that every student should learn by the time they graduate high school. It is intended as a guide for those who are now developing national Common  Core Standards in science (Standards in English Language Arts and Math are already out and 44 states have already adopted both).

As I read the report, two thoughts occurred to me. First, because of overlap in the players who created the Framework and those who are working on new science Standards the Framework is a preview of coming attractions for an intended science curriculum. I say “intended” because once states adopt the Common Core Standards in English and Math–with science next in line– the Standards, hullabaloo over a national curriculum notwithstanding, will not exactly mirror the science content teachers will teach once they close their classroom doors. Moreover, the science that students learn in those classrooms will vary from what the Standards contain and what teachers teach. Finally, what gets tested in national assessments of English, math, and science will differ from what teachers have taught and what students have learned. I elaborate this point of policy-to-practice in this post.

The second thought I had was how familiar the Framework was to me insofar as previous revisions of science curriculum over the past century. I will discuss cycles of science curricula in Part 2.

National curriculum frameworks as an instance of policy-to-practice**

The intended (or official) curriculum is what state and district officials set forth in curricular frameworks and courses of study. Were the science Framework to be adopted in part or wholly as another Core Curriculum Standard by states and districts in upcoming years, parents and school board officials would  expect teachers to teach it; further, they would assume students will learn it. These official curricula increasingly are aligned with state-approved textbooks that teachers are directed to use and state-mandated tests that teachers must administer.

But teachers, working alone in their rooms, choose what to teach and how to present it. Their choices derive from their knowledge of the subject they teach (elementary and secondary school teachers differ greatly in their knowledge of science), their experiences in teaching the content, their affection or dislike for topics, and their attitudes toward the students they face daily. In fact, researchers continually find that teachers in the same building will teach different versions of the same course. Thus, the intended curriculum and what teachers teach may overlap in the title of the course, certain key topics, and the same text, but can differ substantially in actual subject matter and daily lessons. And also students differ in what they learn.

The taught curriculum overlaps with but differs significantly from what students take away from class. Students pick up information and concepts from lessons. They also learn to answer teacher questions, recite, review material, locate sources, seek help, avoid teachers’ intrusiveness, and act attentive. Collateral learnings, in Dewey’s phrase, occur when children pick up ideas from class-mates, copy their teachers’ habits and tics, imitate their humor or sarcasm, or strive to be as autocratic or democratic as the adults. So, the learned curriculum differs from the intended and taught curricula.

And what students learn does not exactly mirror what is in the tested curriculum. Classroom, school, district, state, and national tests, often using multiple-choice and other short-answer  items, do, indeed, capture much–but hardly all–of the official and taught curricula. To the degree that teachers attend to such tests, portions of the intended and taught curricula merge. But what is tested is a limited part of what is intended by policymakers, taught by teachers, and learned by students. Since so many of these tests seek to sort high achieving students from their lower-achieving  peers, the information, ideas, and skills sought on these tests represent an even narrower band of knowledge.

The newly-published science Framework, then, much of which I expect to appear when the Core Standard in science eventually arrives, will be only the initial link in the policy-to-practice chain of intended-taught-learned-tested curricula that characterizes U.S. schooling. The additional links in that chain have to be accounted for because reforming science teaching and learning is far more complicated than standing in a garage and hoping to become a car or truck.

__________

*I made up the quote.

**Much of what follows on four different curricula (intended, taught, learned, tested) is drawn from The Hidden Variable. Citations and references are listed in the article.

3 Comments

Filed under school reform policies

Instead of Focusing on What Students Don’t Know, What Do They Know?

Another piece of evidence that students have not learned (or have forgotten) their science, math, and social studies made a recent splash in the media. This year it is civics. Two decades ago it was history. Diane Ravitch and Chester Finn published in 1988 What Do Our 17 year-olds Know in History and Literature. Their answer: not much.

This focus on how little each generation of students (and adults) know about academic subjects has become a popular ritual–dating back to 1943–that symbolizes–no surprise here–how inadequate U.S. schools are in transmitting to the next generation knowledge, skills, and values held to be essential in a democracy.

Perhaps a better question to ask is not what do students forget or haven’t learned in school but what do students know. Stanford University’s Sam Wineburg and University of Maryland’s Chauncey Monte-Santo, asked precisely that question when surveying students a few years ago. Here is the article that appeared in 2008.

Let’s begin with a brief exercise. Who are the most famous Americans in history, excluding presidents and first ladies? ….

A colleague and I recently put this question to 2,000 11th and 12th graders from all 50 states, curious to see whether they would name (as a great many educators had predicted) the likes of Paris Hilton, Britney Spears, Tupac Shakur, 50 Cent, Barry Bonds, Kanye West or any number of other hip-hop artists, celebrities or sports idols. To our surprise, the young people’s answers showed that whatever they were reading in their history classrooms, it wasn’t People magazine. Their top ten names were all bona fide historical figures.

To our even greater surprise, their answers pretty much matched those we gathered from 2,000 adults age 45 and over. From this modest exercise, we deduced that much of what we take for conventional wisdom about today’s youth might be conventional, but it is not wisdom. Maybe we’ve spent so much time ferreting out what kids don’t know that we’ve forgotten to ask what they do know.

Chauncey Monte-Sano of the University of Maryland and I designed our survey as an open-ended exercise. Rather than giving the students a list of names, we gave them a form with ten blank lines separated by a line in the middle. Part A came with these instructions: “Starting from Columbus to the present day, jot down the names of the most famous Americans in history.” There was only one ground rule—no presidents or first ladies. Part B prompted for “famous women in American history” (again, no first ladies). Thus the questionnaire was weighted toward women, though many kids erased women’s names from the first section before adding them to the second. But when we tallied our historical top ten, we counted the total number of times a name appeared, regardless of which section.

Of course a few kids clowned around, but most took the survey seriously. About an equal number of kids and adults listed Mom; from adolescent boys we learned that Jenna Jameson is the biggest star of the X-rated movie industry. But neither Mom nor Jenna was anywhere near the top. Only three people appeared on 40 percent of all questionnaires. All three were African-American.

For today’s teens, the most famous American in history is…the Rev. Dr. Martin Luther King Jr., appearing on 67 percent of all lists. Rosa Parks was close behind, at 60 percent, and third was Harriet Tubman, at 44 percent. Rounding out the top ten were Susan B. Anthony (34 percent), Benjamin Franklin (29 percent), Amelia Earhart (23 percent), Oprah Winfrey (22 percent), Marilyn Monroe (19 percent), Thomas Edison (18 percent) and Albert Einstein (16 percent). For the record, our sample matched within a few percentage points the demographics of the 2000 U.S. Census: about 70 percent of our respondents were white, 13 percent African-American, 9 percent Hispanic, 7 percent Asian-American, 1 percent Native American.

What about the gap between our supposedly unmoored youth and their historically rooted elders? There was not much of one. Eight of the top ten names were identical. (Instead of Monroe and Einstein, adults listed Betsy Ross and Henry Ford.) Among both kids and adults, neither region nor gender made much difference. Indeed, the only consistent difference was between races, and even there it was only between African-Americans and whites. Whites’ lists comprised four African-Americans and six whites; African-Americans listed nine African-American figures and one white. (The African-American students put down Susan B. Anthony, the adults Benjamin Franklin.)

Trying to take the national pulse by counting names is fraught with problems. To start, we know little about our respondents beyond a few characteristics (gender, race/ethnicity and region, plus year and place of birth for adults). When we tested our questionnaire on kids, we found that replacing “important” with “famous” made little difference, but we used “famous” with adults for the sake of consistency. Prompting for women’s names obviously inflated their total, though we are at a loss to say by how many.

But still: such qualifications cannot mist the clarity of consensus we found among Americans of different ages, regions and races. Eighty-two years after Carter G. Woodson founded Negro History Week, Martin Luther King Jr. has emerged as the most famous American in history. This may come as no surprise—after all, King is the only American whose birthday is celebrated by name as a national holiday. But who would have predicted that Rosa Parks would be the second most named figure? Or that Harriet Tubman would be third for students and ninth for adults? Or that 45 years after the Civil Rights Act was passed, the three most common names appearing on surveys in an all-white classroom in, say, Columbia Falls, Montana, would belong to African-Americans? For many of those students’ grandparents, this moment would have been unimaginable.

In the space of a few decades, African-Americans have moved from blurry figures on the margins of the national narrative to actors on its center stage. Surely multicultural education has played a role. When textbooks of the 1940s and ’50s employed the disingenuous clause “leaving aside the Negro and Indian population” to sketch the national portrait, few cried foul. Not today. Textbooks went from “scarcely mentioning” minorities and women, as a 1995 Smith College study concluded, to “containing a substantial multicultural (and feminist) component” by the mid-1980s. Scanning the shelves of a school library—or even the youth biography section at your local mega-chain bookstore—it’s hard to miss this change. Schools, of course, influence others besides students. Adults learn new history from their children’s homework.

Yet, to claim that the curriculum alone has caused these shifts would be simplistic. It wasn’t librarians, but members of Congress who voted for Rosa Parks’ body to lie in honor in the Capitol Rotunda after she died in 2005, the first woman in American history to be so honored. And it wasn’t teachers, but officials at the United States Postal Service who in 1978 made Harriet Tubman the first African-American woman to be featured on a U.S. postage stamp (and who honored her with a second stamp in 1995). Kids learn about Martin Luther King not only in school assemblies, but also when they buy a Slurpee at 7-Eleven and find free copies of the “I Have a Dream” speech by the cash register.

Harriet Tubman’s prominence on the list was something we wouldn’t have predicted, particularly among adults. By any measure, Tubman was an extraordinary person, ferrying at least 70 slaves out of Maryland and indirectly helping up to 50 more. Still, the Underground Railroad moved 70,000 to 100,000 people out of slavery, and in terms of sheer impact, lesser-known individuals played larger roles—the freeman David Ruggles and his Vigilance Committee of New York, for example, aided a thousand fugitives during the 1830s. The alleged fact that a $40,000 bounty (the equivalent of $2 million today) was offered for her capture is sheer myth, but it has been printed over and over again in state-approved books and school biographies….

It’s much easier to document the accomplishments of the only living person to appear in the top ten list. Oprah Winfrey is not just one of the richest self-made women in America. She is also a magazine publisher, life coach, philanthropist, kingmaker (think Dr. Phil), advocate for survivors of sexual abuse, school benefactor, even spiritual counselor. In a 2005 Beliefnet poll, more than a third of the respondents said she had “a more profound impact” on their spirituality than their pastor.

Some people might point to the inclusion of a TV talk-show host on our list as an indication of decline and imminent fall. I’d say that gauging Winfrey’s influence by calling her a TV host makes as much sense as sizing up Ben Franklin’s by calling him a printer. Consider the parallels: both rose from modest means to become the most identifiable Americans of their time; both became famous for serving up hearty doses of folk wisdom and common sense; both were avid readers and powerful proponents of literacy and both earned countless friends and admirers with their personal charisma.

Recently, the chairman of the National Endowment for the Humanities, Bruce Cole, worried that today’s students don’t learn the kind of history that will give them a common bond. To remedy this, he commissioned laminated posters of 40 famous works of art to hang in every American classroom, including Grant Wood’s 1931 painting “The Midnight Ride of Paul Revere.” “Call them myths if you want,” Cole said, “but unless we have them, we don’t have anything.”

He can relax. Our kids seem to be doing just fine without an emergency transfusion of laminated artwork. Myths inhabit the national consciousness the way gas molecules fill a vacuum. In a country as diverse as ours, we instinctively search for symbols—in children’s biographies, coloring contests, Disney movies—that allow us to rally around common themes and common stories, whether true, embellished or made out of whole cloth.

Perhaps our most famous national hand-wringer was Arthur Schlesinger Jr., whose 1988 Disuniting of America: Reflections on a Multicultural Society predicted our national downfall. “Left unchecked,” he wrote, the “new ethnic gospel” is a recipe for “fragmentation, resegregation and tribalization of American life.”

If, like Schlesinger (who died last year), Monte-Sano and I had focused on statements by the most extreme multiculturalists, we may have come to a similar conclusion. But that’s not what we did. Instead, we gave ordinary kids in ordinary classrooms a simple survey and compared their responses with those from the ordinary adults we found eating lunch in a Seattle pedestrian mall, shopping for crafts at a street fair in Philadelphia or waiting for a bus in Oklahoma City. What we discovered was that Americans of different ages, regions, genders and races congregated with remarkable consistency around the same small set of names. To us, this sounds more like unity than fragmentation.

The common figures who draw together Americans today look somewhat different from those of former eras. While there are still a few inventors, entrepreneurs and entertainers, the others who capture our imagination are those who acted to expand rights, alleviate misery, rectify injustice and promote freedom. That Americans young and old, in locations as distant as Columbia Falls, Montana, and Tallahassee, Florida, listed the same figures seems deeply symbolic of the story we tell ourselves about who we think we are—and perhaps who we, as Americans, aspire to become.

8 Comments

Filed under dilemmas of teaching