Category Archives: comparing medicine and education

Evidence Based Education Policy and Practice: A Conversation (Francis Schrag)

 

This fictitious exchange between two passionate educators over making educational policy and influencing classroom practice through careful scrutiny of evidence–such as has occurred in medicine and the natural sciences–as opposed to relying on professional judgment anchored in expertise gathered in schools brings out a fundamental difference among educators and the public that has marked public debate over the past three decades. The center of gravity in making educational policy in the U.S. has shifted from counting resources that go into schooling and relying on professional judgment to counting outcomes students derive from their years in schools and what the numbers say.

That shift can be dated from the Elementary and Secondary Education Act of 1965 but gained sufficient traction after the Nation at Risk report (1983) to dominate debate over innovation, policy, and practice. Although this is one of the longest guest posts I have published, I found it useful (and hope that viewers will as well) in making sense of a central conflict that exist today within and among school reformers, researchers, teachers, policymakers and parents.

Francis Schrag is professor emeritus in the philosophy of education at the University of Wisconsin, Madison. This article appeared in Teachers College Record, March 14, 2014.

A dialogue between a proponent and opponent of Evidence Based Education Policy. Each position is stated forcefully and each reader must decide who has the best of the argument.

Danielle, a professor of educational psychology and Leo, a school board member and former elementary school teacher and principal, visit a middle-school classroom in Portland Maine where students are deeply engaged in building robots out of Lego materials, robots that will be pitted against other robots in contests of strength and agility.  The project requires them to make use of concepts they’ve learned in math and physics.  Everything suggests that the students are deeply absorbed in what is surely a challenging activity, barely glancing around to see who has entered their classroom.

Leo:  Now this is exciting education. This is what we should be moving towards.  I wish all teachers could see this classroom in action.

Danielle:  Not so fast.  I’ll withhold judgment till I have some data.  Let’s see how their math and science scores at the end of the year compare with those of the conventional classroom we visited this morning.  Granted that one didn’t look too out of the ordinary, but the teacher was really working to get the kids to master the material.

Leo:  I don’t see why you need to wait.  Can’t you see the difference in level of engagement in the two classrooms?  Don’t you think the students will remember this experience long after they’ve forgotten the formula for angular momentum? Your hesitation reminds me of a satirical article a friend showed me; I think it came from a British medical journal.  As I recall the headline went: “Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomized controlled trials.”

Danielle:  Very cute, but let’s get serious.  Spontaneous reactions can be misleading; things aren’t always what they appear to be, as I’m sure you’ll agree.  I grant you that it looks as if the kids in this room are engaged, but we don’t know whether they’re engaged in the prescribed tasks and we don’t know what they’re actually learning, do we?  We’ll have a much better idea when we see the comparative scores on the test.  The problem with educators is that they get taken in with what looks like it works, they go with hunches, and what’s in fashion, but haven’t learned to consult data to see what actually does work.  If physicians hadn’t learned to consult data before prescribing, bloodletting would still be a popular treatment.

Suppose you and I agreed on the need for students to study math and physics.  And suppose that it turned out that the kids in the more conventional classroom learned a lot more math and physics, on average, as measured on tests, than the kids in the robotics classroom.  Would you feel a need to change your mind about what we’ve just seen?  And, if not, shouldn’t you?  Physicians are now on board with Evidence Based Medicine (EBM) in general, and randomized controlled trials (RCTs) in particular, as the best sources of evidence.  Why are teachers so allergic to the scientific method?  It’s the best approach we have to determine educational policy.

Leo:  Slow down Danielle.  You may recall that a sophisticated RCT convincingly showed the benefits of smaller class sizes in elementary schools in Tennessee, but these results were not replicated when California reduced its elementary school class size, because there was neither room in the schools for additional classrooms nor enough highly skilled teachers to staff them.  This example is used by Nancy Cartwright and Jeremy Hardie in their book on evidence-based policy to show that the effectiveness of a policy depends, not simply on the causal properties of the policy itself, but on what they call a “team” of support factors (2012, p. 25).  If any one of these factors were present in the setting where the trial was conducted but is lacking in the new setting, the beneficial results will not be produced.  This lack of generalizability, by the way, afflicts RCTs in medicine too.  For instance, the populations enrolled in teaching hospital RCTs are often different from those visiting their primary care physician.

Danielle:  I have to agree that educators often extrapolate from RCTs in a way that’s unwarranted, but aren’t you, in effect, calling for the collection of more and better evidence, rather than urging the abandonment of the scientific approach.  After all, the Cartwright and Hardie book wasn’t written to urge policy makers to throw out the scientific approach and go back to so-called expert or professional judgment, which may be no more than prejudice or illicit extrapolation based on anecdotal evidence.

Leo:  You seem to be willing to trust the data more than the judgment of seasoned professionals.  Don’t you think the many hours of observing and teaching in actual classrooms counts for anything?

Danielle: If your district has to decide which program to run, the robotics or the traditional, do you really want to base your decision on the judgment of individual teachers or principals, to say nothing of parents and interested citizens?  In medicine and other fields, meta-analyses have repeatedly shown that individual clinical judgment is more prone to error than decisions based on statistical evidence (Howick, 2011, Chap. 11). And, as I already mentioned, many of the accepted therapies of earlier periods, from bloodletting to hormone replacement therapy, turned out to be worse for the patients than doing nothing at all.

Now why should education be different?  How many teachers have “known” that the so-called whole-word method was the best approach to teaching reading, and years later found out from well-designed studies that this is simply untrue?  How many have “known” that children learn more in smaller classes?  No, even if RCTs aren’t always the way to go, I don’t think we can leave these things to individual educator judgment; it’s too fallible.

And you may not need to run a new study on the question at issue.  There may already be relevant, rigorous studies out there, testing more exploratory classrooms against more traditional ones in the science and math area for middle-schoolers.  I recommend you look at the federal government What Works website, which keeps track of trial results you can rely on.

Leo:  I’ve looked at many of these studies, and I have two problems with them.  They typically use test score gains as their indicator of durable educational value, but these can be very misleading.  Incidentally, there’s a parallel criticism of the use of “surrogate end points” like blood levels in medical trials.  Moreover, according to Goodhart’s Law—he was a British economist—once a measure becomes a target, it ceases to be a good indicator.  This is precisely what happens in education: the more intensely we focus on raising a test score by means of increasing test preparation to say nothing of cheating—everything from making sure the weakest, students don’t take the test to outright changing students’ answers—the less it tells us about what kids can do or will do outside the test situation.

Danielle:  Of course we need to be careful about an exclusive reliance on test scores.  But you can’t indict an entire approach because it has been misused on occasion.

Leo: I said there was a second problem, as well.  You recall that what impressed us about the robotics classroom was the level of involvement of the kids.  When you go into a traditional classroom, the kids will always look at the door to see who’s coming in.  That’s because they’re bored and looking for a bit of distraction.  Now ask yourself, what does that involvement betoken. It means that they’re learning that science is more than memorizing a bunch of facts, that math is more than solving problems that have no meaning or salience in the real world, that using knowledge and engaging in hard thinking in support of a goal you’ve invested in is one of life’s great satisfactions.  Most kids hate math and the American public is one of the most scientifically illiterate in the developed world.  Why is that?  Perhaps it’s because kids have rarely used the knowledge they are acquiring to do anything besides solve problems set by the teacher or textbook.

I’m sure you recall from your studies in philosophy of education the way John Dewey called our attention in Experience and Education to what he called, the greatest pedagogical fallacy, “the notion that a person learns only the particular thing he is studying at the time” (Dewey, 1938, p. 48).  Dewey went on to say that what he called “collateral learning,” the formation of “enduring attitudes” was often much more important than the particular lesson, and he cited the desire to go on learning as the most important attitude of all.  Now when I look at that robotics classroom, I can see that those students are not just learning a particular lesson, they’re experiencing the excitement that can lead to a lifetime of interest in science or engineering even if they don’t select a STEM field to specialize in.

Danielle:  I understand what Dewey is saying about “collateral learning.”  In medicine as you know, side effects are never ignored, and I don’t deny that we in education are well behind our medical colleagues in that respect.  Still, I’m not sure I agree with you and Dewey about what’s most important, but suppose I do.  Why are you so sure that the kids’ obvious involvement in the robotics activity will generate the continuing motivation to keep on learning?  Isn’t it possible that a stronger mastery of subject matter will have the very impact you seek?  How can we tell?  We’d need to first find a way to measure that “collateral learning,” then preferably conduct a randomized, controlled trial, to determine which of us is right.

Leo:  I just don’t see how you can measure something like the desire to go on learning, yet, and here I agree with Dewey, it may be the most important educational outcome of all.

Danielle:  This is a measurement challenge to be sure, but not an insurmountable one.  Here’s one idea: let’s track student choices subsequent to particular experiences.  For example, in a clinical trial comparing our robotics class with a conventional middle school math and science curriculum, we could track student choices of math and science courses in high school.  Examination of their high school transcripts could supply needed data.  Or we could ask whether students taking the robotics class in middle school were more likely (than peers not selected for the program) to take math courses in high school, to major in math or science in college, etc.  Randomized, longitudinal designs are the most valid, but I admit they are costly and take time.

Leo: I’d rather all that money went into the kids and classrooms.

Danielle:  I’d agree with you if we knew how to spend it to improve education.  But we don’t, and if you’re representative of people involved in making policy at the school district level, to say nothing of teachers brainwashed in the Deweyian approach by teacher educators, we never will.

Leo:  That’s a low blow, Danielle, but I haven’t even articulated my most fundamental disagreement with your whole approach, your obsession with measurement and quantification, at the expense of children and education.

Danielle:  I’m not sure I want to hear this, but I did promise to hear you out.  Go ahead.

Leo:  We’ve had about a dozen years since the passage of the No Child Left Behind Act to see what an obsessive focus on test scores looks like and it’s not pretty.  More and more time is taken up with test-prep, especially strategies for selecting right answers to multiple-choice questions.  Not a few teachers and principals succumb to the temptation to cheat, as I’m sure you’ve read.  Teachers are getting more demoralized each year, and the most creative novice teachers are finding jobs in private schools or simply not entering the profession.  Meanwhile administrators try to game the system and spin the results.  But even they have lost power to the statisticians and other quantitatively oriented scholars, who are the only ones who can understand and interpret the test results.  Have you seen the articles in measurement journals, the arcane vocabulary and esoteric formulas on nearly every page?

And do I have to add that greedy entrepreneurs with a constant eye on their bottom lines persuade the public schools to outsource more and more of their functions, including teaching itself.  This weakens our democracy and our sense of community.  And even after all those enormous social costs, the results on the National Assessment of Educational Progress are basically flat and the gap between black and white academic achievement—the impetus for passing NCLB in the first place—is as great as it ever was.

Danielle:  I agree that it’s a dismal spectacle.  You talk as if educators had been adhering to Evidence Based Policy for the last dozen years, but I’m here to tell you they haven’t and that’s the main reason, I’d contend, that we’re in the hole that we are.  If educators were less resistant to the scientific approach, we’d be in better shape today.  Physicians have learned to deal with quantitative data, why can’t teachers, or are you telling me they’re not smart enough?  Anyhow, I hope you feel better now that you’ve unloaded that tirade of criticisms.

Leo:  Actually, I’m not through, because I don’t think we’ve gotten to the heart of the matter yet.

Danielle:  I’m all ears.

Leo:  No need to be sarcastic, Danielle.  Does the name Michel Foucault mean anything to you?  He was a French historian and philosopher.

Danielle:  Sure, I’ve heard of him.  A few of my colleagues in the school of education, though not in my department, are very enthusiastic about his work.  I tried reading him, but I found it tough going.  Looked like a lot of speculation with little data to back it up.  How is his work relevant?

Leo:   In Discipline and Punish, Foucault described the way knowledge and power are intertwined, especially in the human sciences, and he used the history of the school examination as a way of illustrating his thesis (1975/1995, pp. 184-194).  Examinations provide a way of discovering “facts” about individual students, and a way of placing every student on the continuum of test-takers.  At the same time, the examination provides the examiners, scorers and those who make use of the scores ways to exercise power over kids’ futures.  Think of the Scholastic Assessment Tests (SATs) for example.  Every kid’s score can be represented by a number and kids can be ranked from those scoring a low of 600 to those with perfect scores of 2400.  Your score is a big determinant of what colleges will even consider you for admission.  But that’s not all: Foucault argued that these attempts to quantify human attributes create new categories of young people and thereby determine how they view themselves.  If you get a perfect SAT score, or earn “straight As” on your report card, that becomes a big part of the way others see you and how you see yourself.  And likewise for the mediocre scorers, the “C” students, or the low scorers who not only have many futures closed to them, but may see themselves as “losers,” “failures,” “screw-ups.”  A minority may, of course resist and rebel against their placement on the scale—consider themselves to be “cool”, unlike the “nerds” who study, but that won’t change their position on the continuum or their opportunities.  Indeed, it may limit them further as they come to be labeled “misfits” “ teens at-risk,” “gang-bangers” and the like. But, and here’s my main point, this entire system is only possible due to our willingness to represent the capabilities and limitations of children and young people by numerical quantities.  It’s nothing but scientism, the delusive attempt to force the qualitative, quirky, amazingly variegated human world into a sterile quantitative straight-jacket.  You recall the statement that has been attributed to Einstein, don’t you, “Not everything that can be counted counts, and not everything that counts can be counted.” I just don’t understand your refusal to grasp that basic point; it drives me mad.

Danielle:  Calm down, Leo.  I don’t disagree that reducing individuals to numbers can be a problem; every technology has a dark side, I’ll grant you that, but think it through.  Do you really want to go back to a time when college admissions folks used “qualitative” judgments to determine admissions?  When interviewers could tell from meeting a candidate or receiving a letter of recommendation if he were a member of “our crowd,” would know how to conduct himself at a football game, cocktail party, or chapel service, spoke without an accent, wasn’t a grubby Jew or worse, a “primitive” black man or foreign-born anarchist or communist.  You noticed I used the masculine pronoun:  Women, remember, were known to be incapable of serious intellectual work, no data were needed, the evidence was right there in plain sight.  Your Foucault is not much of a historian, I think.

Leo:  We have some pretty basic disagreements here.  I know we each believe we’re right.  Is there any way to settle the disagreement?

Danielle:  I can imagine a comprehensive, longitudinal experiment in a variety of communities, some of which would carry out EBEP and control communities that would eschew all use of quantification.  After a long enough time, maybe twenty years, we’d take a look at which communities were advancing, which were regressing.  Of course, this is just an idea; no one would pay to actually have it done.

Leo:  But even if we conducted such an experiment, how would we know which approach was successful?

Danielle:  We shouldn’t depend on a single measure, of course.  I suggest we use a variety of measures, high school graduation rate, college attendance, scores on the National Assessment of Educational Progress, SATs, state achievement tests, annual income in mid-career, and so on.  And, of course, we could analyze the scores by subgroups within communities to see just what was going on.

Leo:  Danielle, I can’t believe it.  You haven’t listened to a word I’ve said.

Danielle:  What do you mean?

Leo:   If my favored policy is to eschew quantitative evidence altogether, wouldn’t I be inconsistent if I permitted the experiment to be decided by quantitative evidence, such as NAEP scores or worse, annual incomes?  Don’t you recall that I reject your fundamental assumption—that durable, significant consequences of educational experiences can be represented as quantities?

Danielle:  Now I’m the one that’s about to scream.  Perhaps you could assess a single student’s progress by looking at her portfolio at the beginning and end of the school year.  How, in the absence of quantification, though, can you evaluate an educational policy that affects many thousands of students?  Even if you had a portfolio for each student, you’d still need some way to aggregate them in order to be in a position to make a judgment about the policy or program that generated those portfolios.  You gave me that Einstein quote to clinch your argument.  Well, let me rebut that with a quotation by another famous and original thinker, the Marquis de Condorcet, an eighteenth century French philosopher and social theorist.  Here’s what he said:  “if this evidence cannot be weighted and measured, and if these effects cannot be subjected to precise measurement, then we cannot know exactly how much good or evil they contain” (Condorcet, 2012, p.138).  The point remains true, whether in education or medicine.  If you can’t accept it, I regret to say, we’ve reached the end of the conversation.

References

Cartwright, N & Hardie, J. (2012). Evidence-based policy:  A practical guide to doing it better.  Oxford and New York: Oxford University Press.

Condorcet, M. (2012). The sketch. In S. Lukes, and N. Urbinati (Eds.), Political Writings (pp. 1-147). Cambridge: Cambridge University Press.

Dewey, J. (1938/1973). Experience and education.  New York: Collier Macmillan Publishers.

Foucault, M. (1995).  Discipline and punish: The birth of the prison. (A. Sheridan, Trans.) New York: Vintage Books. (Original work published in 1975)

Howick, J. (2011). The Philosophy of evidence-based medicine. Oxford: Blackwell Publishing.

21 Comments

Filed under comparing medicine and education, school reform policies

What’s The Evidence on School Devices and Software Improving Student Learning?

The historical record is rich in evidence that research findings have played a subordinate role in making educational policy. Often, policy choices were (and are) political decisions. There was no research, for example, that found establishing tax-supported public schools in the early 19th century was better than educating youth through private academies. No studies persuaded late-19th century educators to import the kindergarten into public schools. Ditto for bringing computers into schools a century later.

So it is hardly surprising, then, that many others, including myself, have been skeptical of the popular idea that evidence-based policymaking and evidence-based instruction can drive teaching practice. Those doubts have grown larger when one notes what has occurred in clinical medicine with its frequent U-turns in evidence-based “best practices.”

Consider, for example, how new studies have often reversed prior “evidence-based” medical procedures.

*Hormone therapy for post-menopausal women to reduce heart attacks was found to be more harmful than no intervention at all.

*Getting a PSA test to determine whether the prostate gland showed signs of cancer for men over the age of 50 was “best practice” until 2012 when advisory panels of doctors recommended that no one under 55 should be tested and those older  might be tested if they had family histories of prostate cancer.

And then there are new studies that recommend women to have annual mammograms, not at age  50 as recommended for decades, but at age 40. Or research syntheses (sometimes called “meta-analyses”) that showed anti-depressant pills worked no better than placebos.

These large studies done with randomized clinical trials–the current gold standard for producing evidence-based medical practice–have, over time, produced reversals in practice. Such turnarounds, when popularized in the press (although media attention does not mean that practitioners actually change what they do with patients) often diminished faith in medical research leaving most of us–and I include myself–stuck as to which healthy practices we should continue and which we should drop.

Should I, for example, eat butter or margarine to prevent a heart attack? In the 1980s, the answer was: Don’t eat butter, cheese, beef, and similar high-saturated fat products. Yet a recent meta-analysis of those and subsequent studies reached an opposite conclusion.

Figuring out what to do is hard because I, as a researcher, teacher, and person who wants to maintain good health has to sort out what studies say and  how those studies were done from what the media report, and then how all of that applies to me. Should I take a PSA test? Should I switch from margarine to butter?

If research into clinical medicine produces doubt about evidence-based practice, consider the difficulties of educational research–already playing a secondary role in making policy and practice decisions–when findings from long-term studies of innovation conflict with current practices. Look, for example, at computer use to transform teaching and improve student achievement.

Politically smart state and local policymakers believe that buying new tablets loaded with new software, deploying them to K-12 classrooms, and watching how the devices engage both teachers and students is a “best practice.” The theory is that student engagement through the device and software will dramatically alter classroom instruction and lead to improved  achievement. The problem, of course–sure, you already guessed where I was going with this example–is that evidence of this electronic innovation transforming teaching and achievement growth is not only sparse but also unpersuasive even when some studies show a small “effect size.”

Turn now to the work of John Hattie, a Professor at the University of Auckland (NZ), who has synthesized the research on different factors that influence student achievement and measured their impact on learning. For example, over the last two decades, Hattie has examined over 180,000 studies accumulating 200, 000 “effect sizes”  measuring the influence of teaching practices on student learning. All of these studies represent over 50 million students.

He established which factors influenced student learning–the “effect size–by ranking each from 0.1 (hardly any influence) to 1.0 or a full standard deviation–almost a year’s growth in student learning. He found that the “typical” effect size of an innovation was 0.4.

To compare different classroom approaches shaped student learning, Hattie used the “typical” effect size (0.4) to mean that a practice reached the threshold of influence on student learning (p. 5). From his meta-analyses, he then found that class size had a .20 effect (slide 15) while direct instruction had a .59 effect (slide 21). Again and again, he found that teacher feedback had an effect size of .72 (slide 32). Moreover, teacher-directed strategies of increasing student verbalization (.67) and teaching meta-cognition strategies (.67) had substantial effects (slide 32).

What about student use of computers (p. 7)? Hattie included many “effect sizes” of computer use from distance education (.09), multimedia methods (.15), programmed instruction (.24), and computer-assisted instruction (.37). Except for “hypermedia instruction” (.41), all fell below the “typical ” effect size (.40) of innovations improving student learning (slides 14-18). Across all studies of computers, then, Hattie found an overall effect size of .31 (p. 4).

According to Hattie’s meta-analyses, then, introducing computers to students will  fall well below other instructional strategies that teachers can and do use. Will Hattie’s findings convince educational policymakers to focus more on teaching? Not as long as political choices trump research findings.

Even if politics were removed from the decision-making equation, there would still remain the major limitation of  most educational and medical research. Few studies  answer the question: under what conditions and with which students and patients does a treatment work? That question seldom appears in randomized clinical trials. And that is regrettable.

 

 

26 Comments

Filed under comparing medicine and education, how teachers teach, technology use

Failure and Error among Doctors and Teachers

I had a conversation with a friend* last week about the two posts detailing my failures as a teacher with three students I have had over the years. He has practiced Family Medicine for over a half-century in Pittsburgh and for years helped resident physicians in doing medical research and now works with hospital residents in improving communication with patients.  He pointed out to me how similar teachers experiencing failures with students is to physicians erring in diagnoses or treatments (or both) of their patients.

I was surprised at his making the comparison and then began to think about the many books I have read about medicine and the art and science of clinical practice. In my library at home, I had two with well-thumbed pages authored by doctors who, in the  first dozen pages, detailed mistakes either they had made with patients or errors committed by other physicians on them or their families.

In one, Jerome Groopman, an oncologist, described what occurred with his 9-month old child after a series of doctor misdiagnoses that almost caused his son’s death. A surgeon, who was a friend of a friend, was called in at the last moment to fix an intestinal blockage and saved his son.

In the other book, surgeon Atul Gawande described how he almost lost an Emergency Room patient who had crashed her car when he fumbled a tracheotomy only for patient to be saved by another surgeon who successfully got the breathing tube inserted. Gawande also has a chapter on doctors’ errors. His point, documented by a paper in the New England Journal of Medicine (1991) and subsequent reports  is that nearly all physicians err.

If nearly all doctors make mistakes, do they talk about them? Privately  with people they trust, yes. In public, that is, with other doctors in academic hospitals, the answer is also yes. There is an institutional mechanism where hospital doctors meet weekly called Morbidity and Mortality Conferences (M & M for short) where, in Gawande’s words, doctors “gather behind closed doors to review the mistakes, untoward events, and deaths that occurred on their watch, determine responsibility, and figure out what to do differently (p. 58).” He describes an M & M (pp.58-64) at his hospital and concludes: “The M & M sees avoiding error as largely a matter of will–staying sufficiently informed and alert to anticipate the myriad ways that things can go wrong and then trying to head off each potential problem before it happens” (p. 62). Protected by law, physicians air their mistakes without fear of malpractice suits.

Nothing like that for teachers in U.S. schools. Sure, privately, teachers tell one another how they goofed with a student, misfired on a lesson, realized that they had provided the wrong information, or fumbled the teaching of a concept in a class. Of course,  there are scattered, well-crafted professional learning communities in elementary and secondary schools where teachers feel it is OK to admit they make mistakes and not fear retaliation. They can admit error and learn to do better the next time. In the vast majority of schools, however, no analogous M & M exists (at least as far as I know).

Of course, there are substantial differences between doctors and teachers. For physicians, the consequences of their mistakes might be lethal or life-threatening. Not so, in most instances, for teachers. But also consider other differences:

*Doctors see patients one-on-one; teachers teach groups of 20 to 35 students four to five hours a day.

*Most U.S. doctors get paid on a fee-for-service basis; nearly all full-time public school teachers are salaried.

*Evidenced-based practice of medicine in diagnosing and caring for patients is more fully developed and used by doctors than the science of teaching accessed by teachers.

While these differences are substantial in challenging comparisons, there are basic commonalities that bind teachers to physicians. First, both are helping professions that seek human improvement. Second, like practitioners in other sciences and crafts, both make mistakes. These commonalities make comparisons credible even with so many differences between the occupations.

Helping professions.

From teachers to psychotherapists to doctors to social workers to nurses, these professionals use their expertise to transform minds, develop skills, deepen insights, cope with feelings and mend bodily ills. In doing so, these helping professions share similar predicaments.

*Expertise is never enough. For surgeons, cutting out a tumor from the colon will not rid the body of cancer; successive treatments of chemotherapy are necessary and even then, the cancer may return.

Some high school teachers of science with advanced degrees in biology, chemistry, and physics believe that lessons should be inquiry driven and filled with hands-on experiences while other colleagues, also with advanced degrees, differ. They argue that naïve and uninformed students must absorb the basic principles of biology, chemistry, and physics through rigorous study before they do any “real world” work in class.

In one case, there is insufficient know-how to rid the body of different cancers and, in the other instance, highly knowledgeable teachers split over how students can best learn science. As important as expertise is to professionals dedicated to helping people, it falls short—and here is another shared predicament–not only for the reasons stated above but also because professionals seeking human improvement need their clients, patients, and students to engage in the actual work of learning and becoming knowledgeable, healthier people.

*Helping professionals are dependent upon their clients’ cooperation. Physician autonomy, anchored in expertise and clinical experience, to make decisions unencumbered by internal or external bureaucracies is both treasured and defended by the medical profession. Yet physicians depend upon patients for successful diagnoses and treatments. If expertise is never enough in the helping professions, patients not only constrain physician autonomy but also influence their effectiveness.

While doctors can affect a patient’s motivation, if that patient is emotionally depressed, is resistant to recommended treatments, or uncommitted to getting healthy by ignoring prescribed medications the physician is stuck. Autonomy to make decisions for the welfare of the patient and ultimate health is irrelevant when patients cannot or do not enter into the process of healing.

For K-12 teachers who face captive audiences among whom are some students unwilling to participate in lessons or who defy the teacher’s authority or are uncommitted to learning what the teacher is teaching, then teachers have to figure out what to do in the face of students’ passivity or active resistance.

Failure and error occur in both medical and teaching practices.
Both doctors and teachers, from time to time, err in what they do with patients and students. Patients can bring malpractice suits to get damages for errors. But that occurs sometimes years after the mistake. What hospital-based physicians do have, however, is an institutionalized way of learning (Mortality and Morbidity conferences) from their mistakes so that they do not occur again. So far, among teachers there are no public ways of admitting mistakes and learning from them (privately, amid trusted colleagues, such admissions occur). For teachers, admitting error publicly can lead directly to job loss).

So while doctors, nurses, and other medical staff have M & M conferences to correct mistakes, most teachers lack such collaborative and public ways of correcting mistakes (one exception might be in special education where various staff come together weekly or monthly to go over individual students’ progress).

Books and articles have been written often about how learning from failure can lead to success. Admitting error without fear of punishment is the essential condition for such learning to occur. There is no sin in being wrong or making mistakes, but in the practice of schooling children and youth today, one would never know that.

_______________________

* Dr. Joel Merenstein and I have been close friends since boyhood in Pittsburgh (PA)

32 Comments

Filed under comparing medicine and education

Best Practices and Bad Habits (Part 2)

Transfer of learning appears to be a simple concept. What you learn in the family or learn in school  can be applied in different situations outside of the family and the classroom. Learning Spanish, for example, helps later in learning Italian. Learning to get along with an older brother or sister helps in learning how to get along with others later in life. Learning math in middle school helps one in high school physics. It doesn’t always work that way, however.

images-2

In Part 1,  I described how I taught a two-week unit on specific critical thinking skills useful to understand history and use in daily life in the early 1960s. My theory was that teaching these thinking skills directly one after the other at the very beginning of the semester would lead to students applying them when I began teaching units on the American Revolution, Immigration, Sectionalism and the Civil War, and the Industrial Revolution.

The response of students to the stories and subsequent discussions and debates almost made me swoon. I was energized by students’ enthusiasm as we went from one specific skill to another using contemporary stories drawn from newspapers,  student lives, and Glenville high school. The two week unit was from all indications a success with student engagement high and even scores on the unit test were higher than I had expected.

Then, when I began my U.S. history units on the American Revolution through World War I, the skills I believed that I had taught my students weeks earlier were missing in action. Root canal work was easier than getting students to distinguish between a biased source and one less so or explain why certain statements were opinions, not facts. I was puzzled.  What had happened?

Years later, I discovered from reading psychologists a great deal about the ins-and-outs of transfer of learning (see, for example, here). Teaching specific critical thinking skills and expecting students to apply what they learned to different situations depended upon many conditions that were, I learned later, missing in my lessons. Even the concept of teaching these skills isolated from the historical content–as I did–undermines the very goal I wanted to achieve (see CritThink).

Nonetheless, puzzled as I was by the absence of students applying what they had learned in the later history units I taught, for the next few years I continued to teach that two week unit on critical thinking at the beginning of the semester, marching through the lessons one skill after another. I repeated again and again this unit because the students were engaged, loved to apply what they learned to their daily lives, and I felt good after each of the five periods I taught. An uncommon experience for a veteran teacher.

Even had a colleague I trusted grabbed me by the shoulders then and told me how I was way off in thinking that my students would transfer the skills they learned in the two-week unit to subsequent history units, I would not have believed that colleague.  I would have continued with what I considered a “best practice” that, in reality, had become a “bad” habit.

mooc+cartoon

Like Dr. Danielle Ofra, I would have given reasons to myself why what I was doing helped students. As I look back, I kept doing the same unit year after year and ignored the signs–the mysterious tug I felt every semester seeing repeatedly that students failed to apply the skills in subsequent history units that they had supposedly learned weeks earlier. I persisted even in light of the evidence of little transfer of learning.

Such “bad” habits, of course are common. From over-eating to smoking to excessive Internet surfing to watching far too much television, “bad” habits–destructive to one’s health and well-being–persist among substantial numbers of youth and adults.

images-1

Such habits are like ruts in road that get deeper and deeper through repetition of the behavior. It is hard to get out of the well worn groove. Yet people do break “bad” habits by replacing them with “good” habits that begin a new groove, and get practiced over and over again. It can be done and does occur.

As for me, my “bad” habit of ignoring evidence of my students not applying what they learned in that two-week thinking skills unit, eventually changed. The baffling lack of application got me to read more and talk to colleagues about what occurred in my teaching. I stumbled into new knowledge about transfer of learning. I made many attempts, some failed badly, to build new units in history where these thinking skills were embedded in the historical content. Eventually, I got into a new groove and created different units and taught them (e.g., Colonization, American Revolution, Causes of the Civil War, The Industrial Revolution, The Kennedy Assassination). See here.

But understanding transfer of learning was a hard road to travel in getting out of that rut I had made for myself as a history teacher many years ago.

11 Comments

Filed under comparing medicine and education, how teachers teach

Best Practices and Bad Habits in Practicing Medicine and Teaching Students

Listen to Danielle Ofri lamenting a fact she discovered about her work in New York City’s Bellevue hospital as a physician: “we often ignore good advice when it conflicts with what we’ve always done.”

Ofri was referring to the latest clinical guideline issued by the Society of General Internal Medicine that recommended against annual physical exams for healthy adults. The scientific evidence shows “the harm of annual visits — overdiagnosis, overtreatment, excess costs — can outweigh the benefits.”  These guidelines become  “best practices” for physicians to follow with patients; they are based upon analysis of many clinical studies.

Keep in mind that the body of evidence producing clinical guidelines for how doctors should practice is based on cumulative research and meta-analyses often involving tens of thousands of patients in control and experimental studies. “Evidence-based medicine”–even with all of the criticism of reversals in the advice doctors receive-is a reality at the fingertips of every doctor tapping keys and watching the computer screen as they take a patient’s history and conduct an annual exam.

Yet Ofri continues to have her patients return every year for an annual exam. How come?

She says: After the research was initially published last year, I grappled with the evidence, or lack thereof, reaching a conclusion that I mainly still supported the annual visit, if only because it establishes a solid doctor-patient relationship. But seeing these new, strongly worded recommendations, I may have to re-evaluate. At the very least, I should take a moment to think before I reflexively recommend the annual visit. But I know that I might still end up doing the same thing, despite the evidence.

She concludes: Humans are creatures of habit. Our default is to continue on the path we’ve always trod.

For some physicians, habit trumps evidence or what was once a “good” habit–annual exams for all of her patients–becomes a “bad” habit. True as well for K-12 teachers.

No such clinical research base, however, exists for recommending “best practices” in teaching reading, math, science, or history. Sure there are single studies, even groups of studies that point in a direction that teachers might consider in teaching long division or teaching six year-olds how to parse vowels and consonants. But for most teachers, “best practices” is a meld of what researchers say practitioners ought to do, what “experts” say should be done in classrooms, lessons learned from personal experiences in teaching, deeply-ingrained beliefs–call it ideology–about how best to teach and how students learn, and, yes, you guessed it: habit.

All of these ways of defining “best practice” for teachers came into play when I taught history to high school students many years ago. Let me explain.

In the fifth year of my teaching at Cleveland’s Glenville high school–it was the early 1960s–I had already introduced materials to my classes on what was then called “Negro history” (see here and here). I then began experimenting with the direct teaching of critical thinking skills. I believed that such skills were crucial in negotiating one’s way through life and understanding history. I wanted my students to acquire and use these skills every day. So I began teaching my U.S. history courses with a two-week unit on thinking skills. My theory was that the students learning these skills at the very beginning of the semester would then apply them when I began teaching units on the American Revolution, Immigration, Sectionalism and the Civil War, and the Industrial Revolution.

In the two-week unit, I selected skills I believed were important for understanding the past such as: figuring out the difference between a fact and opinion, making a hunch about what happened and sorting evidence that would support or contradict the hunch, judging how reliable a source of information is, distinguishing between relevant and irrelevant information in reaching a conclusion.

For each of these, I would go over the specific skill with the class and they and I would give examples from our daily lives, school events, and family happenings. Then,  I chose a contemporary event–a criminal case in the local newspaper, a national scandal that was on television, and occurrences in the school–and wrote out a one-page story that would require each student to apply the particular skill we were discussing such as making an informed guess, collecting evidence to support their hunch, and reaching a judgment. I also gave the class additional sources from which they could (or could not because of biases) select information to support their conclusion.

For the two weeks, each period–I was teaching five classes a day at the time–was filled with engaged students participating in flurries of discussion, debates over evidence, student questioning of each others’ conclusions, and similar excitement. I was elated by the apparent success of my critical thinking skills unit.

After the two weeks of direct instruction of skills, I plunged into the Coming of the American Revolution and subsequent history material. From time to time, over the course of the semester, I would ask questions that I felt would prompt use of those thinking skills we had worked on earlier in the year. Blank stares from most students with occasional “Oh yeah” from others. I designed homework that explicitly called for use of these thinking skills; few students applied what they had presumably learned.    I was thoroughly puzzled.

Which brings me to the concept of transfer. Why did students taught discrete thinking skills directly with a high degree of engagement and apparent learning for two weeks have a difficult time transferring those very same skills to history lessons later in the semester? I take up this issue and my “bad” habit in the next post. 

6 Comments

Filed under comparing medicine and education, how teachers teach

Links and Lessons for K-12 Schools and Hospitals (Jeff Bowen)

Dr. Jeffrey Bowen has served as superintendent of the Yorkshire Pioneer Central School District in western New York, research director for the New York State School Boards Association, and supervisor of on-the-job training in an Air Force hospital.  He is a founding member and vice president of the Healthy Community Alliance in Gowanda, New York.  

Since 2000, hospital bills have increased at an annual average rate of 10 percent.  Surveys show most Americans think that health care costs seriously threaten the economy.  Issues of access and quality complicate the picture.  Health care is gobbling up 20 percent of the Gross Domestic Product (GDP).

Meanwhile, funding cutbacks are distressing K-12 schools.  More than half the states are spending less per student this year compared to last.  As thousands of teachers are laid off, school boards and superintendents are reverting to bare-bones core academic programs and nervously depleting fund balances..

Schools and hospitals have many similarities.  These include for profit and nonprofit types, boards of directors and CEO’s, differentiated patient or pupil treatments, stringent confidentiality of records, specialized and licensed staff, extensive professional development, substantial physical plants and technology, and diverse business functions.  Schools and hospitals are communities as well as bureaucracies.

Hospitals depend on individual plans for each patient.  They keep detailed records regarding what intervention has been tried, which ones have succeeded or failed and for what length of time.  They must measure change carefully because lives depend on it.. At least for nondisabled children, Schools are less conscientious about individual plans for children .

Schools should take the cue from hospitals and strive toward complete, flexible individual plans for all students.  Thereby, interventions can be applied consistently and for the right length of time.  Special education has helped schools build bridges between themselves and the medical world.  Like medical professionals, school staffs have adopted the use of Response to Intervention (RTI).  Teachers vary the time, frequency and duration of an intervention to meet individual needs, assess and compare data regarding its effects, and then if necessary utilize an alternative intervention. RTI gets at problems before the child fails dismally and has to be remediated.   Educators should expand and refine this model, along with other medical inspirations like brain-based learning, doing group rounds, action research, scientifically valid practices (a federal Race To The Top program priority), referrals and preventive counseling.

Teachers hope to be publicly respected like doctors.   This will not happen unless school boards empower teachers to exercise more discretion to generate students’ individual plans.  Joint accountability for results would be a must, but teachers and children would benefit from more responsive and tailored support.

 Teachers and doctors alike are grappling with an explosion of internet-based information and new technology.  Educators are encouraged to coach or guide students to self-directed learning, while doctors and other medical staff are exploring telemedicine to facilitate prevention, diagnosis, treatment, and rehabilitation in home settings.  The key is to use technology more creatively, not as a convenient substitute for the status quo.

A warning: as technology redefines relationships, shoddy education or physical damage can occur as individuals self-diagnose or take intellectual shortcuts over the Internet.  Technology must reinforce rather than substitute for licensed professional expertise, solid thinking and good judgment.

 The intersection of medicine and education could be called health.  Public schools struggle to combine core academic subjects with learning about mental, emotional and physical health.  By the same token, hospitals and doctors are challenged to reach out to their surrounding communities in ways that could definitely strengthen health services

By promoting healthy communities, schools and hospitals could better serve the public at lower cost.

For many years I have served on the board of a nonprofit network called the Healthy Community Alliance (HCA) in rural western New York.  Largely state grant-funded with six full-time employees, the network  provides or coordinates programs that address chronic disease awareness and prevention, youth mental health, parent education and management including physical activity and nutrition.

The Alliance takes advantage of emerging health and lifestyle priorities for both young and older populations.  It maintains an impressive list of partnerships and affiliations, but relationships with both hospitals and school districts are hampered by apathy or uneasiness because silo thinking lingers.  School and hospital executives should prioritize alliances with regional health networks to close community service gaps more efficiently and cost effectively.

Funding is a minefield for both schools and hospitals.  However, public schools operate in a comparatively controlled fiscal environment.  Elected school boards, annual public budget or tax rate referenda, property tax caps, and mandated reporting requirements keep schools more accountable to their constituencies than most hospitals are accountable to theirs.

Hospital charges depend on a confusing combination of costs derived from different sources.  Hospitals do not publicize standardized fees for specific services.  Usually patients are not in any position to make informed choices.

In a recent Time Magazine special report, Stephen Brill urges significantly lowering the eligibility age for Medicare to 40 so that insurance limits can be extended on certain expensive tests, drugs, and services. Medicare controls costs by reimbursement based on certain standards for treatment.  The standards are published, specific, measurable, and reasonably scientific.  Connecting performance standards to cost reimbursements seems to hold promise not just for medicine, but as well for schools where politics often override educational performance. Medicare may have big flaws, but it also saves big money.   

Everyone wants measurable results to assure performance quality and bang for the buckSchool and hospital leaders should make time to discuss their commonalities.  By climbing out of their boxes, these two institutions could reconnect cost with quality and multiply productivity.

  References

Anderson, G. (Johns Hopkins University) as cited in Babcock, C.R. (2013, March 12). Americans without insurance face escalating hospital bills. The Buffalo News. Buffalo, New York.

Brill, S. (2013, March 6). Bitter Pill: How outrageous pricing and egregious profits are destroying our health care. Time Magazine.

CBS News/New York Times. (2009, June 20). The Debate Over Health Care. National Poll Conducted by CBS News/New York Times.

Healthy Community Alliance, Inc. (2012).  Community Value Report. Gowanda, New York: Author.

Mellard, D.F., and Johnson, E. (2008). RTI: A Practitioner’s Guide to Implementing Response to Intervention. Thousand Oaks, CA: Corwin Press.

Oliff, P., Mai, C, and Leachman, M. (2012, September 4). New School Year Brings More Cuts in State Funding For Schools.  Center on Budget and Policy Priorities. Washington, D.C.

Richardson, W. (2013, March). Students First, Not Stuff. Educational Leadership, 7 (6), 10-14.

USDOE. (2012). The Condition of Education. Washington, D.C.: Author.

1 Comment

Filed under comparing medicine and education

Money Spurs Use of New Technologies in Medical Practice: Schools Also?

Physicians have steadily adopted new technologies from the early 19th century stethoscope to the X-ray decades later to late-20th century computer-tomography scans. Such rapid adoption of new technologies has been (and is) common in medicine. What is uncommon is that medical technology spurred by new ways of funding in the past half-century has come to dominate clinical practice among specialty doctors (but less so among primary care physicians whose revenue is largely generated by office visits). How come? [i]

Public and private insurers pay doctors not only for visits to offices, clinics, and hospitals but also for the diagnostic tests they order such as blood work, sonograms, X-rays, scans, and the treatments they deem best in light of an emerging diagnosis. They also prescribe medications and screen healthy patients for possible diseases including more tests since this system of payments–called fee-for-service–encourage such practices. Fee-for-service payments from private and public insurers depend upon counting patient visits, diagnostic tests, and treatments.

images

Surveys of physicians, in part, support this view of doctors ordering additional tests beyond what may be necessary for the patient. Consider “aggressive care.” In a recent survey of primary care physicians, 28 percent said they ordered more tests or referred patients to specialists—their operational definition of “aggressive care.”  When asked why, physicians responded that they feared malpractice litigation, had to meet clinical performance measures set by insurers, and had insufficient time with patients. Recently a group of nine medical groups of specialists laid out recommendations that would reduce diagnostic tests (e.g., imaging tests for many cases of lower back pain).

images-1

To see how fee-for-service infiltrates daily clinical practice, do the following thought experiment. Suppose you want to build a house for your family. You will need experts, expensive equipment, and materials and have to coordinate all of these. Instead of hiring a general contractor to oversee brick masons, carpenters, electricians, plumbers, and other specialists, you paid electricians for every outlet they recommended, carpenters for cabinets they thought you needed, plumbers for faucets they wanted to install, and brick masons for sidewalks they thought you should have. As an owner of the house, you would have hundreds of outlets, scores of cabinets and faucets and sidewalk after sidewalk winding around your house. The expense would be astronomical and the house would be in dire trouble a few years later.

Physicians, researchers, and policymakers say that lack of coordination and mindfulness in building such a house is what has occurred with fee-for-service dominating private and public payments to most physicians. One doctor pointed out that in the analogy the “general contractor” is what primary care physicians do in coordinating health care for patients. But family medicine and primary care practitioners are shrinking in numbers. Economic incentives through fee-for-service, however, nudge specialists to order diagnostic tests and prescribe treatments using the latest technologies.[ii]

When specialists form groups or hospitals invest in the latest equipment (e.g., imaging machines) and procedures (e.g., arterial stents), incentives to use both often multiply. For cardiologists inserting stents (cost: $30-50 thousand per procedure) to keep arteries open is a huge money-maker. As one medical researcher said: “In many hospitals, the cardiac service line [stent] generates 40 percent of the total hospital revenue, so there’s incredible pressure to do more procedures.”

Financial incentives do reshape clinical practice especially when the federal government puts its fiscal muscle behind certain medical technologies. For example, in the 1960s, a machine was invented to cleanse blood of impurities because of kidney failure. This invention saved lives but was so expensive that only the wealthy could afford it. In 1973, The U.S. Congress amended Medicare to cover full costs for dialysis of patients who would otherwise die.

Uses of technological tests for diagnosis and treatment in medical practice is, in part, a function of financial incentives–the market at work–put in place by private and public systems of funding. Whether it makes a difference in caring for patients–either in healing or extending lives–remains unclear.

Now, switch from doctors using new technologies to diagnose and treat patients to teachers and principals working in classrooms and schools. Think about school reformers who press administrators and teachers to use online instruction, blended learning, iPads, etc., etc., etc. What, if anything, can be learned from the impact of fee-for-service funding spurring greater use of high-tech tests in medical practice and how market incentives can be applied to schools and classrooms?


[i] Stanley Reiser, Medicine and the Reign of Technology (New York: Cambridge University Press, 1978). To be clear, when I refer to medical technology I mean the equipment, devices, drugs, procedures, and processes used to deliver diagnoses and treatments to patients.

[ii] Joel Merenstein, MD. suggested this to me. For shrinking number of primary care physicians, see Thomas Bodenheimer, MD, “Primary Care-Will It Survive?” New England Journal of Medicine, 2006, 355, pp. 861-864.

8 Comments

Filed under comparing medicine and education

Cartoons: Doctors and Teachers

Over the past three years, I have written posts about teachers and doctors comparing and contrasting the two professions on uses of technology, pay-for-performance, reliance on research, and similar topics. For this month’s cartoon entry,* I have chosen a few that got me smiling, chuckling, and even laughing, yes, out loud. Perhaps, you will enjoy some.130128_cn-surgery-1_p465

130128_cn-surgery-2_p465

130128_cn-surgery-3_p465

circumcision

doc empathy

FeeService-EN.sflb

images-1

images

PHD literature

__________

*For earlier monthly posts featuring cartoons, see: “Digital Kids in School,” “Testing,” “Blaming Is So American,”  “Accountability in Action,” “Charter Schools,” and “Age-graded Schools,” Students and Teachers, Parent-Teacher Conferences, Digital Teachers, Addiction to Electronic DevicesTesting, Testing, and Testing, Business and Schools, Common Core Standards, Problems and Dilemmas, Digital Natives (2),  Online Courses,  and Students and Teachers Again.

3 Comments

Filed under comparing medicine and education

Trusting Teaching and Doctors

When was the last time you heard an elected policymaker tell lawmakers the following about state and federal mandates for curriculum standards, accountability, and testing?

The laws that are in fashion demand tightly constrained curricula and reams of accountability data. All the better if it requires quiz-bits of information, regurgitated at regular intervals and stored in vast computers. Performance metrics, of course, are invoked like talismans. Distant authorities crack the whip, demanding quantitative measures and a stark, single number to encapsulate the precise achievement level of every child.

We seem to think that education is a thing—like a vaccine—that can be designed from afar and simply injected into our children. But as the Irish poet, William Butler Yeats said, “Education is not the filling of a pail but the lighting of a fire.”

This year, as you consider new education laws, I ask you to consider the principle of Subsidiarity. Subsidiarity is the idea that a central authority should only perform those tasks which cannot be performed at a more immediate or local level. In other words, higher or more remote levels of government, like the state, should render assistance to local school districts, but always respect their primary jurisdiction and the dignity and freedom of teachers and students.

Subsidiarity is offended when distant authorities prescribe in minute detail what is taught, how it is taught and how it is to be measured. I would prefer to trust our teachers who are in the classroom each day, doing the real work – lighting fires in young minds.

Those words come from Governor of California, Jerry Brown, where 300,000 teachers teach six million children in his State of the State address to the Legislature on January 24, 2013.

Re-read that last sentence. “Trust our teachers.”

That is a stunner given the reform-inflated air we breathe filled by federal, state, and district officials eager to evaluate teachers on the basis of student test scores, implement pay-for-performance plans, and get teachers to prepare lessons for state-adopted Common Core standards in reading and math.

On the heels of a statewide referendum that increased taxes—yes, increased taxes for seven years–Jerry Brown is no novice bellowing the latest school reform slogan. He is a 75 year-old veteran of school reform having served eight years as governor before and after Proposition 13 (1978). He is a hard-driving, practical political leader who knows the ins-and-outs of both local and state politics (e.g. mayor of Oakland, 1999-2007, state Attorney General, 2007-2011). When he says “trust our teachers” he means have confidence in teachers to do the right thing and continually work to improve their performance. That is good news for the state and nation.

Does that mean that the reform tide has turned and teachers will no longer be blamed for mediocre schooling and ignored in policy circles? Hardly. For all I know, Governor Jerry Brown may be an outlier and not the vanguard of a movement to restore trust in teachers by shrinking mandates telling them what to teach and how what is taught will be measured. But it is a welcome sign that has been missing for many years.

Less welcome has been the news for doctors who are hip-deep in pay-4-performance plans across the country. There is a national movement afoot to raise the quality of patient care and contain ever-escalating costs by launching plans that measure how well doctors treat their patients. In New York City public hospitals, for example,  doctors’ annual raises will depend upon performance metrics that include:

“how well patients say their doctors communicate with them, how many patients with heart failure and pneumonia are readmitted within 30 days, how quickly emergency room patients go from triage to beds, whether doctors get to the operating room on time and how quickly patients are discharged.”

The evidence over the past century has been clear that when policymakers and managers concoct high-stakes incentives to measure job effectiveness, award dollar bonuses, or fire employees, professionals and non-professionals learn to game the metrics. Incentives corrupt measures time and again.  There is a history in finance, the military, government, education, and other sectors of how metrics get gamed that policymakers ignore repeatedly. What’s worse, of course, is that doctors, like teachers, become the target rather than the political and socioeconomic structures within which they labor. In doing so, their loyalty to the helping profession in which they serve,  erodes.

Thus, over time, teachers and doctors come to see the “quality” measures used to evaluate and pay them as perverse destroying bonds they have to their students and patients, the institutions, and the very process of teaching and healing. Perhaps, the oath that medical school graduates take upon receiving the M.D. degree should also be taken by policymakers eager for pay-for-performance: Do no harm.

One top policymaker, Governor Jerry Brown, understands that admonition. Not trusting teachers (or not trusting doctors’ professionalism) and making rules that attempt to control performance will ultimately do harm.  I do not know whether you are an outlier or a sign of an emerging sensibility but I thank you Governor Brown for your recent words.

10 Comments

Filed under comparing medicine and education

Reshaping Teaching through Managerial Use of Student Test Scores

The path of educational progress more closely resembles the flight of a butterfly than the flight of a bullet.  Philip Jackson, 1968

Top governmental policymakers and private insurance companies, deeply concerned over ever-rising health care costs and unwilling to rely upon doctors to restrain expenditures, have built structures over the past quarter-century to hold physicians accountable for their actions in diagnosing and treating patients. These structures leaned heavily upon a research base built up over decades from clinical trials on screening procedures to the effects of drugs on an array of diseases. Combining evidence-based medicine with incentives and sanctions, public and private insurers have measured, reported, and rewarded doctors’ performance in hospitals, clinics, and office practices.
In copying outcome-driven corporations, these medical policymakers and insurers relied upon performance-based metrics. They assumed that creating economic incentives for individuals and organizations would increase innovation, lower costs, and improve patient care. They identified numerous measures, confirmed in large part by results from randomized clinical trials embedded in evidence-based medicine, and implemented those measures in hospitals, clinics, and doctors’ offices. Physician “report cards” and pay-for-performance plans, however, have yet to yield promised innovations, high quality care, and reduced costs.

Educational policymakers have made a similar set of assumptions in constructing accountability structures and using metrics for managing how teachers are to be evaluated and paid. In doing so, however, these decision-makers lack the knowledge base in educational research that physicians have had available in evidence-based medicine.

While social scientists and educational researchers have used randomized control group studies to uncover what caused phenomena in schools and classrooms, such studies have been the exception, not the rule. Ethical considerations, cost, and the complexity of schools, teaching, and learning reduce experimental-control research designs.

Qualitative research studies using surveys, interviews, case studies, and ethnographies are not designed to draw causal inferences; moreover, they cannot, given the questions asked, the samples drawn, and methodologies used. Qualitative studies ask different questions and provide rich data for exploring other issues that are missing from experimental-control designs.

As a consequence, unlike physicians who can draw from a literature of randomized control trials and use results for diagnosis and treatment of common and uncommon illnesses (e.g., Cochrane Collaborative), only a small and emerging body of knowledge drawn from randomized clinical trials about teaching, learning, and effective schools yet exists that policymakers and practitioners can tap (e.g., U.S Department of Education, “What Works Clearinghouse,” and Campbell Collaborative).[i]

That slim database, however, has not lessened the current passion among educational policymakers and politicians for using test scores to evaluate teacher performance (and pay higher salaries). The current “science” of value-added measures (VAM) leans heavily upon the work of William Sanders. Smart researchers and officials are determined to re-engineer teaching to make it closer to the “flight of a bullet” rather than the “flight of a “butterfly.” In seeking the Holy Grail, they have ignored the long march that researchers and policymakers have slogged through in the past century to make teaching scientific.[ii]

Not many contemporary reformers can recall Franklin Bobbitt in the 1920s, Ralph Tyler and Benjamin Bloom in the 1950s, Nathaniel Gage in the 1970s and 1980s, and many other researchers who worked hard to create a science of curriculum and instruction.  These scholars rejected the notion that teaching can be unpredictable and uncertain–”the flight of a butterfly.” They believed that teaching could be rational and predictable through scientifically engineering classrooms.

In How To Make a Curriculum (1924), Franklin Bobbitt listed 160 “educational objectives” that teachers should pursue in teaching children such as “the ability to use language …required for proper and effective participation in community life.” Colleagues in math listed 300 for teachers in grades 1-6 and nearly 900 for social studies. This scientific movement to graft “educational objectives” onto daily classroom lessons collapsed of its own weight by the 1940s, and was largely ignored by teachers.[iii]

By the early 1960s, another generation of social scientists had advanced the idea that teachers should use “behavioral objectives” to guide lessons. Ralph Tyler, Benjamin Bloom and others created taxonomies that provided teachers with “prescriptions for the formulation of educational objectives.” Teachers generally ignored these scientific prescriptions in their daily lessons.[iv]

In the 1970s and 1980s, Nathaniel Gage and others sought to establish “ a scientific basis for the art of teaching.” They focused on teaching behaviors (how teachers asked questions, which students are called upon, etc.)–the process of teaching leading to the products of effective teaching, student scores on standardized tests. This line of research called “process-product” continued the behavioral tradition from an earlier generation committed to a science of teaching. Using experimental methods to identify teaching behaviors that were correlated to student gains in test scores on standardized tests, Gage and others came up with “teacher should” statements that were associated with improved student achievement.[v]

The limitations of establishing a set of scientifically prescribed teaching behaviors soon became apparent as critics pointed out how many other factors (e.g., teacher knowledge and beliefs, the content of the lesson, students themselves, the classroom environment, the school) come into play when teachers teach students. Again, teachers generally ignored the results from “process-product” studies.[vi]

And here in 2013, re-engineering teaching through science again seeks “the flight of the bullet.” Evaluating and paying teachers on the basis of student test scores through value-added measures dominates policy talk and action.

In establishing new accountability structures that used squishy metrics and attached high-stakes rewards (e.g., cash bonuses for individual teachers) and sanctions (e.g., no diploma for failing high school students; teachers fired for being ineffective) educational policymakers have plunged into a highly contested arena where the search for teacher effectiveness—“the flight of the bullet”—has generated anger, fear, and lowered morale among those who work daily in classrooms. And, at the same time, generated political gains for elected policymakers.

Recall that under President George W. Bush, the Teacher Incentive Fund made grants to districts for overhauling their teacher evaluation systems. After Barack Obama became President in 2009, the U.S. Department of Education launched Race to the Top, a multi-billion dollar competition among states during a recession when school budgets were cut. To win, states had to meet certain conditions to collect federal dollars. One of those conditions was that states had to create new systems of teacher evaluation that included student test scores. Furthermore, in another federal initiative to turn around failing schools, the U.S. Secretary of Education dispensed School Improvement Grants to districts to overhaul schools with persistent low academic achievement. One of the strategies to turn around such schools included using student test scores to evaluate teachers.[vii]

Philanthropists have pursued similar policies. The Bill and Melinda Gates Foundation awarded grants to six districts to create and establish “fair and reliable measures of effective teaching” including the use of student test scores.  Yet even with all this federal and private money being spent the question remains whether these structures and metrics have reshaped classroom practices.[viii]


[i] Michael Feuer, Lisa Towne, and Richard Shavelson, “Scientific Culture and Educational Research, Educational Researcher, 2002, 31(8), pp. 4-14; for a direct comparison between EBM and EBE see: John Willinsky, “Extending the prospects of evidence-based education. In: Insight, Vol. 1, No. 1, pp. 23-41. For the Cochrane Collaborative, see http://www.cochrane.org/ ; for Campbell Collaborative, see: http://www.campbellcollaboration.org/ ; for What Works Clearinghouse, see: http://ies.ed.gov/ncee/wwc/ .

[ii] Quote comes from: Philip Jackson, Life in Classrooms,  (New York: Holt,Rinehart, and Winston,1968, pp. 166-167. William Sanders and Sandra Horn, “The Tennessee Value-Added Assessment System (TVAAS): Mixed Model Methodology in Educational Assessment,” Journal of Personnel Evaluation in Education, 1994, 8(3), pp. 299-311; Daniel McCaffrey, et. al., “Models for Value-Added Modeling of Teacher Effects,”  Journal of Educational Behavioral Statistics, 2004,  29(1), pp. 67-101.

[iii] Elliot Eisner, “Educational Objectives: Help or Hindrance?” The School Review, 1967, 75, pp. 250-260.

[iv] Ibid.

[v] N.L. Gage, The Scientific Basis for the Art of Teaching (New York: Teachers College Press, 1978).

[vi] Walter Doyle, “Paradigms for Research on Teacher Effectiveness,” Review of Research in Education, 1977, 5, pp. 163-198; N.L. Gage and Margaret Needels, “Process-Product Research on Teaching: A Review of Criticisms,” The Elementary School Journal, 1989, 89(3), pp. 253-300.

[vii] Sarah Garland, “Federal teacher evaluation requirement has wide impact,” http://hechingerreport.org/content/federal-teacher-evaluation-requirement-has-wide-impact_8360/ .

[viii] Bill and Melinda Gates Foundation, “Working with Teachers to Develop Fair and Reliable Measures of Effective Teaching: The MET Project,” 2010. The MetLife Survey of The American Teacher: Teachers, Parents, and the Economy, 2011 (Report published, March 2012), pp. 6-7;  See Scholastic, Inc. and Bill & Melinda Gates Foundation, Primary Sources: 2012: America’s Teachers on the Teaching Profession, pp. 27-29.

10 Comments

Filed under comparing medicine and education, school reform policies