Pay-for-Performance for CEOs and Teachers

Oracle CEO Larry Ellison earns $37,692 an hour. No, that is not a typo or misplaced comma. Ellison’ annual salary ran $78.4 million, much of it in stock option awards. His salary was based on the annual performance of the company’s stock. Oracle’s Board of Directors set the pay scale (Ellison owns one-fourth of the company’s shares) to spur better management to increase profits and shareholders’ dividends.

They pay Ellison to perform well on the metric they have chosen (“company earnings before income taxes minus the costs of stock-based compensation, acquisitions, restructuring, and other items.” This CEO’s performance pay is not, however, a metric used by other major corporations for paying their top person. I return to the point of different measures used by companies to judge CEO performance later.

Switch now to the average U.S. public school teacher who earns an annual salary of over $55,000. That figure translates to around $27.00 for a 40-hour week. Like Ellison, hundreds of thousands of teachers are involved in pay-for-performance plans. In response to the federal Race To the Top competition, many states have mandated that teachers’ performance and salary be tied to students’ test scores to spur better teaching and student learning. Those test scores, as a factor in assessing effectiveness and determining salary (or bonuses), can range from as much as over half to one-quarter of the decision to set salary and retain or fire a teacher.

While I have written about this pay-for-performance reform over the past few years (see here, here, and here), for this post I want to inspect how the private sector–often a model for U.S. school reform–has its own problems, often undisclosed by business-oriented champions of school metrics, in determining CEO pay.

The lesson to learn from this post is: Paying for CEO performance in companies and schools is as flawed as the measures used to determine it.

A recent study of the metrics used in 195 large companies over the past five years showed that the most popular gauge measuring CEO performance was “total shareholder return.” Over half of the companies using that measure, however,  lost nearly two percent over the five-year period. Companies using less popular equations such as “earnings-per-share growth” gained almost three percent.

Now, here’s the clincher. Most companies judging CEO performance are relying on a metric that yielded loses for investors (“total shareholder return”)  yet,  at the same time, those very same companies continued to give their CEOs substantial raises year after year.

The authors of the study believe that the popularity of the performance measure, i.e., “total shareholder return,” stems from how easy it is for boards of directors and CEOs to manipulate the metric by “removing costs from the equation” such as “discontinuing product lines or closing factories.” Boards of directors then can reward CEOs with higher compensation packages. Earnings-per-share growth, a less popular metric and one of multiple measures that many firms use, sorts out under-performing from high performing firms, the authors found. This one as well as other measures, they concluded, are less easily manipulated by top corporate officials. CEO pay, then, can be better associated with company performance.

The main takeaways from this study is that boards of directors and CEOs do manipulate the numbers,  “one size does not fit all when measuring pay for performance,”and that multiple measures for determining effectiveness and salary have a better chance of capturing performance than single ones do.

Now, consider teacher pay-for-performance where one measure–student test scores–is often used to determine to what degree a teacher is effective. Like “total shareholder return” there are serious problems of using this metric alone or even in concert with other measures to judge teacher performance (see here and here).

Consider the following:

Incentives corrupt measures.

Since the mid-1970s, social scientists have criticized the use of specific quantitative measures to monitor or steer policies because those implementing such policies alter their practices to insure better numbers. The work of social scientist Donald T. Campbell and economists in the mid-1970s about the perverse outcomes of incentives was available but have largely been ignored. Campbell wrote in 1976.

“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor” (Campbell 1976, p.54)

Campbell used examples drawn from statistics on police solving crimes (p. 55), the Soviets setting numerical goals in industry (p. 57), and the U.S.’s use of “body counts” in Vietnam as evidence of winning (p.58). For public schools, Campbell said that “achievement tests are … highly corruptible indicators (p.57).”

That was nearly forty years ago. In the past decade, researchers have documented  (also see here) the link between standardized test scores and narrowed instruction to prepare students for test items, instances of  state policymakers fiddling with cut-off scores on tests, increased dropouts, and straight out cheating. Although how the distortions occur are unclear, the evidence confirms Campbell’s insight.

Easy To Measure Indicators Trump Hard To Measure Ones

Few in business, medicine or education question that some indicators are easier to quantify than others. In medicine, for example, hospital mortality and surgical procedures are fairly easy to measure but the results even when compared to other hospitals and surgeons hide as much as they reveal about effective health care. So it is with standardized tests.

Because test scores are inexpensive and efficient to collect, they draw attention away from important but hard-to-measure aspects of teaching and learning such as student engagement, rapport between teachers and students, academic climate in classroom and school, and principal leadership. Cumulative practitioner experience and stories about teaching over centuries have established these as crucial factors in working with gifted and vulnerable students.

*******************************************************************

These are known results of using single measures to judge individual or organizational performance. Consequences of their use can be anticipated. Historical examples abound. Some districts (e.g., Denver) wisely have moved to using multiple measures with student outcomes included that go beyond test scores but in most states where such mandates reign, test scores still remain a major part of the equation used to judge teacher performance (e.g., New York City, Washington, D.C., Houston, Texas) and allocate bonuses to teachers and principals.

This manipulation of data and one-size-fits-all measures show up in businesses as well as schools raising serious questions about the worth of this frenetic passion for pay-for-performance in both public and private sectors.

In the meantime, if Oracle’s Larry Ellison read this post in his office–say 10 minutes–he would have earned over $6,000. Ah, to be a CEO.

 

 

 

1 Comment

Filed under school reform policies

Evidence Based Education Policy and Practice: A Conversation (Francis Schrag)

 

This fictitious exchange between two passionate educators over making educational policy and influencing classroom practice through careful scrutiny of evidence–such as has occurred in medicine and the natural sciences–as opposed to relying on professional judgment anchored in expertise gathered in schools brings out a fundamental difference among educators and the public that has marked public debate over the past three decades. The center of gravity in making educational policy in the U.S. has shifted from counting resources that go into schooling and relying on professional judgment to counting outcomes students derive from their years in schools and what the numbers say.

That shift can be dated from the Elementary and Secondary Education Act of 1965 but gained sufficient traction after the Nation at Risk report (1983) to dominate debate over innovation, policy, and practice. Although this is one of the longest guest posts I have published, I found it useful (and hope that viewers will as well) in making sense of a central conflict that exist today within and among school reformers, researchers, teachers, policymakers and parents.

Francis Schrag is professor emeritus in the philosophy of education at the University of Wisconsin, Madison. This article appeared in Teachers College Record, March 14, 2014.

A dialogue between a proponent and opponent of Evidence Based Education Policy. Each position is stated forcefully and each reader must decide who has the best of the argument.

Danielle, a professor of educational psychology and Leo, a school board member and former elementary school teacher and principal, visit a middle-school classroom in Portland Maine where students are deeply engaged in building robots out of Lego materials, robots that will be pitted against other robots in contests of strength and agility.  The project requires them to make use of concepts they’ve learned in math and physics.  Everything suggests that the students are deeply absorbed in what is surely a challenging activity, barely glancing around to see who has entered their classroom.

Leo:  Now this is exciting education. This is what we should be moving towards.  I wish all teachers could see this classroom in action.

Danielle:  Not so fast.  I’ll withhold judgment till I have some data.  Let’s see how their math and science scores at the end of the year compare with those of the conventional classroom we visited this morning.  Granted that one didn’t look too out of the ordinary, but the teacher was really working to get the kids to master the material.

Leo:  I don’t see why you need to wait.  Can’t you see the difference in level of engagement in the two classrooms?  Don’t you think the students will remember this experience long after they’ve forgotten the formula for angular momentum? Your hesitation reminds me of a satirical article a friend showed me; I think it came from a British medical journal.  As I recall the headline went: “Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomized controlled trials.”

Danielle:  Very cute, but let’s get serious.  Spontaneous reactions can be misleading; things aren’t always what they appear to be, as I’m sure you’ll agree.  I grant you that it looks as if the kids in this room are engaged, but we don’t know whether they’re engaged in the prescribed tasks and we don’t know what they’re actually learning, do we?  We’ll have a much better idea when we see the comparative scores on the test.  The problem with educators is that they get taken in with what looks like it works, they go with hunches, and what’s in fashion, but haven’t learned to consult data to see what actually does work.  If physicians hadn’t learned to consult data before prescribing, bloodletting would still be a popular treatment.

Suppose you and I agreed on the need for students to study math and physics.  And suppose that it turned out that the kids in the more conventional classroom learned a lot more math and physics, on average, as measured on tests, than the kids in the robotics classroom.  Would you feel a need to change your mind about what we’ve just seen?  And, if not, shouldn’t you?  Physicians are now on board with Evidence Based Medicine (EBM) in general, and randomized controlled trials (RCTs) in particular, as the best sources of evidence.  Why are teachers so allergic to the scientific method?  It’s the best approach we have to determine educational policy.

Leo:  Slow down Danielle.  You may recall that a sophisticated RCT convincingly showed the benefits of smaller class sizes in elementary schools in Tennessee, but these results were not replicated when California reduced its elementary school class size, because there was neither room in the schools for additional classrooms nor enough highly skilled teachers to staff them.  This example is used by Nancy Cartwright and Jeremy Hardie in their book on evidence-based policy to show that the effectiveness of a policy depends, not simply on the causal properties of the policy itself, but on what they call a “team” of support factors (2012, p. 25).  If any one of these factors were present in the setting where the trial was conducted but is lacking in the new setting, the beneficial results will not be produced.  This lack of generalizability, by the way, afflicts RCTs in medicine too.  For instance, the populations enrolled in teaching hospital RCTs are often different from those visiting their primary care physician.

Danielle:  I have to agree that educators often extrapolate from RCTs in a way that’s unwarranted, but aren’t you, in effect, calling for the collection of more and better evidence, rather than urging the abandonment of the scientific approach.  After all, the Cartwright and Hardie book wasn’t written to urge policy makers to throw out the scientific approach and go back to so-called expert or professional judgment, which may be no more than prejudice or illicit extrapolation based on anecdotal evidence.

Leo:  You seem to be willing to trust the data more than the judgment of seasoned professionals.  Don’t you think the many hours of observing and teaching in actual classrooms counts for anything?

Danielle: If your district has to decide which program to run, the robotics or the traditional, do you really want to base your decision on the judgment of individual teachers or principals, to say nothing of parents and interested citizens?  In medicine and other fields, meta-analyses have repeatedly shown that individual clinical judgment is more prone to error than decisions based on statistical evidence (Howick, 2011, Chap. 11). And, as I already mentioned, many of the accepted therapies of earlier periods, from bloodletting to hormone replacement therapy, turned out to be worse for the patients than doing nothing at all.

Now why should education be different?  How many teachers have “known” that the so-called whole-word method was the best approach to teaching reading, and years later found out from well-designed studies that this is simply untrue?  How many have “known” that children learn more in smaller classes?  No, even if RCTs aren’t always the way to go, I don’t think we can leave these things to individual educator judgment; it’s too fallible.

And you may not need to run a new study on the question at issue.  There may already be relevant, rigorous studies out there, testing more exploratory classrooms against more traditional ones in the science and math area for middle-schoolers.  I recommend you look at the federal government What Works website, which keeps track of trial results you can rely on.

Leo:  I’ve looked at many of these studies, and I have two problems with them.  They typically use test score gains as their indicator of durable educational value, but these can be very misleading.  Incidentally, there’s a parallel criticism of the use of “surrogate end points” like blood levels in medical trials.  Moreover, according to Goodhart’s Law—he was a British economist—once a measure becomes a target, it ceases to be a good indicator.  This is precisely what happens in education: the more intensely we focus on raising a test score by means of increasing test preparation to say nothing of cheating—everything from making sure the weakest, students don’t take the test to outright changing students’ answers—the less it tells us about what kids can do or will do outside the test situation.

Danielle:  Of course we need to be careful about an exclusive reliance on test scores.  But you can’t indict an entire approach because it has been misused on occasion.

Leo: I said there was a second problem, as well.  You recall that what impressed us about the robotics classroom was the level of involvement of the kids.  When you go into a traditional classroom, the kids will always look at the door to see who’s coming in.  That’s because they’re bored and looking for a bit of distraction.  Now ask yourself, what does that involvement betoken. It means that they’re learning that science is more than memorizing a bunch of facts, that math is more than solving problems that have no meaning or salience in the real world, that using knowledge and engaging in hard thinking in support of a goal you’ve invested in is one of life’s great satisfactions.  Most kids hate math and the American public is one of the most scientifically illiterate in the developed world.  Why is that?  Perhaps it’s because kids have rarely used the knowledge they are acquiring to do anything besides solve problems set by the teacher or textbook.

I’m sure you recall from your studies in philosophy of education the way John Dewey called our attention in Experience and Education to what he called, the greatest pedagogical fallacy, “the notion that a person learns only the particular thing he is studying at the time” (Dewey, 1938, p. 48).  Dewey went on to say that what he called “collateral learning,” the formation of “enduring attitudes” was often much more important than the particular lesson, and he cited the desire to go on learning as the most important attitude of all.  Now when I look at that robotics classroom, I can see that those students are not just learning a particular lesson, they’re experiencing the excitement that can lead to a lifetime of interest in science or engineering even if they don’t select a STEM field to specialize in.

Danielle:  I understand what Dewey is saying about “collateral learning.”  In medicine as you know, side effects are never ignored, and I don’t deny that we in education are well behind our medical colleagues in that respect.  Still, I’m not sure I agree with you and Dewey about what’s most important, but suppose I do.  Why are you so sure that the kids’ obvious involvement in the robotics activity will generate the continuing motivation to keep on learning?  Isn’t it possible that a stronger mastery of subject matter will have the very impact you seek?  How can we tell?  We’d need to first find a way to measure that “collateral learning,” then preferably conduct a randomized, controlled trial, to determine which of us is right.

Leo:  I just don’t see how you can measure something like the desire to go on learning, yet, and here I agree with Dewey, it may be the most important educational outcome of all.

Danielle:  This is a measurement challenge to be sure, but not an insurmountable one.  Here’s one idea: let’s track student choices subsequent to particular experiences.  For example, in a clinical trial comparing our robotics class with a conventional middle school math and science curriculum, we could track student choices of math and science courses in high school.  Examination of their high school transcripts could supply needed data.  Or we could ask whether students taking the robotics class in middle school were more likely (than peers not selected for the program) to take math courses in high school, to major in math or science in college, etc.  Randomized, longitudinal designs are the most valid, but I admit they are costly and take time.

Leo: I’d rather all that money went into the kids and classrooms.

Danielle:  I’d agree with you if we knew how to spend it to improve education.  But we don’t, and if you’re representative of people involved in making policy at the school district level, to say nothing of teachers brainwashed in the Deweyian approach by teacher educators, we never will.

Leo:  That’s a low blow, Danielle, but I haven’t even articulated my most fundamental disagreement with your whole approach, your obsession with measurement and quantification, at the expense of children and education.

Danielle:  I’m not sure I want to hear this, but I did promise to hear you out.  Go ahead.

Leo:  We’ve had about a dozen years since the passage of the No Child Left Behind Act to see what an obsessive focus on test scores looks like and it’s not pretty.  More and more time is taken up with test-prep, especially strategies for selecting right answers to multiple-choice questions.  Not a few teachers and principals succumb to the temptation to cheat, as I’m sure you’ve read.  Teachers are getting more demoralized each year, and the most creative novice teachers are finding jobs in private schools or simply not entering the profession.  Meanwhile administrators try to game the system and spin the results.  But even they have lost power to the statisticians and other quantitatively oriented scholars, who are the only ones who can understand and interpret the test results.  Have you seen the articles in measurement journals, the arcane vocabulary and esoteric formulas on nearly every page?

And do I have to add that greedy entrepreneurs with a constant eye on their bottom lines persuade the public schools to outsource more and more of their functions, including teaching itself.  This weakens our democracy and our sense of community.  And even after all those enormous social costs, the results on the National Assessment of Educational Progress are basically flat and the gap between black and white academic achievement—the impetus for passing NCLB in the first place—is as great as it ever was.

Danielle:  I agree that it’s a dismal spectacle.  You talk as if educators had been adhering to Evidence Based Policy for the last dozen years, but I’m here to tell you they haven’t and that’s the main reason, I’d contend, that we’re in the hole that we are.  If educators were less resistant to the scientific approach, we’d be in better shape today.  Physicians have learned to deal with quantitative data, why can’t teachers, or are you telling me they’re not smart enough?  Anyhow, I hope you feel better now that you’ve unloaded that tirade of criticisms.

Leo:  Actually, I’m not through, because I don’t think we’ve gotten to the heart of the matter yet.

Danielle:  I’m all ears.

Leo:  No need to be sarcastic, Danielle.  Does the name Michel Foucault mean anything to you?  He was a French historian and philosopher.

Danielle:  Sure, I’ve heard of him.  A few of my colleagues in the school of education, though not in my department, are very enthusiastic about his work.  I tried reading him, but I found it tough going.  Looked like a lot of speculation with little data to back it up.  How is his work relevant?

Leo:   In Discipline and Punish, Foucault described the way knowledge and power are intertwined, especially in the human sciences, and he used the history of the school examination as a way of illustrating his thesis (1975/1995, pp. 184-194).  Examinations provide a way of discovering “facts” about individual students, and a way of placing every student on the continuum of test-takers.  At the same time, the examination provides the examiners, scorers and those who make use of the scores ways to exercise power over kids’ futures.  Think of the Scholastic Assessment Tests (SATs) for example.  Every kid’s score can be represented by a number and kids can be ranked from those scoring a low of 600 to those with perfect scores of 2400.  Your score is a big determinant of what colleges will even consider you for admission.  But that’s not all: Foucault argued that these attempts to quantify human attributes create new categories of young people and thereby determine how they view themselves.  If you get a perfect SAT score, or earn “straight As” on your report card, that becomes a big part of the way others see you and how you see yourself.  And likewise for the mediocre scorers, the “C” students, or the low scorers who not only have many futures closed to them, but may see themselves as “losers,” “failures,” “screw-ups.”  A minority may, of course resist and rebel against their placement on the scale—consider themselves to be “cool”, unlike the “nerds” who study, but that won’t change their position on the continuum or their opportunities.  Indeed, it may limit them further as they come to be labeled “misfits” “ teens at-risk,” “gang-bangers” and the like. But, and here’s my main point, this entire system is only possible due to our willingness to represent the capabilities and limitations of children and young people by numerical quantities.  It’s nothing but scientism, the delusive attempt to force the qualitative, quirky, amazingly variegated human world into a sterile quantitative straight-jacket.  You recall the statement that has been attributed to Einstein, don’t you, “Not everything that can be counted counts, and not everything that counts can be counted.” I just don’t understand your refusal to grasp that basic point; it drives me mad.

Danielle:  Calm down, Leo.  I don’t disagree that reducing individuals to numbers can be a problem; every technology has a dark side, I’ll grant you that, but think it through.  Do you really want to go back to a time when college admissions folks used “qualitative” judgments to determine admissions?  When interviewers could tell from meeting a candidate or receiving a letter of recommendation if he were a member of “our crowd,” would know how to conduct himself at a football game, cocktail party, or chapel service, spoke without an accent, wasn’t a grubby Jew or worse, a “primitive” black man or foreign-born anarchist or communist.  You noticed I used the masculine pronoun:  Women, remember, were known to be incapable of serious intellectual work, no data were needed, the evidence was right there in plain sight.  Your Foucault is not much of a historian, I think.

Leo:  We have some pretty basic disagreements here.  I know we each believe we’re right.  Is there any way to settle the disagreement?

Danielle:  I can imagine a comprehensive, longitudinal experiment in a variety of communities, some of which would carry out EBEP and control communities that would eschew all use of quantification.  After a long enough time, maybe twenty years, we’d take a look at which communities were advancing, which were regressing.  Of course, this is just an idea; no one would pay to actually have it done.

Leo:  But even if we conducted such an experiment, how would we know which approach was successful?

Danielle:  We shouldn’t depend on a single measure, of course.  I suggest we use a variety of measures, high school graduation rate, college attendance, scores on the National Assessment of Educational Progress, SATs, state achievement tests, annual income in mid-career, and so on.  And, of course, we could analyze the scores by subgroups within communities to see just what was going on.

Leo:  Danielle, I can’t believe it.  You haven’t listened to a word I’ve said.

Danielle:  What do you mean?

Leo:   If my favored policy is to eschew quantitative evidence altogether, wouldn’t I be inconsistent if I permitted the experiment to be decided by quantitative evidence, such as NAEP scores or worse, annual incomes?  Don’t you recall that I reject your fundamental assumption—that durable, significant consequences of educational experiences can be represented as quantities?

Danielle:  Now I’m the one that’s about to scream.  Perhaps you could assess a single student’s progress by looking at her portfolio at the beginning and end of the school year.  How, in the absence of quantification, though, can you evaluate an educational policy that affects many thousands of students?  Even if you had a portfolio for each student, you’d still need some way to aggregate them in order to be in a position to make a judgment about the policy or program that generated those portfolios.  You gave me that Einstein quote to clinch your argument.  Well, let me rebut that with a quotation by another famous and original thinker, the Marquis de Condorcet, an eighteenth century French philosopher and social theorist.  Here’s what he said:  “if this evidence cannot be weighted and measured, and if these effects cannot be subjected to precise measurement, then we cannot know exactly how much good or evil they contain” (Condorcet, 2012, p.138).  The point remains true, whether in education or medicine.  If you can’t accept it, I regret to say, we’ve reached the end of the conversation.

References

Cartwright, N & Hardie, J. (2012). Evidence-based policy:  A practical guide to doing it better.  Oxford and New York: Oxford University Press.

Condorcet, M. (2012). The sketch. In S. Lukes, and N. Urbinati (Eds.), Political Writings (pp. 1-147). Cambridge: Cambridge University Press.

Dewey, J. (1938/1973). Experience and education.  New York: Collier Macmillan Publishers.

Foucault, M. (1995).  Discipline and punish: The birth of the prison. (A. Sheridan, Trans.) New York: Vintage Books. (Original work published in 1975)

Howick, J. (2011). The Philosophy of evidence-based medicine. Oxford: Blackwell Publishing.

21 Comments

Filed under comparing medicine and education, school reform policies

What’s The Evidence on School Devices and Software Improving Student Learning?

The historical record is rich in evidence that research findings have played a subordinate role in making educational policy. Often, policy choices were (and are) political decisions. There was no research, for example, that found establishing tax-supported public schools in the early 19th century was better than educating youth through private academies. No studies persuaded late-19th century educators to import the kindergarten into public schools. Ditto for bringing computers into schools a century later.

So it is hardly surprising, then, that many others, including myself, have been skeptical of the popular idea that evidence-based policymaking and evidence-based instruction can drive teaching practice. Those doubts have grown larger when one notes what has occurred in clinical medicine with its frequent U-turns in evidence-based “best practices.”

Consider, for example, how new studies have often reversed prior “evidence-based” medical procedures.

*Hormone therapy for post-menopausal women to reduce heart attacks was found to be more harmful than no intervention at all.

*Getting a PSA test to determine whether the prostate gland showed signs of cancer for men over the age of 50 was “best practice” until 2012 when advisory panels of doctors recommended that no one under 55 should be tested and those older  might be tested if they had family histories of prostate cancer.

And then there are new studies that recommend women to have annual mammograms, not at age  50 as recommended for decades, but at age 40. Or research syntheses (sometimes called “meta-analyses”) that showed anti-depressant pills worked no better than placebos.

These large studies done with randomized clinical trials–the current gold standard for producing evidence-based medical practice–have, over time, produced reversals in practice. Such turnarounds, when popularized in the press (although media attention does not mean that practitioners actually change what they do with patients) often diminished faith in medical research leaving most of us–and I include myself–stuck as to which healthy practices we should continue and which we should drop.

Should I, for example, eat butter or margarine to prevent a heart attack? In the 1980s, the answer was: Don’t eat butter, cheese, beef, and similar high-saturated fat products. Yet a recent meta-analysis of those and subsequent studies reached an opposite conclusion.

Figuring out what to do is hard because I, as a researcher, teacher, and person who wants to maintain good health has to sort out what studies say and  how those studies were done from what the media report, and then how all of that applies to me. Should I take a PSA test? Should I switch from margarine to butter?

If research into clinical medicine produces doubt about evidence-based practice, consider the difficulties of educational research–already playing a secondary role in making policy and practice decisions–when findings from long-term studies of innovation conflict with current practices. Look, for example, at computer use to transform teaching and improve student achievement.

Politically smart state and local policymakers believe that buying new tablets loaded with new software, deploying them to K-12 classrooms, and watching how the devices engage both teachers and students is a “best practice.” The theory is that student engagement through the device and software will dramatically alter classroom instruction and lead to improved  achievement. The problem, of course–sure, you already guessed where I was going with this example–is that evidence of this electronic innovation transforming teaching and achievement growth is not only sparse but also unpersuasive even when some studies show a small “effect size.”

Turn now to the work of John Hattie, a Professor at the University of Auckland (NZ), who has synthesized the research on different factors that influence student achievement and measured their impact on learning. For example, over the last two decades, Hattie has examined over 180,000 studies accumulating 200, 000 “effect sizes”  measuring the influence of teaching practices on student learning. All of these studies represent over 50 million students.

He established which factors influenced student learning–the “effect size–by ranking each from 0.1 (hardly any influence) to 1.0 or a full standard deviation–almost a year’s growth in student learning. He found that the “typical” effect size of an innovation was 0.4.

To compare different classroom approaches shaped student learning, Hattie used the “typical” effect size (0.4) to mean that a practice reached the threshold of influence on student learning (p. 5). From his meta-analyses, he then found that class size had a .20 effect (slide 15) while direct instruction had a .59 effect (slide 21). Again and again, he found that teacher feedback had an effect size of .72 (slide 32). Moreover, teacher-directed strategies of increasing student verbalization (.67) and teaching meta-cognition strategies (.67) had substantial effects (slide 32).

What about student use of computers (p. 7)? Hattie included many “effect sizes” of computer use from distance education (.09), multimedia methods (.15), programmed instruction (.24), and computer-assisted instruction (.37). Except for “hypermedia instruction” (.41), all fell below the “typical ” effect size (.40) of innovations improving student learning (slides 14-18). Across all studies of computers, then, Hattie found an overall effect size of .31 (p. 4).

According to Hattie’s meta-analyses, then, introducing computers to students will  fall well below other instructional strategies that teachers can and do use. Will Hattie’s findings convince educational policymakers to focus more on teaching? Not as long as political choices trump research findings.

Even if politics were removed from the decision-making equation, there would still remain the major limitation of  most educational and medical research. Few studies  answer the question: under what conditions and with which students and patients does a treatment work? That question seldom appears in randomized clinical trials. And that is regrettable.

 

 

26 Comments

Filed under comparing medicine and education, how teachers teach, technology use

Business as Usual in Corporations and Schools

In the heart of Silicon Valley where start-ups are a way of life, open space offices and teams rule the landscape. Even at the biggest of the big companies such as Google and Facebook, power struggles among and between bureaucrats are a thing of the past. “Move fast and break things” is a Facebook’s slogan. Flat organizations, no elaborate hierarchies, and constant change dominate. Or so, everyone seems to say. See here, here, and here.

Then along comes a Stanford professor who says: “Sorry Kids, Corporate Power Hasn’t Changed.” Jeffrey Pfeffer at the Graduate School of Business since 1979 has studied organizations for decades. According to Pfeffer, Silicon Valley firms–big and small–recruit engineers and programmers to become managers by saying:

We’re not political here. We’re young, cool, socially networked, hip, high-technology people focused on building and selling great products. We’re family-friendly, have fewer management levels and less hierarchy, and make decisions collegially.

It just ain’t so, according to Pfeffer. He points out that hierarchies exists in all organizations and power, acquiring status, and control over ideas and practices are in play unceasingly. He points to the power struggles that occurred at the birth of Twitter and the frequent turnover of CEOs as Hewlett-Packard. And hierarchy is alive and well at Facebook and Google where dual classes of stock “allow the founders to retain the lion’s share of control.” An infographic on hierarchy at both firms would have strengthened his argument even further.

Pfeffer concludes his article with the flat-out statement:

Competition for status and advancement exists not only over time and across countries but also in virtually all species. In short, whether we like it or not, the rules of power abide largely unchanged. People who ignore these principles do so at their peril.

I was struck by Pfeffer’s points that amid all of the talk about change, flat organizations, and team-work, the constancy of competition within companies for power and status remains. Even in Silicon Valley.

Power, Status, and Hierarchy in Public Schools

A similar rhetoric pervades the quest for effective schooling. Reformers, both on the political left and right, say teachers need to collaborate, network, and build strong school cultures where instruction and learning are primary goals. See here, here, and here. But talk is cheap. Beyond the words, what are the organizational realities (i.e., tall or flat, hierarchical or teams) in public schools?

Most U.S. elementary schools are already “flat” organizationally. There is a principal, a few administrative and instructional aides, building staff, and the largest group of all, the  teachers who report to the principal. That’s it. In larger secondary schools there are more administrators, staff, and rules but few hierarchical strata separate teachers from their principals. The largest number of staff in middle and high schools are teachers. But rules also come from district and state offices.

Regulations abound in schools because districts are creatures of the state which, in turn, makes educational policy for everyone. So district administrators try to make sure that local and state policies are followed in schools. School-site principals do the same with teachers. In short, even with a flat school-site organization, bureaucratic levels exist in school districts and the state which means that elbowing for higher status and getting more clout occur in schools, districts, and state departments of education. Here’s the catch, however.

With all of the rules and hierarchical levels from classroom through the state superintendent of education, teachers have one things in their power to do: close the classroom door. They are (and have been) gatekeepers for student content, skills, and attitudes.

What about charter schools that have autonomy and are free from most district and state regulations? KIPP, Aspire, and other groups of charter schools have state and national organizations that make rules for individual schools to follow. As in public schools, however, charter school teachers can close their doors.

Teachers as gatekeepers exist because the organizational reality of both regular and charter schools is that they are age-graded and each teacher has a self-contained classroom with a door to close. Teachers have power within their classroom but little outside of it unless they develop a support network, a culture within the school. And, from time to time, that has occurred in both charter and regular schools.

Consider all the talk of moving to project-based learning and shifting the teacher’s role from the sage-on-stage to guide-on-the-side. Periodically, school reformers for more than a century have coerced, urged, and pleaded with teachers to change their dominant teacher-centered forms of instruction into more student-centered ones along the lines mentioned above.

On occasion, some of the reforms have stuck in some schools where teachers  weathered criticism and supported one another through cascades of hype and criticism. In these scattered instances, teachers kept their doors open and built a stable school culture supporting such instructional reforms as teaching in small groups regularly, sustaining open classrooms, using project-based learning, and creating rich student-centered activities (see here and here). But not most teachers who returned time and again to practices that worked better for them than the “new” reform simply because they could close their classroom doors.

These are (and have been) abiding features of public schools and companies that no amount of talk and hype about doing business differently has changed. Even in Silicon Valley.

 

17 Comments

Filed under leadership, Reforming schools

Don’t Help Your Kids With Their Homework (Dana Goldstein)

Dana Goldstein is a Brooklyn-based journalist, a Schwartz Fellow at the New America Foundation, and a Puffin Fellow at the Nation Institute. This article appeared March 19, 2014 in Atlantic Online

One of the central tenets of raising kids in America is that parents should be actively involved in their children’s education: meeting with teachers, volunteering at school, helping with homework, and doing a hundred other things that few working parents have time for. These obligations are so baked into American values that few parents stop to ask whether they’re worth the effort.

Until this January, few researchers did, either. In the largest-ever study of how parental involvement affects academic achievement, Keith Robinson, a sociology professor at the University of Texas at Austin, and Angel L. Harris, a sociology professor at Duke, mostly found that it doesn’t. The researchers combed through nearly three decades’ worth of longitudinal surveys of American parents and tracked 63 different measures of parental participation in kids’ academic lives, from helping them with homework, to talking with them about college plans, to volunteering at their schools. In an attempt to show whether the kids of more-involved parents improved over time, the researchers indexed these measures to children’s academic performance, including test scores in reading and math.

What they found surprised them. Most measurable forms of parental involvement seem to yield few academic dividends for kids, or even to backfire—regardless of a parent’s race, class, or level of education.

Do you review your daughter’s homework every night? Robinson and Harris’s data, published in The Broken Compass: Parental Involvement With Children’s Education, show that this won’t help her score higher on standardized tests. Once kids enter middle school, parental help with homework can actually bring test scores down, an effect Robinson says could be caused by the fact that many parents may have forgotten, or never truly understood, the material their children learn in school.

Similarly, students whose parents frequently meet with teachers and principals don’t seem to improve faster than academically comparable peers whose parents are less present at school. Other essentially useless parenting interventions: observing a kid’s class; helping a teenager choose high-school courses; and, especially, disciplinary measures such as punishing kids for getting bad grades or instituting strict rules about when and how homework gets done. This kind of meddling could leave children more anxious than enthusiastic about school, Robinson speculates. “Ask them ‘Do you want to see me volunteering more?

Going to school social functions? Is it helpful if I help you with homework?’ ” he told me. “We think about informing parents and schools what they need to do, but too often we leave the child out of the conversation.”

One of the reasons parental involvement in schools has become dogma is that the government actively incentivizes it. Since the late 1960s, the federal government has spent hundreds of millions of dollars on programs that seek to engage parents—especially low-income parents—with their children’s schools. In 2001, No Child Left Behind required schools to establish parent committees and communicate with parents in their native languages. The theory was that more active and invested mothers and fathers could help close the test-score gap between middle-class and poor students. Yet until the new study, nobody had used the available data to test the assumption that close relationships between parents and schools improve student achievement.

While Robinson and Harris largely disproved that assumption, they did find a handful of habits that make a difference, such as reading aloud to young kids (fewer than half of whom are read to daily) and talking with teenagers about college plans. But these interventions don’t take place at school or in the presence of teachers, where policy makers exert the most influence—they take place at home.

What’s more, although conventional wisdom holds that poor children do badly in school because their parents don’t care about education, the opposite is true. Across race, class, and education level, the vast majority of American parents report that they speak with their kids about the importance of good grades and hope that they will attend college. Asian American kids may perform inordinately well on tests, for example, but their parents are not much more involved at school than Hispanic parents are—not surprising, given that both groups experience language barriers. So why are some parents more effective at helping their children translate these shared values into achievement?

Robinson and Harris posit that greater financial and educational resources allow some parents to embed their children in neighborhoods and social settings in which they meet many college-educated adults with interesting careers. Upper-middle-class kids aren’t just told a good education will help them succeed in life. They are surrounded by family and friends who work as doctors, lawyers, and engineers and who reminisce about their college years around the dinner table. Asian parents are an interesting exception; even when they are poor and unable to provide these types of social settings, they seem to be able to communicate the value and appeal of education in a similarly effective manner.

As part of his research, Robinson conducted informal focus groups with his undergraduate statistics students at the University of Texas, asking them about how their parents contributed to their achievements. He found that most had few or no memories of their parents pushing or prodding them or getting involved at school in formal ways. Instead, students described mothers and fathers who set high expectations and then stepped back. “These kids made it!,” Robinson told me. “You’d expect they’d have the type of parental involvement we’re promoting at the national level. But they hardly had any of that. It really blew me away.”

Robinson and Harris’s findings add to what we know from previous research by the sociologist Annette Lareau, who observed conversations in homes between parents and kids during the 1990s. Lareau found that in poor and working-class households, children were urged to stay quiet and show deference to adult authority figures such as teachers. In middle-class households, kids learned to ask critical questions and to advocate for themselves—behaviors that served them well in the classroom.

Robinson and Harris chose not to address a few potentially powerful types of parental involvement, from hiring tutors or therapists for kids who are struggling, to opening college savings accounts. And there’s the fact that, regardless of socioeconomic status, some parents go to great lengths to seek out effective schools for their children, while others accept the status quo at the school around the corner.

Although Robinson and Harris didn’t look at school choice, they did find that one of the few ways parents can improve their kids’ academic performance—by as much as eight points on a reading or math test—is by getting them placed in the classroom of a teacher with a good reputation. This is one example for which race did seem to matter: white parents are at least twice as likely as black and Latino parents to request a specific teacher. Given that the best teachers have been shown to raise students’ lifetime earnings and to decrease the likelihood of teen pregnancy, this is no small intervention.

All in all, these findings should relieve anxious parents struggling to make time to volunteer at the PTA bake sale. But valuing parental involvement via test scores alone misses one of the ways in which parents most impact schools. Pesky parents are often effective, especially in public schools, at securing better textbooks, new playgrounds, and all the “extras” that make an educational community come to life, like art, music, theater, and after-school clubs. This kind of parental engagement may not directly affect test scores, but it can make school a more positive place for all kids, regardless of what their parents do or don’t do at home. Getting involved in your children’s schools is not just a way to give them a leg up—it could also be good citizenship.

 

11 Comments

Filed under raising children

Moving Forward without a Backward Glance: MOOCs and Technological Innovations

In a recent commentary on the rock star Sting’s dipping back into his childhood to revitalize his song writing, David Brooks said: “how important it is to ground future vision in historical consciousness.” I agree with Brooks when it comes to the half-life  of technological innovations. The experience of Massive  Open Online Courses (MOOCs) over the past few years is an unexpected example of what Brooks meant.

Much has been written about MOOCs  since they went viral in the past three years (see here, here, here, and here). This vision of creating platforms for college-level courses that would give anyone with an Internet connection access to college courses while reducing ever-escalating costs of higher education has turned some professors into academic entrepreneurs. Here is a two-for-one innovation (increased efficiency and equity) that has married new technologies with global access to higher education. MOOCs spread rapidly among elite institutions (e.g., Harvard, MIT, Stanford) and some second- and third-tier universities. For those familiar with the Gartner hype cycle–which many acolytes of MOOCs somehow either missed or ignored–the first two phases of the cycle were textbook examples:

“Technology Trigger: A potential technology breakthrough kicks things off. Early proof-of-concept stories and media interest trigger significant publicity. Often no usable products exist and commercial viability is unproven.

Peak of Inflated Expectations: Early publicity produces a number of success stories—often accompanied by scores of failures. Some companies take action; many do not.”

Recent articles (see here and here) express disappointment mixed with hope over how MOOCs have fared since the first blush of the academic love affair with the innovation. The evidence thus far is ample: high dropout rates, little knowledge of what students who completed a MOOC actually learned, lack of faculty enthusiasm, and the real sticking point for universities–how to make money from offering MOOCs? No surprise, then, that the birth rate of new MOOCs has plummeted. We are now in the “Trough of Disillusionment” phase of the cycle.

images

 

 

 

 

The high hopes and inspired rhetoric pushing MOOCs have collapsed. Looking back, the creators were pained–one of them, Sebastian Thrun, has departed from the MOOC scene–and I must add, terribly innocent about earlier technological innovations in education.

Of course, I do not know how (or whether) the next phase (“Slope of Enlightenment”) will unfold. No one does. It is a work in progress. But how does all of this current disappointment with MOOCs connect to the point I raised in the first paragraph: “how important it is to ground future vision in historical consciousness?”

Would knowing the checkered history of technological innovations in K-12 schools and higher education–including the Hype Cycle–help high-tech innovators “ground their future vision?” Yes, it would but I doubt if lessons drawn from earlier innovations would help them alter what they will do anyway. While innovators are creative and hopeful about the future they may be, in David Brooks’ words, “necessarily naive.”

And it is that phrase “necessarily naive” that creates the paradox previous high-tech innovators and school reformers have faced and do so now.

The paradox works like this: If I know well what has occurred with past technological innovations seeking to reshape K-12 and higher education, that is, most fail in the first few years, I would not even try. However, if I don’t care about those past efforts  but still forge ahead because I have faith that what I propose will work regardless of the odds, then I can succeed.

The paradox of forging ahead without a backward glance is 100 percent  American.  Consider often described characteristics of being American: highly individualistic, competitive, optimistic, believes in change, especially technological, as an unvarnished good and that anyone with grit who works hard can overcome any obstacle. There are other characteristics associated with being American including beliefs in equality, a strong work ethic, and fairness.

Running like a red thread in the white fabric of being American, however, is the pervasive belief that if you know the past well, it can be a drag–a disincentive, economists would say–for action, invention, and making progress. To avoid looking backward in order to innovate, one has to be “necessarily naive” in the face of past failures in new technologies. Hence, with “naive” entrepreneurs ignoring the past, there has been a swift rise in and decline of MOOCs.

A skeptic might say: Really, Larry, what would you have to know about past technological innovations that might have helped the founders of MOOCs avoid the “trough of disillusionment?”

My answer is:

1. Technological innovations aimed primarily at increasing productivity and efficiency in schooling have largely ignored teacher knowledge and expertise.

2. High-tech innovators seldom ask the questions teachers ask about a new classroom technology.

3. Innovators have cared little about whether their new technology can be integrated into teachers’ routines because their priorities are to transform teaching and learning, increase student productivity, and keep costs low.

A backward glance to lessons drawn from previous technological innovations, then, might help start-up entrepreneurs from being “necessarily naive” about MOOCs or the next new thing for K-12 classrooms. Will that happen? I doubt it.

 

12 Comments

Filed under school reform policies, technology use

Troubled Youth, Troubled Learning (Dave Reid)

Dave Reid is a high school mathematics teacher in his third year of teaching.   He received his MA in Education and credential in secondary mathematics and physics from Stanford University in 2011.  Dave spent a quarter of a century in high-tech primarily in the wireless and Global Positioning System (GPS) industries.  He earned a BS degree in electrical engineering from George Mason University, and an MBA in finance and marketing from Santa Clara University.  He also attended the U.S. Military Academy at West Point. He blogs as Mr. Math Teacher and tweets as @mathequality.

While the title for this post does not always ring true, in my few years teaching at Title I schools, it often reflects reality.  In fact, rarely does a day go by where no student disrupts the classroom learning environment for one reason or another.  As a fifty-something, I knew this going into teaching; what I did not know was how deleterious these disruptions are to continuity, sanity, and in the limit: opportunity, for my students, not me.  As someone in the classroom every day, hoping above all hope that my students can break out of their behavioral binds, it challenges my every fiber of existence to keep the class focused on our learning objective(s) for the day.

Troubled youth make for troubled learning, not only for themselves, but also for everyone in the classroom.  It is a huge force multiplier of the negative type.  In spite of what is heralded as the balm for these troubles, compassion, empathy, and other soft moves are frequently insufficient to overcome years of ingrained indifference, frustration, anger, resentment, or a host of other emotions, feelings, or attitudes that have overtaken an adolescent overwhelmed by his or her circumstances.  The older the youth, the more deeply embedded the issue or issues.  Now, extend these to one or more adolescents in a classroom, and you get a snapshot of teaching in a Title I school.

A few days ago, for instance, I taught three block periods: two of which are split into two sections apiece of algebra 1 and remedial mathematics, and one AP Calculus section.  The split sections are my attempt to support students who do not possess the arithmetic skill or understanding needed to succeed in algebra.  Fortunately, my administration and the district office support me in this effort.

The AP Calculus students are rarely “egregiously” troublesome, aside from the fact that they have yet to realize that frequent side conversations among the eight groups of four students each frequently distracts others.  At times, when teaching these students, it feels as if I am an onstage performer at a dinner theater with the audience commenting back and forth to each other about their meal, the show, or what not.  Periodically, I tell them that the classroom is not their living room, or a movie theater, where they freely watch or chat as they see fit during “the show.”  They seem a bit startled when I make them aware of their behavior, which puzzles me even more; it is as if I am the first and only teacher to ask them to consider their impact on a classroom.  Notwithstanding their surprise, I persist, as I do not believe college professors will tolerate their behavior any more than I do, for the majority of my calculus students are college bound this fall.

Yet, this is not a post about my privileged students, who make up most of my calculus students.  For they, mostly, are buffered, or far removed, from the intense psychosocial trauma faced by many low-income families.  Simply put, they live free from most of the burdens of poverty.  Burdens, which manifest themselves in low-income families, that inhibit attaining outcomes at the same level as those more privileged for the same level of effort.

My most challenged students, behaviorally and academically, frequent my algebra sections.  Their presence cannot be missed: whether visually or aurally.  While it only takes one student to derail the trajectory of a class, it is a rare day, indeed, when only one student in a class acts to call attention to themselves.  The duration, intensity, and frequency of the derailments vary based on the class composition.

In the face of these ever-present disruptions, I have to: keep students’ attention focused on moving forward with their learning; address the momentary outburst and its subsequent ripples throughout the classroom; all the while doing my best to stay passionate, motivated, and encouraging without having a mental breakdown.  I say that somewhat tongue in cheek.  However, it is not too far from reality.  Whoever mentioned that a teacher has nearly as demanding a job as an air traffic controller was pretty close to the truth.

Which brings me to the student who inspired this post.  John rarely participates positively in class. He seems to possess a boundless ability to draw negative attention to himself throughout a class period. He failed first semester and is on track to do the same this semester. I hope with all of my heart that he wakes up soon and understands how important it is to his future that he pay attention in class, attempt some of his homework, and learn as much as is humanly possible, for he is quite intelligent in spite of what he may believe.

John reminds me of how my younger brother, now deceased, might have been in school.  My brother was often truant.  He ran with the wrong crowd, experimented with things I never knew existed at his age, and dropped out of high school shortly after starting.  My brother may have been one of the silent ones, the student who attempts to disappear among the thirty or so classmates.  He might have giggled frequently chatting away with his classmates.  Regardless, he did not learn.  He missed out on that opportunity, as he was deeply troubled.  I will not go into details except to say that his burdens were too much for him.  They may have been too much for his teacher, if they manifested themselves while in school: I simply do not know.  What I do know is that I became a teacher, in part, to help those like my younger brother, of whom this one student reminds me.  .

I will not hold my breath for John.  I will encourage him as often as possible, in between addressing his behavioral shortcomings, for they do impact the class.  His mother is at her wits end and unsure what to do about him.  I believe my parents felt similarly some thirty plus years ago.  Life is amazingly complex.  Teaching is crazy hard.  It drains me nearly every day.  Yet, there is rarely a day I leave home headed to my classroom not eager to teach. Yes, some troubled youth await me; they are whom I most hope to help.  Yet, I only can do my part to work toward keeping them on a path to graduate; they need to do their part as well.  Time will only tell.

26 Comments

Filed under how teachers teach