The “Little Soldiers” of Chinese Education

The second day of school, Rainey came home with a story. Four times he found egg in his mouth. He hadn’t placed it there himself; instead, his most hated food made its way past his teeth by the hand of the fearsome Teacher Chen.

‘ She put it there,’ Rainey told me, mouth wide, finger pointing inside. Then what happened,? I asked.

“I cried and spit it out.”

Then what. She did it again,” Rainey said….”I cried and spit it out again.”

Rainey is three years old and enrolled in one of the best Shanghai kindergartens in the city. Lenora Chu, Mom, journalist and author of Little Soldiers: An American Boy, A Chinese School, and the Global Race to Achieve,(2017) from which this incident is taken, decides to meet with Teacher Chen to discuss force-feeding egg to her son. Chu describes the conference with Teacher Chen:

“We don’t use such methods of force in America,” I blurted in Mandarin, my son clutching my hand. (I was born and raised in America but grew up speaking Chinese at home.)

“Oh? How do you do it?” Teacher Chen challenged.

“We explain that egg eating is good for them, that the nutrients help build strong bones and teeth and helps with eyesight,” I said, trying to sound authoritative. “We motivate them to choose…we trust them with the decision.”

“Does it work?” Teacher Chen said….

“Well, not always,” I admitted.

Chen nodded. “Rainey needs to eat eggs. We think eggs are good nutrition, and all young children must eat them….”

Later, Teacher Chen pulled me aside for a lecture. “In front of the children, you should say, ‘Teacher is right, and Mom will do things the same way,’ OK?’ ” (pp.22-24, 27.)

Through the seven years that Rainey was in public Chinese schools, Chu uses his experiences in kindergarten and public primary school grades as a template for describing and analyzing Chinese education in Shanghai and the rest of the nation over the past decade. And it is a fascinating, eye-opening journey that she captures in an easily read, clear, description of children, schooling, and a top-down, state-directed system deeply anchored in the Confucian tradition of teaching and learning while exerting strong political control of what and how teachers teach.

Every country’s system of schooling mirrors the basic values in the culture that both the state and parents want to see in the behavior of, and school outcomes for their sons and daughters. Where in the U.S., a core value is the individual’s growth and development and belonging to a community comes second, in China, the collective comes first. And so does obeying the teacher.

The 28 kindergartners in Rainey’s class learned this song the second day of school:

I am a good baby

Little hands always in place

Little feet refined

Little ears listening well

Little eyes looking at the teacher

Before I speak, always raising my hand (p.64)

The principal granted permission to Chu to observe her son’s class. With three year-olds, incidents invariably occur. She describes what she saw occur during a lesson. Teacher Wang had difficulties with a little boy a head taller than the other children and filled with inexhaustible energy. For observer Chu, it was easy for her to remember Wang Wu Ze because the two teachers had yelled his name repeatedly over the first week of school:

Wang Wu Ze, sit down! Wang Wu Ze put your two feet side by side! Wang Wu Ze, what is wrong with you? Do you want your mommy to come and get you today?

One day, while Teacher … was talking, three-year old Wang Wu Ze left his seat and wandered over to a few toys in a corner . Chu writes that the teacher “lost her temper.” Teacher Wang said: “Wang Wu Ze, you don’t get a chair. YOU WILL STAND!” Chu describes the teacher moving quickly to where the boy was standing and “swatting his chair away. It fell over, clattered against the floor a few times, then lay still….” The boy looked at the “toppled chair and tears came to his eyes….” He went to the teacher and threw his arms around her waist and she said: “Bu bao–I won’t hold you…. Do you want a chair now?”

“Yes, yes, I want a chair.“

“Then you sit in it,” Wang said. “If you don’t sit in it, I won’t give it to you. And your mom won’t come get you after school.” (pp. 66-67)

Conformity to the group, obedience to the teacher, and constant attention to mastering content, skills, and behavior begin early in both how and what teachers teach, according to Chu.

Consider the curriculum standards, the Ministry of Education sets for learning to read Mandarin.

Chu reports that:

“First- and second-graders should recognize 1,600 characters and write 800 of them from memory. By fourth grade, the level is 2,500 characters, and by the sixth grade it’s 3,000 characters and writing almost as many….Full literacy requires an astonishing 3,500 frequently used characters to be committed to memory, according to the Chinese curriculum standards for full-time compulsory education.” (p. 86)

And competition within and across classes is both fierce and public. Testing is constant and student-by-student scores on tests and performance of tasks are publicly displayed. Chu describes the large bulletin board outside Rainey’s kindergarten classroom.

Big Board might post teachers’ assessment of each child, a report card displayed for all to see. Who clocked in timely arrivals at school? Which child greeted the teacher with a smile? Who finished every lunchtime grain of rice? Star stickers and happy faces were pasted next to the name of each child who’d made the grade….As the months passed into the first year of school, Big Board began to display information that directly compared performance and ability…. With each presentation, parents gathered eagerly. and I could always tell when Big Board posted new information by the number of bodies gathered around, heads bobbing with anticipation.

The following year, the Big Board would display prowess at recorder play [a small woodwind musical instrument] for all to see:

The ring finger of Student No. 20 is not stable

Student No. 30 doesn;t cover the old hole while changing to a new one.

Student No. 16 doesn’t blow out enough air.

Student No. 3 doesn’t cover the holes properly.

Chu says that Besides Rainey’s number, No. 27, the teacher had scrawled the same punishing diagnosis as that for No. 8

Doesn’t follow rhythm. (pp. 98-99)

Chu visited schools elsewhere in China including rural schools, and describes their classrooms. She speaks with education experts at universities and Ministry of Education officials. Yet China has over a quarter-billion students (yes, over 260 million) taught by over 15 million teachers housed in more than a half-million schools (2014). The numbers stagger the minds of non-Chinese educators. *

Obviously, Chu can only include so much in a book that focuses upon her son, two Shanghai teenagers in secondary schools, visits to a handful of schools, and interviews with various teachers and educators. Fortunately, Chu provides the necessary context for the Shanghai and rural schools that she visits by describing the state-directed system of schooling

As a centralized state where policy is controlled by a Ministry of Education imbued with Communist doctrine, top down curricular and instructional mandates provide a constant flow of regulations to each province’s schools. Ministry officials are well aware of the internal criticisms of rote memorization, disciplinary norms, and inequalities between rural and urban schools insofar as available resources, teacher experiences, and difficulties in getting teachers to alter their daily lessons. Top officialas have looked to other countries, including the U.S. for reforms to improve Chinese education.

Part 2 will take up some of these reforms as Chu identified them and difficulties of implementation.

____________________

*OECD, “Education in China: A Snapshot, 2016”, p. 9

Does Classroom Use of Computers Cause Gains in Students’ Academic Achievement?

A decade ago when a surge of interest in placing computers in classrooms was joined to available public and private money, U.S. schools went on a rampage of buying and distributing devices to students and teachers. At that time I wrote a post about a puzzling fact that was overlooked  or simply ignored:  With increased access to new technologies, there is little reliable and valid evidence showing that these technology investments have yielded gains in student achievement.

That was then and this is now. Amid a pandemic-induced shift of face-to-face lessons to online instruction, the question is irrelevant. The need is that every single public school child must have a device to use for remote instruction.

While there was certainly a digital divide a decade ago even amidst the splurge spending for devices, school closings due to the pandemic have pushed districts to triple-time distribution of devices and Internet connections to families lacking either or both.  Yet, the lack of solid evidence that student computer use–especially now when lessons are online–is strongly associated with increased test scores and other academic outcomes remains puzzling. So I updated that post and publish it now.

After decades of school and classroom use of new technologies, some facts have emerged that puzzle me.

Here’s one.

Since the early 1980s, the federal government, states, and districts—not to mention philanthropists—have invested billions of dollars in wiring schools, buying and deploying machines, and preparing teachers and students to use high tech devices. Nearly all teachers now have access to one or more computers at school. As for students, the number of students per computer across the U.S. has gone from 125 per computer in 1983 to 4 per computer in 2006. Teacher and student access to computers has increased even more in the past decade with thousands of schools issuing computers to each and every student and teacher.

With increased access to new technologies, there is little reliable and valid evidence showing that these technology investments have yielded gains in student achievement.

Why?

One answer is simply that access to machines does not necessarily lead to teachers regularly using high-tech devices in daily lessons. Consider that after nearly 30 years of access to computers in the U.S., based on national surveys and research studies of schools, over 60 percent of teachers are regular users, that is, using computers for instruction one or more times a week (see here). These teachers use interactive white boards, laptops, and hand-held devices to have students do Internet searches, turn in typed rather than hand-written homework, take notes on lectures, watch videos, and other familiar classroom activities. A small sub-set of these teachers, however, do use electronic devices weekly, even daily in far more creative and imaginative ways inside and outside classrooms with their students.

But the majority of teachers, most of whom–paradoxically–use their home computers a few hours each night, are either occasional or non-users in integrating available machines into their daily lessons.

So one explanation for the first puzzling fact is the flawed assumption that deploying computers to teachers and students will lead to teachers regularly using high-tech devices for instruction. Note that without regular use by teachers, establishing either a correlational or causal link between computers and, say, student test scores, is impossible.

Another explanation for the puzzle of so little linkage between computers and student achievement examines how researchers go about studying the connections between technology and student outcomes.

Many researchers fail to consider that the common designs and methodologies they use to determine linkages between classroom technology use and student achievement cannot capture the inherent complexity and unpredictability of teaching and learning. So researchers use shortcuts to get around that complexity and unpredictability.

I need to unpack the previous sentence. Consider that teaching students involves many factors relating to who the teacher is, what content and skills are taught, and what activities and tasks occur while teaching. Also consider student factors: who they are, what experiences, motivations and interests they bring to the classroom, and what they do during lessons. Then consider the school itself, its organization, culture, and its neighborhood. Finally consider the district, its resources, leadership, and culture of learning or non-learning that it cultivates. All of these interacting factors, sometimes unpredictably, affect classroom teaching and learning.

Yet look at the majority of research designs and methods used to determine the effects of teachers using computers with student. Most common are surveys of teachers and students who report their perceptions of classroom use supplemented by researchers’ descriptions of practices, and interviews with teachers and students. Some researchers set up comparison groups of classes that use computers to study a topic with classes not using computers studying the same topic. Then the classes using and not using computers are pre- and post-tested.

Both research designs have serious defects. Short of establishing an experimental and control design with students and teachers randomly assigned to each group, it is nearly impossible to establish a causal linkage between the use of high-tech devices and student achievement. Such experimental or quasi-experimental designs are uncommon and usually too expensive to mount.

Because surveys, class-comparisons, or studies of small groups of students using computers are less expensive in dollars and labor, thousands of studies have been done since the introduction of desktop computers into schools in the early 1980s. Many show minute gains or “no significant difference” in test scores from student use of computers. The results, however, are correlations—associations between presence of computers and gains in test scores, not evidence that student use of the machines caused a rise in test scores (see here).

Here, then, are two ways to make sense of the puzzling fact over the paltry results in student outcomes of so much investment in high-tech devices and so little return on those dollars.

When one shifts to the evidence of online instruction being associated with academic achievement gains, a similar story unfolds.

A brief look at the many K-12 and higher education studies that have sought an answer to the question of the effectiveness of online instruction in producing student gains in achievement may help readers.

Scores of studies of outcomes for online instruction have been contested because most have had serious design and methodological flaws. Moreover, many of these studies lumped together full-time virtual schools, hybrids, and online courses, And the results have been underwhelming.  That is where heartburn enters the picture (see here).

Even when researchers over the past few decades have performed meta-analyses of a smaller number of studies that have met higher standards of quality they found that virtual instruction in its various modes, at best, is equivalent to regular face-to-face classroom instruction. At worst, some studies showed less achievement gains than traditional teaching. And keep in mind that these meta-analyses were of studies where online instruction occurred in mostly math, reading, and science courses—not other academic subjects. Nor in areas of great concern such as kindergarten and primary grades, the arts, and social and emotional learning. The overall picture is considerably dimmer than promoters of full- and part-time virtual schooling have promised or leaders had expected. *

So with such little evidentiary support for huge expenditures of school budget funds for hardware, software, maintenance, and professional development, why keep buying the stuff?

There was one explanation I omitted when I wrote this post in 2010 because it did not register with me then but over the following years has emerged as a powerful reason for more classroom use.

The rationale for giving devices to every student and teacher has shifted from linkage to student outcomes to the simple fact that all standardized testing will be online.  With the Covid-19 pandemic, insuring that every student had access to a device and the Internet took precedence over teacher’s decisions to use or not use devices for their lessons. Until in-person schooling resumes for most U.S. students, all lessons will continue to be online–for better or worse.

____________________

*Cathy Cavanaugh, et. al. “The Effects of Distance Education on K–12 Student Outcomes: A Meta-Analysis.” 2004 Naperville, IL: Learning Point Associates ; Rosina Smith, et. al.  “A Synthesis of New Research on K-12 Online Learning”. 2005, Naperville, IL: Learning Point Associates; Barbara Means, et. al., “Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies,” (U.S. Department of Education, Office of Planning, Evaluation, and Policy Development, 2010).

When Algorithms Give Real Students Imaginary Grades (Meredith Broussard)

“Meredith Broussard (@merbroussard) is a data journalism professor at New York University and the author of “Artificial Unintelligence: How Computers Misunderstand the World.” She is working on a book about race and technology.” This op-ed piece appeared in the New York Times on September 9, 2020.

Isabel Castañeda’s first words were in Spanish. She spends every summer with relatives in Mexico. She speaks Spanish with her family at home. When her school, Westminster High in Colorado, closed for the pandemic in March, her Spanish literature class had just finished analyzing an entire novel in translation, Albert Camus’s “The Plague.”She got a 5 out of 5 on her Advanced Placement Spanish exam last year, following two straight years of A+ grades in Spanish class.

And yet, she failed her International Baccalaureate Spanish exam this year.

When she got her final results, Ms. Castañeda was shocked. “Everybody believed that I was going to score very high,” she told me. “Then, the scores came back and I didn’t even score a passing grade. I scored well below passing.”

How did this happen? An algorithm assigned a grade to Ms. Castañeda and 160,000 other students. The International Baccalaureate — a global program that awards a prestigious diploma to students in addition to the one they receive from their high schools — canceled its usual in-person final exams because of the pandemic. Instead, it used an algorithm to “predict” students’ grades, based on an array of student information, including teacher-estimated grades and past performance by students in each school.

Ms. Castañeda wasnot alone in receiving a surprising failing grade — tens of thousands of International Baccalaureate students protested their computer-assigned grades online and in person. High-achieving, low-income students were hit particularly hard: many took the exams expecting to earn college credit with their scores and save thousands of dollars on tuition.

Nor was the International Baccalaureate the only organization to use a computer program to assign students grades amid the pandemic. The United Kingdom’s in-person A-level exams, which help determine which universities students go to, were also canceled and replaced with grades-by-algorithm. Students who were denied university entrance because of these imaginary grades took to the streets, chanting anti-algorithm slogans. Only after an uproar did the government change course, though many students were left in limbo without university admission.

The lesson from these debacles is clear: Algorithms should not be used to assign student grades. And we should think much more critically about algorithmic decision-making overall, especially in education. The pandemic makes it tempting to imagine that social institutions like school can be replaced by technological solutions. They can’t.

The bureaucrats who decided to use a computer to assign grades are guilty of a bias I call technochauvinism: the idea that technological solutions are superior. It’s usually accompanied by equally bogus notions like, “Computers make neutral decisions” or, “Computers are objective because their decisions are based on math.”

Computers are excellent at doing math, but education is not math — it’s a social system. And algorithmic systems repeatedly fail at making social decisions. Algorithms can’t monitor or detect hate speech, they can’t replace social workers in public assistance programs, they can’t predict crime, they can’t determine which job applicants are more suited than others, they can’t do effective facial recognition, and they can’t grade essays or replace teachers.

In the case of the International Baccalaureate program, grades could have been assigned based on the sample materials that students had already submitted by the time schools shut down. Instead, the organization decided to use an algorithm, which probably seemed like it would be cheaper and easier.

The process worked like this: Data scientists took student information and fed it into a computer. The computer then constructed a model that outputted individual student grades, which International Baccalaureate claimed the students would have gotten if they had taken the standardized tests that didn’t happen. It’s a legitimate data science method, similar to the methods that predict which Netflix series you’ll want to watch next or which deodorant you’re likely to order from Amazon.

The problem is, data science stinks at making predictions that are ethical or fair. In education, racial and class bias is baked into the system — and an algorithm will only amplify those biases.

Crude generalizations work for Netflix predictions because the stakes are low. If the Netflix algorithm suggests a show and I don’t like it, I ignore it and move on with my day. In education, the stakes are much higher. A transcript follows you for years; when I was 25 and well out of college, I applied for a job that asked for my SAT scores.

In Ms. Castañeda’s case, her failing grade was most likely due in part to the fact that historical performance data for her school was one of the inputs to the algorithm. The computer assumed that the students at Westminster, who are mostly low-income students of color, would continue to do poorly.

“Everyone I know got downgraded one or two levels,” Ms. Castañeda told me. “It’s not fair that our scores were brought down because of our school’s history. It’s unfair to punish students for where they live.”

Another input to the algorithm was teacher prediction of the students’ grades. Teachers tend to have lower expectations for Black and Brown students compared to white students; this bias is well known in the education community and ignored in the data science community. Thus a very human bias prevailed in the computational system.

International Baccalaureate and Ofqual, the agency that administers Britain’s A-level exams, have reluctantly realized that algorithmic grades were a mistake. Since the outcry over algorithm-assigned grades, both organizations have been sued. Many students, including Ms. Castañeda, ended up receiving new, higher imaginary grades.

Roger Taylor, chair of the Ofqual board, apologized in front of a House of Commons educational oversight committee this week. “We are sorry for what happened this summer,” he said. “With hindsight it appears unlikely that we could ever have delivered this policy successfully.”

As we stare down the fall semester online, there are going to be infinitely many technochauvinist calls to transform online education and use algorithmic tools that promise to evaluate individual student learning. Resist these calls.

With Pass-Fail, What’s the Point of Grades? (Jack Schneider)

This opinion column appeared in New York Times, June 25, 2020

Jack Schneider (@Edu_Historian), an assistant professor of leadership in education at the University of Massachusetts Lowell, is the director of research for the Massachusetts Consortium for Innovative Education Assessment.

In the wake of the novel coronavirus pandemic, countless colleges and universities shifted from to A-F grades to a pass/fail system. As officials at Wellesley College explained, the general aim in doing so is to “support one another without being required to make judgments.”

Many K-12 school districts have done the same. From Palo Alto, Calif., to Wake County, N.C., local officials have concluded that now is not the time for grades. As teachers in Wisconsin’s Madison Metropolitan School District declared, “We cannot grade with equity when students’ experiences learning at home will be so varied.” And it’s not yet clear that most schools that have made this switch will fully return to letter grades in the fall.

But not everyone is happy with this outcome.

Some parents and activists are anxious that, without grades, students won’t receive adequate feedback on their work. Others worry that altering or eliminating the traditional grading scale will undermine student motivation and reward slacking off. As one Oregon parent pointedly asked in one of many online petitions pushing for the reinstatement of letter grades, “How do I explain to my child that has great grades that she should keep working hard when anything that is D- and above will still ‘pass’? This is ridiculous.” A similar but separate concern, expressed by ambitious students and their parents, is that without letter grades students will be at a disadvantage when competing for scholarships, college admission and merit aid.

The logistical calamities presented by the coronavirus have suddenly, and forcefully, surfaced an underlying problem frequently ignored before the crisis: A-F grades serve several different purposes, and those purposes are too often in conflict with one another. Americans may come to recognize by the end of this schooling crisis that we would all be better off without letter grades.

The original aim of grading, which can be traced back several centuries to English universities like Oxford and Cambridge, was to motivate students. As educators found, students tended to work harder if there was a brass ring for them to reach. This fact became more important in the latter half of the 19th century, as an increasing number of states made schooling compulsory.

With a new influx of reluctant pupils, many K-12 teachers were faced with a challenge even greater than keeping the average students focused: maintaining the attention of students who didn’t want to be there at all. Grades, then, also became a mechanism for coercion — rewards, but also punishments, with bad grades meant to serve as a socializing source of shame.

Grades as we know them now have yet another origin too, rooted in efforts to communicate with students and their families. Feedback, as any educator knows, is essential to learning. But as class sizes grew larger in the 19th century, American teachers were increasingly pressed for time. Looking for a shortcut, many schools developed new systems for providing feedback to students. The boldest of these physically rearranged students in the classroom — hence the phrase “head of the class.” What endured, however, was the “report card,” which used pre-identified codes — like numbers or letter grades — to streamline the process of evaluation.

Unlike student seating charts, report cards could be sent home to parents, strengthening communication with families. And eventually, policy leaders realized that if grades could relate something about student learning to parents and families, they could also communicate info more broadly — to other schools, to state offices of education and to employers.

Standardized report cards, in essence, could create a national market for student knowledge and skill, in the same way that letter grades for products like grain had created commodities markets. Just as the quality of Grade A beef could be understood without firsthand knowledge, so could the quality of an A student. With the evolution of the transcript — the permanent record for storing grades — student performance could be communicated across both time and space.

But there was still a lack of uniformity. Unlike many countries where a central ministry of education directs policy across the nation’s schools, the U.S. system has always been characterized by decentralization. Subject to local control, many schools and districts used 1-100 scoring systems; others used letters; some even relied on 0-4 systems. Eventually, however, the pressure for standardization from elites led to a grand merger: A 1-100 score that could be converted into an A-F grade, which, in turn, was convertible again into a grade-point average.

The merger was highly useful for these domestic policy elites, dealing with a rapidly growing nation, who then used the new regimen to connect America’s fragmented educational system. Schools, colleges and employers could nominally work together without actually changing their independent models.

By the early 20th century, grades as we generally see them now had become a core feature of American education. But as any programmer can tell you, tasking a single technology with multiple distinct roles is a bad idea. Letter grades do several different things, none of them well, and the result undermines student learning.

Consider the fact that the permanent nature of grades makes them an incredibly high-stakes affair for students. This has a serious impact on the degree to which teachers can use grades to effectively communicate student progress. Think of how a low grade, intended to convey that a young person doesn’t yet understand a concept, will instead read to the student as an act of cruelty — an attempt to ruin her future. And the student wouldn’t necessarily be wrong to see it that way; transcripts in a self-proclaimed meritocratic world mean that grades, like diamonds, are forever.

Similarly, using letter grades as a currency across agencies and institutions has, in reality, negatively distorted student motivation for generations. Regardless of their inclination to learn, many students strive first and foremost to get good grades. This was even the case in 1918, when American economist and sociologist Thorsten Veblen observed that the pursuit of grades “progressively sterilizes all personal initiative and ambition that comes within its sweep.” And a century later, it remains true, as students scramble for prized, résumé building credentials at the expense of their own intellectual curiosity.

Americans have long been aware of the problems with the current model. Grade inflation is an epidemic, particularly at elite schools. And “grade-grubbing,” the pestering of teachers to change scores, is a scourge too. Parents complain about stressed out students haunted by the prospect of an imperfect “life-ruining” transcript. And teachers bemoan the endless grind of grading.

Yet despite these obvious problems, grades are deeply embedded into the culture and function of American education: They are used for state graduation requirements, military eligibility, community college transfers, and scholarship determinations; and they are one of the chief mechanisms for linking America’s 100,000 schools with over 5,000 colleges and universities.

In short, this grading conundrum won’t be easy to solve. But the inherent flaws of A-F grading have never been clearer. And, because of that, several sensible reform ideas — unreasonably ambitious in normal times — may offer a path forward.

One smart proposal involves the use of student portfolios. Rather than reducing everything a student has learned to a single score or letter symbol, schools and colleges might ask students to assemble evidence of what they know and can do. Models of this can be found in progressive networks of public and private schools, as well as in programs like International Baccalaureate and the Advanced Placement program. Portfolios are by no means a silver bullet, but they have a number of important strengths: emphasizing the substance of learning, encouraging revision and acknowledging the different paces at which students reach proficiency. Perhaps most importantly, they motivate students to improve their work and not merely their grades.This, of course, is only a brief sketch — a map of future prospects rather than a concrete plan. And it would require fairly unprecedented coordination across different organizations and government agencies. But the road to reform always begins with an awakening to possibility.

Our present use of grades is a matter of historical accident, not design. The result is that grades fail to advance the multiple purposes they ostensibly serve.

Pass/Fail grading — the stopgap that many have turned to in the wake of the pandemic — is not a long-term solution. The problem can only be addressed at its root. Shaken from our complacency by a crisis, perhaps we can begin the conversation about what comes next.

Worried About Those “Big” Losses on School Tests Because Of Extended Stays At Home? They May Not Even Happen, And If They Do, They May Not Matter Much At All! (David Berliner)

David Berliner is Regents Professor Emeritus at the Mary Fulton Teachers College, Arizona State University. This appeared on Diane Ravitch’s blog May 28, 2020

Although my mother passed away many years ago, I need now to make a public confession about a crime she committed year in and year out. When I was young, she prevented me from obtaining one year of public schooling. Surely that must be a crime!

Let me explain. Every year my mother took me out of school for three full weeks following the Memorial Day weekend. Thus, every single year, from K through 9th grade, I was absent from school for 3 weeks. Over time I lost about 30 weeks of schooling. With tonsil removal, recurring Mastoiditis, broken bones, and more than the average ordinary childhood illnesses, I missed a good deal of elementary schooling.
How did missing that much schooling hurt me? Not at all! 

First, I must explain why my mother would break the law. In part it was to get me out of New York City as the polio epidemic hit U.S. cities from June through the summer months. For each of those summers, my family rented one room for the whole family in a rooming house filled with working class families at a beach called Rockaway. It was outside the urban area, but actually still within NYC limits.

I spent the time swimming every day, playing ball and pinochle with friends, and reading. And then, I read some more. Believe it or not, for kids like me, leaving school probably enhanced my growth! I was loved, I had great adventures, I conversed with adults in the rooming house, I saw many movies, I read classic comics, and even some “real” literature. I read series after series written for young people: Don Sturdy, Tom Swift, the Hardy Boys, as well as books by Robert Louis Stevenson and Alexander Dumas.

So now, with so many children out of school, and based on all the time I supposedly lost, I will make a prediction: every child who likes to read, every child with an interest in building computers or in building model bridges, planes, skyscrapers, autos, or anything else complex, or who plays a lot of “Fortnite,” or “Minecraft,” or plays non-computer but highly complex games such as “Magic,” or “Ticket to Ride,” or “Codenames” will not lose anything measurable by staying home. If children are cared for emotionally, have interesting stuff to play with, and read stories that engage them, I predict no deficiencies in school learning will be detectable six to nine months down the road.
It is the kids, rich or poor, without the magic ingredients of love and safety in their family, books to engage them, and interesting mind-engaging games to play, who may lose a few points on the tests we use to measure school learning. There are many of those kinds of children in the nation, and it is sad to contemplate that.

But then, what if they do lose a few points on the achievement tests currently in use in our nation and in each of our states? None of those tests predict with enough confidence much about the future life those kids will live. That is because it is not just the grades that kids get in school, nor their scores on tests of school knowledge, that predict success in college and in life. Soft skills, which develop as well during their hiatus from school as they do when they are in school, are excellent predictors of a child’s future success in life.

Really? Deke and Haimson (2006), working for Mathmatica, the highly respected social science research organization, studied the relationship between academic competence and some “soft” skills on some of the important outcomes in life after high school. They used high school math test scores as a proxy for academic competency, since math scores typically correlate well with most other academic indices. The soft skills they examined were a composite score from high school data that described each students’ work habits, measurement of sports related competence, a pro-social measure, a measure of leadership, and a measure of locus of control.

The researchers’ question, just as is every teacher’s and school counselor’s question, was this: If I worked on improving one of these academic or soft skills, which would give that student the biggest bang for the buck as they move on with their lives? 

Let me quote their results (emphasis by me)
Increasing math test scores had the largest effect on earnings for a plurality of the students, but most students benefited more from improving one of the nonacademic competencies. For example, with respect to earnings eight years after high school, increasing math test scores would have been most effective for just 33 percent of students, but 67 percent would have benefited more from improving a nonacademic competency. Many students would have secured the largest earnings benefit from improvements in locus of control (taking personal responsibility) (30 percent) and sports-related competencies (20 percent). Similarly, for most students, improving one of the nonacademic competencies would have had a larger effect than better math scores on their chances of enrolling in and completing a postsecondary program.

​This was not new. Almost 50 years ago, Bowles and Gintis (1976), on the political left, pointed out that an individual’s noncognitive behaviors were perhaps more important than their cognitive skills in determining the kinds of outcomes the middle and upper middle classes expect from their children. Shortly after Bowles and Gintis’s treatise, Jencks and his colleagues (1979), closer to the political right, found little evidence that cognitive skills, such as those taught in school, played a big role in occupational success.

Employment usually depends on certificates or licenses—a high school degree, an Associate’s degree, a 4-year college degree or perhaps an advanced degree. Social class certainly affects those achievements. But Jenks and his colleagues also found that industriousness, leadership, and good study habits in high school were positively associated with higher occupational attainment and earnings, even after controlling for social class. It’s not all about grades, test scores, and social class background: Soft skills matter a lot!

Lleras (2008), 10 years after she studied a group of 10th grade students, found that those students with better social skills, work habits, and who also participated in extracurricular activities in high school had higher educational attainment and earnings, even after controlling for cognitive skills! Student work habits and conscientiousness were positively related to educational attainment and this in turn, results in higher earnings. 

It is pretty simple: students who have better work habits have higher earnings in the labor market because they are able to complete more years of schooling and their bosses like them. In addition, Lleras’s study and others point to the persistent importance of motivation in predicting earnings, even after taking into account education. The Lleras study supports the conclusions reached by Jencks and his colleagues (1979), that noncognitive behaviors of secondary students were as important as cognitive skills in predicting later earnings.
So, what shall we make of all this? I think poor and wealthy parents, educated and uneducated parents, immigrant or native-born parents, all have the skills to help their children succeed in life. They just need to worry less about their child’s test scores and more about promoting reading and stimulating their children’s minds through interesting games – something more than killing monsters and bad guys. Parents who promote hobbies and building projects are doing the right thing. So are parents who have their kids tell them what they learned from watching a PBS nature special or from watching a video tour of a museum. Parents also do the right thing when they ask, after their child helps a neighbor, how the doing of kind acts makes their child feel. This is the “stuff” in early life that influences a child’s success later in life even more powerfully than do their test scores.

So, repeat after me all you test concerned parents: non-academic skills are more powerful than academic skills in life outcomes. This is not to gainsay for a minute the power of instruction in literacy and numeracy at our schools, nor the need for history and science courses. Intelligent citizenship and the world of work require subject matter knowledge. But I hasten to remind us all that success in many areas of life is not going to depend on a few points lost on state tests that predict so little. If a child’s stay at home during this pandemic is met with love and a chance to do something interesting, I have little concern about that child’s, or our nation’s, future.

Bowles, S., & Gintis, H. (1976). Schooling in Capitalist America. New York: Basic Books.

Deke, J. & Haimson, J. (2006, September). Expanding beyond academics: Who benefits and how? Princeton NJ: Issue briefs #2, Mathematica Policy Research, Inc. Retrieved May 20, 2009 from: http://www.eric.ed.gov:80/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/28/09/9f.pdf
Matematicapolicyresearch Inc.

Lleras, C. (2008). Do skills and behaviors in high school matter? The contribution of noncognitive factors in explaining differences in educational attainment and earnings. Social Science Research, 37, 888–902.

Jencks, C., Bartlett, S., Corcoran, M., Crouse, J., Eaglesfield, D., Jackson, G., McCelland, K., Mueser, P., Olneck, M.,
Schwartz, J., Ward, S., and Williams, J. (1979). Who Gets Ahead?: The Determinants of Economic Success in America.
New York: Basic Books.

A School Experiment to Remember

Between 1933-1941, thirty high schools in the country and over 300 universities and colleges joined an experiment sponsored by the Progressive Education Association.

Called “The Eight Year Study,” each high school decided for itself what curricula, schedules, and class sizes would be. There were no college admission requirements or must-take tests. Old lesson plans were scrapped. One school sent classes into the West Virginia coal region to study unions. Science, history, art, and math were often combined in projects that students and teachers planned together.

Needless to say, there were stumbles also. A few principals blocked the experiment. Some school faculties divided into warring factions.

While there was much variation among the schools, there were also common elements. Many of the large public high schools (of the 30, fifteen were private) created small schools within the larger one. Principals increased the authority of teachers to design and steer the program; teachers crossed departmental boundaries and created a core curriculum (math/science and English/social studies), set aside three hours a day for teams to work with groups of students, and planned weekly units with students.

What happened to these students when they attended college? To find an answer, evaluators established 1,475 pairs of college students, each consisting of a graduate from an experimental school and one graduate of another high school matched as closely as possible as to age, sex, race, social class, and academic performance. They then compared their performance in college.

Evaluators found that graduates of the thirty schools earned a slightly higher grade average and more academic honors than those who attended regular high school. Furthermore, the “guinea pigs,” as they were called, were more precise in their thinking, displayed more ingenuity in meeting new situations, and demonstrated an active interest in national and world issues than their matched counterpart.

What these startling results showed over 80 years ago was that there was no one single best way of schooling teenagers. The fears that parents and taxpayers had about experimenting with high school courses, organization, and teaching proved hollow in “The Eight Year Study.”

The results of these studies appeared during World War II. The war effort swallowed up any further interest in experimenting with high school programs. Whatever the reasons, “The Eight Year Study” lapsed into the obscurity of scholarly footnotes. Later generations of reformers seldom inquired or cared about this large-scale, non-federally funded experiment that showed convincingly that schools, given the freedom to experiment, could produce graduates that not only did well academically in college but, far more important, displayed an active interest in civic affairs, were resourceful in handling new situations, and could think clearly.*

So what does this half-century old experiment say to us in the in the 21st century about Progressive school reform?

1.When engaged teachers, administrators, and students are given the freedom to experiment and the help to do it, they will come through.
2. There is no one best way of schooling youth.
3. Students can graduate high school who are academically prepared, involved in their communities, and thoughtful problem-solvers.
4. Standards of excellence that work in schools can be set and done locally by adults and students—not imposed from the top-down.

In 2020, federal and state decision-makers and policy elites drive school reform. They set standards, test students, and hold schools and students accountable for their performance. What “The Eight Year Study” (recall that it was sponsored by the Progressive Education Association) demonstrated nearly a century ago is that there are locals–districts, schools, and practitioners—who have the expertise and can be trusted to design and implement different ways to organize schools.

Am I suggesting that this is what all U.S. high schools should do? No, I am not. There are, for example, schools in largely poor urban and rural areas—both minority and white—that can use far more state and federal aid in supplying experienced teachers and coaches, reducing class size and expanding community services. And, yes, even the freedom to try different ways of organizing schools for better teaching and learning.  

If only those who govern and fund schools could learn the essential lesson that every parent with more than one child and every experienced teacher and principal has learned over the years:

*There is no one best way to learn.

*There is no one best way to teach.

*There is no one best way to organize schools.

Amen.

_____________________________

*Additional sources for the Eight Year Study are:

William Wraga, Democracy’s High School: The Comprehensive High School and Educational Reform in the United States. University Press of America. pp. 61–65.

Craig Kridel and Robert Bullough Jr. (2012). Stories of the Eight-Year Study: Reexamining Secondary Education in America. SUNY Press.

Wikipedia

“How The Other Half Learns”: A Review (Part 1)

Robert Pondiscio’s recent book about a New York City elementary school is an uncommon example of research and writing on school reform.

Why uncommon?

Few former teachers, journalists, and academic researchers have done what he did in spending a year at Bronx 1, part of the network of Success Academies in New York City that former city official Eva Moskowitz founded in 2006. Now a charter network of 47 schools in New York City enrolling 17,000 low-income children of color, Success Academies are both extolled and criticized (especially in the media). There is precious little middle ground when it comes to reformers, parents, teachers, and others when it comes to judging the network’s worth. Writing in 2014, a few years before Pondiscio became embedded in one Success Academy school, he wrote an op-ed in a New York newspaper asking: “Is Eva Moskowitz the Michael Jordan of Education Reform, or is she the Mark McGuire? (p. 10)”. One athlete, the finest of all basketball players in the 20th century and the other a disgraced steroid-filled home run hitter. He wasn’t sure. But two years later he wanted to find out.

Surprisingly, Moskowitz agreed to let Pondiscio to spend a year at Bronx 1 observing classes and teacher meetings, shadowing the principal and staff members, interviewing parents, teachers and administrators, meeting with children–“scholars” as they are called–in and out of school. He also attended teacher training sessions and staff development workshops. That Pondiscio was a former teacher in a Bronx low-income, low-scoring elementary school and affiliated with the Thomas B. Fordham Institute, an organization that boosts charter schools and whose leadership had praised her work in New York City may have helped in making her decision.

Which brings me to another reason for the book’s uniqueness. Except for Jay Mathews book on the Knowledge Is Power Program (KIPP) a decade earlier (Work Hard. Be Nice.: How Two Inspired Teachers Created the Most Promising Schools in America, Pondiscio is the only journalist and former teacher to examine in depth a charter network that prides itself on high test scores year after year outscoring affluent elementary schools in the state—a focus that drives anti-charter critics and student-centered enthusiasts to sucking their thumbs.

In the past 35 years in U.S. schools the unfurled umbrella of business-driven standards-based reform, testing and scores have been the center of attention in improving urban schools. The resulting literature on school reform has been dominated by pundits, policymakers, and passers-by who equate high test scores with teacher and school effectiveness. Test score gains have become the coin of the realm and schools blessed with those higher scores have become darlings of donors and policymakers. Success Academies have produced annual gains even outscoring those New York City schools with low poverty rates and mostly white enrollments.

This book, then, is an anomaly in its in-depth portrayal of children, staff, parents, and leadership in a neighborhood that by all accounts–including the nearby one in which Pondiscio once taught–should have been overwhelmed by its surroundings. Yet Bronx 1 was not. It excelled insofar as test scores. Although Pondiscio makes clear that test scores are a narrow measure of student learning, he gives readers his take on why the school did excel.

In praising the uncommonness of this book, I have not forgotten academic researchers who have gone into schools. They (including doctoral students) have indeed spent time in schools using both quantitative and qualitative methods to paint pictures of schools and classrooms at one point in time. And they have written about their before-and-after studies of experimental programs in schools and deeply detailed case studies–all using the outcome measure of standardized test scores.

Few researchers, however, who have written about schools and districts–the basic units of reform–have taught in similar settings and have actually spent at least a year in those classrooms, schools and districts observing, interviewing, and capturing incidents and details that make school cultures in those places dance before a reader’s eyes.* To do so takes experience of teaching in such schools and the skills of a writer to pull out the significant incidents and flesh out the main players in words that grab attention and stay fixed in the readers’ mind. Not an academic researcher but ex-teacher and journalist, Pondisco does exactly that. **

Still I wanted to see if other academic researchers, journalists, or others had written comparable volumes that would make How the Other Half Learns part of a tradition rather than being uncommon. So before I sat down at my computer to write this review I went through my library and pulled out the books that did what no pundit, policymaker, or passer-by could do. I do not claim that my library covers the entire literature of school reform in districts and schools yet in my selective collection of books I found a handful that met the above criteria of skillful writing, spending a year or so in the setting, and getting the story published. None of the writers, however, had teaching experience. Sure there are other studies that I missed. So be it. But these, I believe, are comparable to Pondiscio’s book.

#Former Washington Post journalist and later a university sociologist, Gerald Grant spent a year at a Syracuse High School working with teachers and students recording daily activities, interviewing teachers and students, and observing classrooms and meetings. He places the high school in a historical context, that is, going from a mostly white, privileged enrollment to a desegregated one with a substantial minority presence. The turmoil of the Vietnam War, court decisions expanding students’ rights and ending, shifted authority and the ways that students and teachers interacted. The World We Created at Hamilton High School was published in 1988.

#Linda Perlstein, another journalist at the Washington Post, published Tested: One American School Struggles To Make the Grade in 2007. She writes of her year spent at Tyler Heights Elementary School in Annapolis, (MD). Mostly African American children from low-income homes, the school scored well on state tests and had earned a reputation for academic excellence. Perlstein describes the principal, teachers, and students over the course of a year.

#Jay Mathews, longtime Washington Post columnist on education wrote about the two teacher founders of the KIPP schools in 2009 (see above). He observed classes, spent time in schools, and revealed to a general audience what he saw thereby challenging the prevailing myths that surrounded these particular charter schools.

#Finally there is a high school in San Francisco that let a journalist spend four years–yes, four years–to observe classes, interview staff and students, and meet with them inside and outside school. Kristina Rizga’s Mission High School (2015) unravels the puzzle of a high school with low test scores year after year and yet over four of five graduates get admitted to college. Challenging the existing concentration on test scores as a proper measure of school and student achievement, Rizga’s analysis provides answers to this disparity between low test scores and college going graduates.

These books are the ones that I found in my library. Readers might supply their own examples of embedded journalists and researchers, some with teaching experiences and some not, who have spent considerable time in classrooms, and with students and parents. Each of these books, as Pondiscio’s, has embedded the all-important contexts–local, state, and national–into their accounts. Even were my list incomplete, when one counts up such rich examinations of schools, they are but a thimbleful of the literature on urban school reform.

Parts 2 and 3 dig into How the Other Half Learns raising questions about the tilt that the author has toward parental choice for low-income and working class minority parents, a single curriculum for all students, and a culture that makes extraordinary demands upon both parents and children.

___________________________________

*One example (there are probably others) is researcher Louis Smith at Washington University in St. Louis who teamed up with William Geoffrey, a seventh grade teacher in a local school. Smith the outside observer recorded what happened every day for a semester in Geoffrey’s classroom. This micro-ethnography was published in 1968 as Complexities of an Urban Classroom: An Analysis toward a General Theory of Teaching.

**In this review, I omit first-hand accounts by teachers (e.g., Dangerous Minds, Freedom Writers) and principals (e.g., Lean on Me) because they narrowly describe one classroom or one school from only the teacher’s or administrator’s view. Observing and interviewing many teachers reveals the variation that exists in a school. Awareness of the school and district detailed contexts rarely appears in such books.

The New Stupid Replaces the Old Stupid (Rick Hess)

From an interview conducted in 2009 with Rick Hess, then Resident Scholar at The American Enterprise Institute. I have lightly abridged the interview. The original article upon which this interview is based is here.

Q: Rick, you recently published an article in Educational Leadership
arguing that the ways in which we rely on data to drive decisions in
schools has changed over time. Yet, you note that we have unfortunately only
succeeded in moving from the “old stupid” to the “new stupid.” What do you do
you mean by this?

A: A decade ago, it was only too easy to find education leaders who dismissed
student achievement data and systematic research as having only limited utility
when it came to improving schools. Today, we’ve come full circle. You can’t
spend a day at an education gathering without hearing excited claims about
“data-based decision making” and “research-based practice.” Yet these phrases
can too readily serve as convenient buzzwords that obscure more than they
clarify and that stand in for careful thought. There is too often an unfortunate
tendency to simply embrace glib solutions if they’re packaged as “data-driven.”
Today’s enthusiastic embrace of data has waltzed us directly from a petulant
resistance to performance measures to a reflexive reliance on a few simple
metrics–namely, graduation rates, expenditures, and grade three through eight
reading and math scores. The result has been a race from one troubling mindset
to another–from the “old stupid” to the “new stupid.”

Q: Can you give us an example of the “new stupid”?

A: Sure, here’s one. I was giving a presentation to a group of aspiring
superintendents. They were eager to make data-driven decisions and employ
research to serve kids. There wasn’t a shred of the old stupid in sight. I
started to grow concerned, however, when our conversation turned to value-added
assessment and teacher assignments. The group had recently read a research brief
highlighting the effect of teachers on achievement and the inequitable
distribution of teachers within districts. They were fired up and ready to put
this knowledge to use. One declared to me, to widespread agreement, “Day one,
we’re going to start identifying those high value-added teachers and moving them
to the schools that aren’t making AYP.” [AYP is the acronym from No Child Left Behind law (2002-20015); it means “Adequate Yearly Progress” in test scores for different groups of students.]

Now, I sympathize with the premise, but the certainty worried me. I started
to ask questions: Can we be confident that teachers who are effective in their
current classrooms would be equally effective elsewhere? What effect would
shifting teachers to different schools have on the likelihood that teachers
would remain in the district? Are the measures in question good proxies for
teacher quality? My concern was not that they lacked firm answers to these
questions–that’s natural enough even for veteran superintendents–it was that
they seemingly regarded such questions as distractions.

Q: What’s a concrete example of where educators and advocates
overenthusiastically used data to tout a policy, but where the results didn’t
pan out? What went wrong?

A: Take the case of class-size reduction. For two decades, advocates of
smaller classes have referenced the findings from the Student Teacher
Achievement Ratio (STAR) project, a class-size experiment conducted in Tennessee
in the late 1980s. Researchers found significant achievement gains for students
in small kindergarten classes and additional gains in first grade. The results
were famously embraced in California, which in 1996 adopted a program to reduce
class sizes that cost nearly $800 million in its first year. But the dollars
ultimately yielded disappointing results, with the only major evaluation–by AIR
and RAND–finding no effect on achievement.

What happened? Policymakers ignored nuance and context. California encouraged
districts to place students in classes of no more than 20–but that class size
was substantially larger than those for which STAR found benefits. Moreover,
STAR was a pilot program serving a limited population, which minimized the need
for new teachers. California’s statewide effort created a voracious appetite for
new educators, diluting teacher quality and encouraging well-off districts to
strip-mine teachers from less affluent communities. The moral is that even
policies or practices informed by rigorous research can prove ineffective if the
translation is clumsy or ill considered….

Q: In your mind, what are some of the main limitations of research as
they apply to schooling?

A: First, let me be clear: Good research has an enormous contribution to
make–but, when it comes to policy, this contribution is more tentative than we
might prefer. Scholarship’s greatest value is not the ability to end policy
disputes, but to encourage more thoughtful and disciplined debate.

In particular, rigorous research can establish parameters as to how big an
effect a policy or program might have, even if it fails to conclusively answer
whether it “works.” For instance, quality research has quieted assertions that
national-board-certified teachers are likely to have heroic impacts on student
achievement or that Teach For America recruits might adversely affect their
students.

Especially when crafting policy, we should not expect research to dictate
outcomes but should instead ensure that decisions are informed by the facts and
insights that science can provide. Education leaders should not expect research
to ultimately resolve thorny policy disputes over school choice or teacher pay
any more than medical research has ended contentious debates over health
insurance or tort reform….

Q: What do you see as the main motivation behind the “new stupid”? Is
it simply an example of good intentions gone awry?

A: In a word: yes. It’s a strategy pursued with the best of intentions. But
the problem is threefold. First, as we’ve discussed, too many times those of us
in K-12 are unsophisticated about what a particular study or a particular data
set can tell us. Second, the very passion that infuses the K-12 sector creates a
sense of urgency. People want to fix problems now, using whatever tools are at
hand–and don’t always stop to realize when they’re trying to fix a Swiss watch
with a sledgehammer. Third, the reality is that we still don’t have the kinds of
data and research that we need. So, too often, the choice is to misapply extant
data or simply go data-free. Everyone involved means well; the trick is provide
the right training, the right data, and for practitioners, policymakers, and
reformers to ensure that compassion doesn’t swamp common sense.

 

Can Superintendents Raise Test Scores?

I first asked this question in a post published over six years ago. I have updated and revised that post because the answer is popularly and resoundingly “yes” although the evidence is squirmy.  I revisit both the question and answer.

 

After Atlanta (GA) school administrators and teachers went to trial and were convicted and sentenced to jail for cheating and before that the El Paso (TX) superintendent convicted of the same charge and in prison, the generally accepted idea that district superintendents can pump up student  achievement has taken a serious hit. Cheating scandals across the country have turned the belief in superintendents raising test scores into something tawdry.

For decades, many superintendents have been touted as earnest instructional leaders, expert managers, and superb politicians who can mobilize communities and teacher corps to improve schools and show gains in students’ test scores. From Arlene Ackerman  in Philadelphia to Joel Klein in New York City to Kaya Henderson in Washington, D.C., big city superintendents are at the top rung of those who can turn around failing districts.

Surely the Atlanta cheating scandal and others around the country have tarnished the image of dynamic superintendents taking urban schools from being in dumpsters to $1 million Broad Prize winners. A tainted image, however, will not weaken the Velcro belief that smart district superintendents will lead districts to higher student achievement. Just look at contracts that school boards and mayors sign with new superintendents. Contract clauses call for student test scores, graduation rates, and other academic measures to increase during the school chief’s tenure (see here and here).

Then along comes a study that asks whether superintendents are “vital or irrelevant.” Drawing on state student achievement data from North Carolina and Florida for the years 1998-2009, researchers sought to find out how much of a relationship existed between the arrival of new superintendents, how long they served, and student achievement in districts (see PDF SuperintendentsBrown Center9314 ).

Here is what the researchers found:

1. School district superintendent is largely a short-term job. The typical superintendent has been in the job for three to four years.
2. Student achievement does not improve with longevity of superintendent service within their districts.
3. Hiring a new superintendent is not associated with higher student achievement.
4. Superintendents account for a small fraction of a percent (0.3 percent) of student differences in achievement. This effect, while statistically significant, is orders of magnitude smaller than that associated with any other major component of the education system, including: measured and unmeasured student characteristics; teachers; schools; and districts.
5. Individual superintendents who have an exceptional impact on student achievement cannot be reliably identified.

Results, of course, are from only one study and must be handled with care. The familiar cautions about the limits of the data and methodology are there. What is remarkable, however, is that the iron-clad belief that superintendents make a difference in student outcomes held by the American Association of School Administrators, school boards, and superintendents themselves has seldom undergone careful scrutiny. Yes, the above study is correlational. It does not get into the black box of exactly how and what superintendents do improves student achievement.

Ask superintendents how they get scores or graduation rates to go up.  The question is often answered with a wink or a shrug of the shoulders. Among most researchers and administrators who write and grapple with this question of whether superintendents can improve test scores, there is no explicit model of effectiveness. That is correct, there is no theory of change, no theory of action.

How exactly does a school chief who is completely dependent on an elected school board, district office staff, a cadre of principals whom he or she may see monthly, and teachers who shut their doors once class begins–raise test scores, decrease dropouts, and increase college attendance? Without some theory by which a superintendent can be shown to have causal effects, test scores going up or down remain a mystery or a matter of luck that the results occurred during that school chief’s tenure (I exclude cheating episodes where superintendents have been directly involved because they have been rare).

Many school chiefs, of course, believe–a belief is a covert theory–that they can improve student achievement. They hold dear the Rambo model of superintending. Strong leader + clear reform plan + swift reorganization + urgent mandates + crisp incentives and penalties =  desired student outcomes. Think former New York City Chancellor Joel Klein, ex-Miami-Dade Superintendent Rudy Crew, ex-Chancellor of Washington D.C.and ex-school chief Alan Bersin in San Diego. Don’t forget John Deasy in Los Angeles Unified School District. And now, Pedro Martinez in San Antonio Independent School District

There are, of course, other less heroic models or theories of action that mirror more accurately the complex, entangled world of moving school board policy to classroom practice. One model, for example, depicts stable, ongoing, indirect influence where superintendents slowly shape a district culture of improvement, work on curriculum and instruction, insure that  principals run schools consistent with district goals, support and prod teachers to take on new classroom challenges, and communicate often with parents about what’s happening. Think ex-superintendents Carl Cohn in Long Beach (CA), Tom Payzant in Boston (MA) and Laura Schwalm in Garden Grove (CA). Such an indirect approach is less heroic, takes a decade or more, and ratchets down the expectation that superintendents be Supermen or Wonder Women.

Whether school chiefs or their boards have a Rambo model, one of indirect influences, or other models, some theory exists to explain how they go about improving student performance. Without some compelling explanation for how they influence district office administrators, principals, teachers, and students to perform better than they have, most school chiefs have to figure out their own personal cause-effect model, rely upon chance, or even in those rare occasions, cheat.

What is needed is a crisp GPS navigation system imprinted in school board members’ and superintendents’ heads that contain the following:

*A map of the political, managerial, and instructional roles superintendents perform, public schools’ competing purposes, and the constant political responsiveness of school boards to constituencies that inevitably create persistent conflicts.

*a clear cause-effect model of how superintendents directly influence principals and teachers and they, in turn,influence students to do better as in creating incentives and sanctions, a culture of trust that encourages both risk-taking and willingness to learn.

*a practical and public definition of what constitutes success for school boards, superintendents, principals,teachers, and students beyond standardized test scores, higher graduation rates, and college admissions.

Such a navigation system and map are steps in the right direction of answering the question of whether superintendents can raise test scores.

What Makes a Great School? (Jack Schneider)

Jack Schneider is an Assistant Professor of Education at the University of Massachusetts, Lowell. He is  “a historian and policy analyst who studies the influence of politics, rhetoric, culture, and information in shaping attitudes and behaviors. His research examines how educators, policymakers, and the public develop particular views about what is true, what is effective, and what is important. Drawing on a diverse mix of methodological approaches, he has written about measurement and accountability, segregation and school choice, teacher preparation and pedagogy, and the relationship between research and practice. His current work, on how school quality is conceptualized and quantified, has been supported by the Spencer Foundation and the Massachusetts State Legislature.

The author of three books, Schneider is a regular contributor to “The Washington Post” and “The Atlantic” and co-hosts the education policy podcast “Have You Heard.” He also serves as the Director of Research for the Massachusetts Consortium for Innovative Education Assessment.”

This piece appeared October 23, 2017.

 

What are the signs that a school is succeeding?

Try asking someone. Chances are, they’ll say something about the impact a school makes on the young people who attend it. Do students feel safe and cared for? Are they being challenged? Do they have opportunities to play and create? Are they happy?

If you’re a parent, getting this kind of information entails a great deal of effort — walking the hallways, looking in on classrooms, talking with teachers and students, chatting with parents, and watching kids interact on the playground.

Since most of us don’t have the time or the wherewithal to run our own school-quality reconnaissance missions, we rely on rumor and anecdote, hunches and heuristics, and, increasingly, the Internet.

So what’s out there on the web? Are our pressing questions about schools being answered by crowdsourced knowledge and big data sets?

As it turns out, no.

There’s information, certainly. But mostly it doesn’t align with what we really want to know about how schools are doing. Instead, most of what we learn about schools online — on the websites of magazines, on school rating sites, and even on real estate listings — comes from student standardized test scores. Some may include demographic information or class size ratios. But the ratings are derived primarily from state-mandated high stakes tests.

The first problem with this state of affairs is that test scores don’t tell us a tremendous amount about what students are learning in school. As research has demonstrated, school factors explain only about 20 percent of achievement scores — about one-third of what student and family background characteristics explain. Consequently, test scores often indicate much more about demography than about schools.

Even if scores did reflect what students were learning in school, they’d still fail to address the full range of what schools actually do. Multiple-choice tests communicate nothing about school climate, student engagement, the development of citizenship skills, student social and emotional health, or critical thinking. School quality is multidimensional. And just because a school is strong in one area does not mean that it is equally strong in another. In fact, my research team has found that high standardized test score growth can be correlated with low levels of student engagement. Standardized tests, in short, tell us very little about what we actually value in schools.

One consequence of such limited and distorting data is an impoverished public conversation about school quality. We talk about schools as if they are uniformly good or bad, as if we have complete knowledge of them, and as if there is agreement about the practices and outcomes of most value.

Another consequence is that we can make unenlightened decisions about where to live and send our children to school. Schools with more affluent student bodies tend to produce high test scores. Perceived

as “good,” they become the objects of desire for well-resourced and quality-conscious parents. Conversely, schools with more diverse student bodies are dismissed as bad.

GreatSchools.org gives my daughter’s school — a highly diverse K–8 school — a 6 on its 10-point scale. The state of Massachusetts labels it a “Level 2” school in its five-tier test score-based accountability system. SchoolDigger.com rates it 456th out of 927 Massachusetts elementary schools.

How does that align with reality? My daughter is excited to go to school each day and is strongly attached to her current and former teachers. A second-grader, she reads a book a week, loves math, and increasingly self-identifies as an artist and a scientist. She trusts her classmates and hugs her principal when she sees him. She is often breathlessly excited about gym. None of this is currently measured by those purporting to gauge school quality.

Of course, I’m a professor of education and my wife is a teacher. Our daughter is predisposed to like school. So what might be said objectively about the school as a whole? Over the past two years, suspensions have declined to one-fifth of the previous figure, thanks in part to a restorative justice program and an emphasis on positive school culture. The school has adopted a mindfulness program that helps students cope with stress and develop the skill of self-reflection. A new maker space is being used to bring hands-on science, technology, engineering, and math into classrooms. The school’s drama club, offered free after school twice a week, now has almost 100 students involved.

The inventory of achievements that don’t count is almost too long to list.

So if the information we want about schools is too hard to get, and the information we have is often misleading, what’s a parent to do?

Four years ago, my research team set out to build a more holistic measure of school quality. Beginning first in the city of Somerville, Massachusetts, and then expanding to become a statewide initiative — the Massachusetts Consortium for Innovative Education Assessment — we asked stakeholders what they actually care about in K–12 education. The result is a clear, organized, and comprehensive framework for school quality that establishes common ground for richer discussions and recognizes the multi-dimensionality of schools.

Only after establishing shared values did we seek out measurement tools. Our aim, after all, was to begin measuring what we value, rather than to place new values on what is already measured.

For some components of the framework, we turned to districts, which often gather much more information than ends up being reported. For many other components, we employed carefully designed surveys of students and teachers — the people who know schools best. And though we currently include test score growth, we are moving away from multiple-choice tests and toward curriculum-embedded performance assessments designed and rated by educators rather than by machines.

Better measures aren’t a panacea. Segregation by race and income continues to menace our public schools, as does inequitable allocation of resources. More accurate and comprehensive data systems won’t wash those afflictions away. But so much might be accomplished if we had a shared understanding of what we want our schools to do, clear and common language for articulating our aims, and more honest metrics for tracking our progress.