Category Archives: Uncategorized

Did That Edtech Tool Really Cause That Growth? (Mary Jo Madda)

The quality of research on technology use in schools and classrooms leaves much to be desired. Yet academics and vendors crank out studies monthly. And they are often cited to justify using particular programs. How practitioners can make sense of research studies is an abiding issue. This post offers viewers some cautionary words in looking carefully at findings drawn from studies of software used in schools.

“Mary Jo Madda (@MJMadda) is Senior Editor at EdSurge, as well as a former STEM middle school teacher and administrator. In 2016, Mary Jo was named to the Forbes ’30 Under 30′ list in education.” This post appeared in EdSurge, August 10, 2016.

How do you know whether an edtech product is effective in delivering its intended outcomes? As the number of edtech products has ballooned in the past five years, educators—and parents—seek information to help them make the best decision. Companies, unsurprisingly, are happy to help “prove” their effectiveness by publishing their own studies, sometimes in partnership with third-party research groups, to validate the impact of a product or service.

But oftentimes, that research draws incorrect conclusions or is “complicated and messy,” as Alpha Public Schools’ Personalized Learning Manager Jin-Soo Huh describes it. With a new school year starting, and many kids about to try new tools for the first time, now is a timely moment for educators to look carefully at studies, scrutinizing marketing language and questioning the data for accuracy and causation vs. correlation. “[Educators] need to look beyond the flash of marketing language and bold claims, and dig into the methodology,” Huh says. But it’s also up to companies and startups to question their own commissioned research.

To help educators and companies alike become the best critics, here are a few pieces of advice from administrators and researchers to consider when reviewing efficacy studies—and deciding whether or not the products are worth your time or attention.

For Educators

#1: Look for the “caveat statements,” because they might discredit the study.

According to Erin Mote, co-founder of Brooklyn Lab Charter School in New York City, one thing she and her team look for in studies are “caveat statements,” where the study essentially admits that it cannot fully draw a link between the product and an outcome.

“[There are] company studies that can’t draw a definitive causal link between their product and gains. The headline is positive, but when you dig down, buried in three paragraphs are statements like this,” she tells EdSurge, pointing to a Digital Learning Now study about math program Teach to One (TtO):

The report concludes, “The TtO students generally started the 2012-13 academic year with mathematics skills that lagged behind national norms. Researchers found that the average growth of by TtO students surpassed the growth achieved by students nationally. Although these findings cannot be attributed to the program without the use of an experimental design, the results appear encouraging. Achievement gains of TtO students, on average, were strong.”

Mote also describes her frustration with companies that call out research studies as a marketing tactic, such as mentioning both studies and the product within a brief, 140-character Tweet orFacebook post—even though the study is not about the product itself, as in the Zearn Tweet below. “I think there is danger in linking studies to products which don’t even talk about the efficacy of that product,” Mote says, calling out that companies that do this effectively co-opt research that is unrelated to their products.

Research from @RANDCorporation shows summer learning is key. Use Zearn this summer to strengthen math skills.”

#2: Be wary of studies that report “huge growth” without running a proper experiment or revealing complexities in the data.

Research at Digital Promise, something consumers should look for is “whether or not the study is rigorous,” specifically by asking questions like the following four:

  • Is the sample size large enough?
  • Is the sample size spread across multiple contexts?
  • Are the control groups mismatched?
  • Is this study even actually relevant to my school, grade, or subject area?

Additionally, what if a company claims massive growth as indicated by a study, but the data in the report doesn’t support those claims?

Back in the early 2000s, John Pane and his team at the RAND Corporation set out to demonstrate the effectiveness of Carnegie Cognitive Tutor Algebra. Justin Reich, an edtech researcher at Harvard University, wrote at length about the study, conceding that the team “did a lovely job with the study.”

However, Reich pointed out that users should be wary of claims made by Carnegie Learning marketers that the product “doubles math learning in one year” when, as Reich describes, “middle school students using Cognitive Tutor performed no better than students in a regular algebra class.” He continues:

“In a two-year study of high school students, one year Cognitive Tutor students performed the same as students in a regular algebra class, and in another year they scored better. In the year that students in the Cognitive Tutor class scored better, the gains were equivalent to moving an Algebra I student from the 50th to the 58th percentile.”

Here’s another example: In a third-party study released by writing and grammar platform NoRedInk involving students at Shadow Ridge Middle School in Thornton, CO, the company claims that every student who used NoRedInk grew at least 3.9 language RIT (student growth) points on the popularly-used MAP exam or—by equivalence—at least one grade level, demonstrated in a graph (shown below) on the company’s website. But upon further investigation, there are a few issues with the bar graph, says Alpha administrator Jin-Soo Huh.


While the graph shows that roughly 3.9 RIT points equate to one grade level of growth, there’s more to the story, Huh says. That number is the growth expected for an average student at that grade level, but in reality, this number varies from student to student: “One student may need to grow by 10 RIT points to achieve one year of typical growth, while another another student may just need one point,” Huh says. The conclusion: these NoRedInk student users who grew 3.9 points “may or may not have hit their yearly growth expectation.”

Additionally, one will find another “caveat” statement on Page 4 of the report, which reads: “Although answering more questions is generally positively correlated with MAP improvement, in this sample, there was not a statistically significant correlation with the total number of questions answered.”

According to Jean Fleming, NWEA’s VP of Communications, “NWEA does not vet product efficacy studies and cannot offer insight into the methodologies used on studies run outside our organization when it comes to MAP testing. Hence, all the more reason for users to be aware of potential snags.

For Companies

#1: Consider getting your “study” or “research” reviewed.

No one is perfect, but according to Alpha administrator Jin-Soo Huh, “Edtech companies have a responsibility when putting out studies to understand data clearly and present it accurately.”

To help, Digital Promise launched on Aug. 9 an effort to help evaluate whether or not a research study meets its standard of quality. (Here are a few studies that the nonprofit says pass muster, listed on DP’s “Research Map.“) Digital Promise and researchers from Columbia Teachers College welcome research submissions between now and September from edtech companies in three categories:

  • Learning Sciences: How developers use scientific research to justify why a product might work
  • User Research: Rapid turnaround-type studies, where developers collect and use information (both quantitative and qualitative) about how people are interacting with their product
  • Evaluation Research or Efficacy Studies: How developers determine whether a product has a direct impact on learning outcomes

#2: Continue conducting or orchestrating research experiments.

Jennifer Carolan, a teacher-turned-venture capitalist, says both of her roles have required her to be skeptical about product efficacy studies. But Carolan is also the first to admit that efficacy measurement is hard, and needs to continue happening:

“As a former teacher and educational researcher, I can vouch for how difficult it can be to isolate variables in an educational setting. That doesn’t mean we should stop trying, but we need to bear in mind that learning is incredibly complex.”

When asked about the state of edtech research, Francisco responds that it’s progressing, but there’s work to be done. “We still have a long way to go in terms of being able to understand product impact in a lot of different settings,” she writes. However, she agrees with Carolan, and adds that the possibility of making research mishaps shouldn’t inhibit companies from conducting or commissioning research studies.

“There’s a lot of interest across the community in conducting better studies of products to see how they impact learning in different contexts,” Francisco says.

Disclosure: Reach Capital is an investor in EdSurge.



Filed under Uncategorized

Cartoons about Silicon Valley and Technology

For this month, I have found a dozen or so cartoons that poke fun at the culture of Silicon Valley and life with technology. Enjoy !


























Filed under Uncategorized

Seventh Anniversary of This Blog

Dear Readers,

This post marks my seventh anniversary as a blogger. I want to thank those readers who regularly read my twice-weekly posts, those who have dipped into them occasionally, those who have subscribed to the post, and finally those who have taken the time to write thoughtful comments. Nearly 1.4 million viewers from around the world have clicked on to the blog since August 2009. Not exactly viral but, for me, most gratifying.

For the 852 posts I have written since 2009, I have followed three rules:

  1. Write about 800 words.
  2. Write clearly on school reform and classroom practice.
  3. Take a position and back it up with evidence.

For anyone who blogs or writes often knows that sticking to these rules has been no easy task. Yet after seven years, it has been very satisfying. I remain highly motivated to write about policymakers, administrators, teachers, parents, and students–all who inhabit the policy-to-practice continuum–and all who in different ways, with varied ideas, seek to improve schooling.

To me, writing is a form of learning and teaching. The learning part comes from figuring out what I want to say on a topic, researching it, drafting a post, and then revising it more times than I would ever admit so that the post says what I want it to say. Learning also has come from the surprises and mistakes I have found in the suggestions and comments readers post—“Did I really say that?,” “Oops!,” “Sorry, I didn’t expect what you said, “ or “I had never considered that point.”

The teaching part comes from putting my ideas out there in a clearly expressed logical argument, buttressed by evidence, for others who may agree or disagree about an issue I am deeply interested in. As in all teaching, planning enters the picture in how I frame the central question I want readers to consider and how I put the argument and evidence together in a clear, coherent, and crisp blog of about 800 words. When readers send in thoughtful and engaged comments–that is the precious interaction that teachers need for learning to occur–I respond and the act of teaching (and, yes, learning) occurs.

Because of my background as a high school teacher, administrator, and historian of education I often give a question or issue its historical context. I do so, and here I put my teaching hat on, since I believe that current policy-driven reforms and their journey into schools and classrooms are deeply rooted in the past. Learning from how earlier generations of reformers coped with the complexities of improving schools and districts, I believe, can inform current reformers about the tasks they face. Contemporary reformers, equally well intentioned as their predecessors, in too many instances ignore what has occurred previously and end up stumbling and repeating errors that occurred before. These frustrated reformers then blame teachers and principals (or “the system”) for not executing properly their reform-driven policies.

Knowing the historical context is important in understanding the cornucopia of policy-driven reforms that have spilled over U.S. public schools for over a half-century. For those unacquainted with that history, in every decade since World War II, policymakers have sought to use public schools–an essentially conservative community institution–as engines of reform to solve national and local political, economic, and social problems. From ending racial segregation in schools to defending the nation against the Soviet Union to ending poverty to growing a strong economy, national leaders have turned to local public schools to end vexing problems. This steadfast belief in education curing problems has trumped time and again political action in the larger society to alter deep-seated economic, political, and social structures that have created and sustained many of the problems afflicting the U.S. That reluctance to look beyond public schools as the solvent for national problems is just as evident in 2016 as it was in 1950.

In the upcoming year, I will look anew and historically at the policy-to-practice continuum in my continuing effort to persuade viewers that adopted policies are merely words unless put into practice. And because too many reform-driven policymakers are inattentive to what has occurred in past efforts and what occurs daily in classrooms, chances of full or even moderate implementation approach nil. It is that journey from making policy in decision-makers’ suites to K-12 classrooms that has occupied me for decades. And so I continue for another year.

Again, readers, thank you.

Larry Cuban



Filed under Uncategorized

Cartoons on Schooling and Reform

The batch of cartoons I have collected for this month are about schooling, reform, and counter-reforms in the second decade of the 21st century. I hope you smile, chuckle, and grin. Or maybe, grind your teeth or slap your brow. Whatever your reaction, enjoy!merit+pay


























Filed under Uncategorized

Cartoons: A Mix This Month

Over the past seven years, I have published monthly cartoons on various aspects of policy and practice, school reform, etc. This month, the cartoons are a mix of ones that tickled me over the year. Yeah, I know, these choices tilt toward life in a digital world. So be it. Enjoy!



parents-kids Groupon






















Filed under Uncategorized

“Transforming” Public Schools: Enough already with an Overhyped Word!

We have the opportunity to completely reform our nation’s schools. We’re not talking about tinkering around the edges here. We’re talking about a fundamental re-thinking of how our schools function—and placing a focus on teaching and learning like never before…. With the first decade of the 21st century now history, we’ve committed to securing the vitality of our nation by transforming the way we teach our students.  U.S. Secretary of Education, Arne Duncan, 2010


Transform the way teachers teach and how children learn by replacing group-based, teacher-centered instruction with personalized, learner-centered instruction….

Transform the quality of work life for teachers, administrators, and support staff by transforming a school system’s organization culture, its reward system, job descriptions, and so on, to align with the requirements of the new teaching and learning processes….

Transform the way in which educators’ create change by replacing piecemeal change strategies with whole-system change strategies.... Francis Duffy, 2010


Computers, the Internet, online courses, smart phones, cameras, interactive whiteboards, and other digital tools play an important role in improving and, yes, transforming schools.  The role of technology in schools will increase, and as we use these new tools wisely, they help make schools more effective and engaging.    Andrew Zucker, 2012


Harness Technology to transform your School: With technology, anything is possible and today’s students experience and use technology every hour of every day. Shouldn’t your classrooms have the technology products and solutions to help your students move forward?    Advertisement for conference on technology held by HB Communications, 2016



If you enter “school reform” in a Google search you will get 12, 100,000 hits. But were you to type in “transformed schools,” you would get 111,000,000 hits (as of May 17, 2016). When it comes to school reform, as the quotes above indicate, the word “transform” hits the jackpot of overhyped words in reformers’ vocabulary. Another highly touted word that has become puffery is “disrupt” as in “disrupting schools through technological innovations” (which got a measly 1,430,000 Google “results” on May 19, 2016). But for today, one overrated word is enough.  I will concentrate on “transform”

The dictionary meaning of the verb and noun (see here and here) refers to dramatic changes in form, appearance, and conditions. Often used as an example is the metamorphosis of the butterfly.




But “transform” applied to institutions is less biological, less genetic and far more hand-made. Humans manufacture changes.  But not just any change. In the world of school reformers, “transform,” implies not only dramatic changes but ones that make better schools. Also implied is that “better” means fundamental or radical, not incremental or tinkering changes. Moreover, these fundamental changes are instituted speedily rather than slowly. Here are some images that capture the range of meanings for the verb and noun when applied to individuals and organizations:







This post, then, is about this over-used, pumped-up word and its implications especially how meaningless it has become in policy-talk. Keep in mind that historically there have been proof-positive “transformations.” One-room rural schoolhouses in the 19th century changed into brick-and-mortar age-graded schools with scores of classrooms by the end of that century. A few decades later, reformers launched the innovative comprehensive high school. Previously about 10 percent of students had graduated high school in 1890; a century later, about 75 percent graduated the comprehensive high school. Those are “transformations” in school organization that strongly influenced teachers and students in schedule, curriculum, and instruction (see here and here).

Think about the Brown v. Board of Education decision (1954) and the subsequent Civil Rights Act that enforced school desegregation. With court-ordered desegregation in district after district, by the mid-1980s, more black students in the South were going to schools with whites than elsewhere in the nation. That was a “transformation.” With subsequent U.S. Supreme Court decisions that returned authority to local districts in assigning students to neighborhood schools (thus, reflecting residential segregation), re-segregation has reappeared (see here and here).

Yes, I have gotten allergic to the word “transform” when it is applied to schooling. That allergy has prompted me to ask any policymaker, researcher, practitioner, high-tech entrepreneur, venture capitalist, or parent using the word, certain questions about what he or she means.

1. What does “transform” mean to you?

Sometimes I use above images (e.g., like a before/after photo of an overweight man? A butterfly?) to prompt the picture of the change that resides in the head of the person .

2. What are the problems to which “transformed” schools is the solution?

Is the problem academic achievement falling behind other nations? Or is it the long-term achievement gap between whites and minorities? Or is it the technological backwardness of schools compared to other industries?

3. What exactly is to be transformed?  school structures? Cultures? Classroom teaching? Learners?

Public schools as an institution are complex organizations with many moving parts, some being tightly coupled to one another while some are often unconnected to one another. What, then is the target for the “transformation?”

4. Transform to what? what are the outcomes that you want to achieve?

This is the key question that gets at what the believer in “transforming” schools wants to be better. It reveals the person’s value about the place of schooling in a democratic society and the kinds of teaching and learning that are “good.”  Of all the questions, this cannot be skipped.

5.  How fast should the “transformation” be?

Nearly always, believers in “transformed” schools believe in speedy action, grand moves while the window of opportunity is open. Not in making changes slowly or in small increments.

6. How will you know that the “transformation” will be better than what you already have?

Ah, the evaluation question that captures in another way the desired outcomes, the better school.

So, if viewers want to end the promiscuous use of a word leached of its meaning in policy-talk, I suggest asking these questions. To do so, may lose you an acquaintance or colleague but, in the end, both parties gain a larger and deeper sense of what the words “transform schools” mean. And maybe I will stop sneezing when the word comes up.



Filed under school reform policies, Uncategorized

More Cartoons on Teaching and Learning Math in and out of School

Cartoons about teaching academic subjects have appeared in this monthly feature many times. Teaching and learning math in and out of school, it seems, gives cartoonists’ pens ample subject matter to make fun of. This month is no exception. Enjoy these!






























Filed under Uncategorized