Category Archives: technology use

Algorithms in Use: Evaluating Teachers and “Personalizing” Learning (Part 2)

In Part 1, I made the point that consumer-driven or educationally-oriented algorithms for all of their mathematical exactness and appearance of objectivity in regression equations contain different values among which programmers judge some to be more important than others.  In making value choices (like everyone else, programmers are constrained by space, time, and resources), decisions get made that have consequences for both teachers and students. In this post, I look first at those algorithms used to judge teachers’ effectiveness (or lack of it) and then I turn to “personalized learning” algorithms customized for individual students.

Washington, D.C.’s IMPACT program of teacher evaluation

Much has been written about the program that Chancellor Michelle Rhee created during her short tenure (2007-2010) leading the District of Columbia public schools (see here and here). Under Rhee, IMPACT,  a new system of teacher evaluation has been put into practice. The system is anchored in The “Teaching and Learning Framework,”  that D.C. teachers call the “nine commandments” of good teaching.

1. Lead well-organized, objective-driven lessons.

2. Explain content clearly.

3. Engage students at all learning levels in rigorous work.

4. Provide students with multiple ways to engage with content.

5. Check for student understanding.

6. Respond to student misunderstandings.

7. Develop higher-level understanding through effective questioning.

8. Maximize instructional time.

9. Build a supportive, learning-focused classroom community.

IMPACT uses multiple measures to judge the quality of teaching. At first, 50 percent of an annual evaluation was based upon student test scores; 35 percent based on judgments of instructional expertise (see “nine commandments”) drawn from five classroom observations by the principal and “master educators,” and 15 percent based on other measures. Note that policymakers initially decided on these percentages out of thin air. Using these multiple measures, IMPACT has awarded 600 teachers (out of 4,000) bonuses ranging from $3000 to $25,000 and fired nearly 300 teachers judged as “ineffective” in its initial years of full operation. For those teachers with insufficient student test data, different performance measures were used. Such a new system caused much controversy in and out of the city’s schools (see here and here)

Since then, changes have occurred. In 2012, the 50 percent of a teacher’s evaluation based on student test scores had been lowered to 35 percent (why this number? No one says) and the number of classroom observations had been reduced. More policy changes have occurred since then (e.g., “master educator” observations have been abolished and now principals do all observations; student surveys of teachers added). All of these additions and subtractions to IMPACT mean that the algorithms used to judge teachers have had to be tweaked, that is, altered because some variables in the regression equation were deemed more (or less) important than others. These policy changes, of course, are value choices. For a technical report published in 2013 that reviewed IMPACT, see here.

And the content of the algorithms have remained secret. An email exchange between the overseer of the algorithm in the D.C. schools and a teacher (who gave her emails to a local blogger) in 2010-2011 reveal the secrecy surrounding the tinkering with such algorithms (see here). District officials have not yet revealed in plain language the complex algorithms to teachers, journalists, or the general public. That value judgments are made time and again in these mathematical equations is clear. As are judgements in the regression equations used to “personalize learning.”

Personalized Learning algorithms

“The consumerist path of least resistance in America takes you to Amazon for books, Uber for transportation, Starbucks for coffee, and Pandora for songs. Facebook’s ‘Trending’ list shows you the news, while Yelp ratings lead you to a nearby burger. The illusion of choice amid such plenty is easy to sustain, but it’s largely false; you’re being herded by algorithms from purchase to purchase.”

Mario Bustillos, This Brand Could be Your Life, June 28, 2016

Bustillos had no reason to look at “personalized learning” in making her case that consumers are “herded by algorithms from purchase to purchase.” Had she inquired into it, however, she would have seen the quiet work of algorithms constructing “playlists” of lessons for individual students and controlling students’ movement from one online lesson to another absent any teacher hand-prints on the skills and content being taught. Even though the rhetoric of “personalized learning” mythologizes the instructional materials and learning as student-centered, algorithms (mostly proprietary and unavailable for inspection) written by programmers making choices about what students should learn next are in control. “Personalized learning” is student-centered in its reliance on lessons tailored to ability and performance differences among students. And the work of teachers is student-centered in coaching, instructing, and individualizing their attention as well as monitoring small groups working together. All of that is important, to be sure. But the degree to which students are making choices out of their interests and strengths in a subject area, such as math, they have little discretion. Algorithms rule (see here, here, and here).

Deeply embedded in these algorithms are theories of learning that seldom are made explicit. For example, adaptive or “personalized learning” are contemporary, high-tech versions of old-style mastery learning. Mastery learning, then and now, is driven by behavioral theories of learning. The savaging of “behaviorism” by cognitive psychologists and other social scientists in the past few decades has clearly given the theory a bad name. Nonetheless, behaviorism and its varied off-shoots drive contemporary affection for “personalized learning” as it did for “mastery learning” a half-century ago (see here and here). I state this as a fact, not a criticism.

With advances in compiling and analyzing masses of data by powerful computers, the age of the algorithm is here. As consumers, these rules govern choices we make in buying material goods and, as this post claims, in evaluating teachers and “personalized learning.”







Filed under school reform policies, technology use

Consumer Choice in Schooling: Algorithms and Personalized Learning (Part 1)

“The consumerist path of least resistance in America takes you to Amazon for books, Uber for transportation, Starbucks for coffee, and Pandora for songs. Facebook’s ‘Trending’ list shows you the news, while Yelp ratings lead you to a nearby burger. The illusion of choice amid such plenty is easy to sustain, but it’s largely false; you’re being herded by algorithms from purchase to purchase.”

Mario Bustillos, This Brand Could be Your Life, June 28, 2016


I wish I had written that paragraph. It captures a definite feature not only of our consumerist-driven society but also in recent school reform (e.g., the growth of charter schools and expanded parental choice). I also include the media hype and techno-enthusiasm for “personalized learning.” The centerpiece of any form of “personalized learning” (or “adaptive learning“) is algorithms for tailoring lessons to individual students (see here, here, and here). What Bustillos omits  in the above article about the dominance of consumerism driven by algorithms is that regression equations embedded in algorithms make predictions based on data. Programmers decide on how much weight to put on particular variables in the equations. Such decisions are subjective; they contain value judgments about the independent and dependent variables and their relationship to one another. The numbers hide the subjectivity within these equations.

Like Facebook designers altering its algorithm so as to direct news tailored to each Facebook consumer “to put a higher priority on content shared by friends and family,” software engineers create different versions of  “personalized learning” and insert value judgments into the complicated regression equations with which they have written for online lessons. These equations are anchored in the data students produce in answering questions in previous lessons. These algorithms predict (not wholly since engineers and educators do tweak–“massage” is a favored word–the equations) what students should study and absorb in individualized, daily, online software lessons (see here).

Such “personalized” lessons alter the role of the teacher for the better, according to promoters of the trend. Instead of covering content and directly teaching skills, teachers can have students work online thereby freeing up the teacher to coach, give individual attention to students who move ahead of their classmates and those who struggle.

Critics, however, see the spread of online, algorithmic-based lessons as converting teaching to directing students to focus on screens and automated lessons thereby shrinking the all-important role of teacher-student relationships, the foundation for social, moral, and cognitive learning in public schools. Not so, advocates of “personalized learning” aver. There might be fewer certified teachers in schools committed to lessons geared to individual students (e.g., Rocketship) but teachers will continue to perform as mentors, role models, coaches, and advisers not as mere purveyors of content and skills.

As in other policy discussions, the slippage into either/or dichotomies beckons. The issue is not whether or not to use algorithms since each of us uses algorithmic thinking daily. Based on years of experiential data we have compiled in our heads (without regression equations) step-by-step routines just to get through the day (e.g., which of the usual routes to work should I take; how best to get the class’s attention at the beginning of a lesson). Beyond our experiences, however, we depend on mathematical algorithms embedded in the chips that power our Internet searches Internet, control portions of our driving cars and operate home appliances.

The issue is not that algorithms are value-free (they are not) or data rich (they are). The issue is whether practitioners and parents–consumers of fresh out-of-the-box products–come to depend automatically on carefully constructed algorithms which contain software designers’ value judgments displayed in flow charts and written into code for materials and lessons students will use tomorrow. Creators of algorithms (including ourselves) juggle certain values (e.g., favorite theory of learning, student-centered instruction, small group collaboration, correctness of information, increasing productivity and decreasing cost, ease of implementation) and choose among them  in constructing their equations. They judge what is important and select among those values since time, space, and other resources are limited in creating the “best” or “good enough” equation for a given task. Software designers choose to give more weight to some variables over others–see Facebook decision above. Rich, profuse data, then, never speaks for itself. Look for the values embedded in the algorithmic equations. Such simple facts are too often brushed aside.

What are algorithms?

Wikipedia’s definition of an algorithm is straight forward: a sequence of steps taken to solve a problem and complete a task. Some images make the point for simple algorithms.











Or if you want a Kahn Academy video to explain an algorithm, see here.

Complex algorithms

Most algorithms are hardly simple, however. Amazon’s proprietary algorithms on searches and popularity of books, for example, are unavailable to the public yet are heavily leaned upon by advertisers, authors, and consumers (e.g., also Amazon’s  algorithmic feature that appears on your screen: “customers who viewed this also viewed….”).  Among school reformers interested in evaluating teachers on the basis of students’ test scores, algorithms and their complex regression equations have meant the difference between getting a bonus or getting fired, for example,  in Washington, D.C. . And for those “personalized learning” advocates eager to advance student-centered classrooms,  algorithms  contain theories of action of what-causes-what that tilt toward one way of learning. In short, software designers’ value judgements matter as to what pops out at the other end of the equation. and then is used in making an evaluative judgment and an instructional decision.

Part 2 will look at values in algorithms that evaluate teachers and customize learning.



Filed under how teachers teach, school reform policies, technology use

Hype on Steroids: Self-Driving Cars and School Technologies

A full week of mainstream and social media swept across the nation about the death of a Tesla car owner killed in Florida using the self-driving option. With the auto-pilot function turned on, the Tesla driver collided with a tractor-trailer and became the first known fatality in the industry’s surge to produce self-driving cars. Google and Tesla and 30 other companies (e.g., Honda, Ford, GM,Toyota) compete for what is hyped as the “next big thing”; such cars, they claim, will “disrupt” the century-old personal transportation market.

A Morgan Stanley Blue Paper announced in 2013:

Autonomous cars are no longer just the realm of science fiction.They are real and will be on roads sooner than you think. Cars with basic autonomous capability are in showrooms today, semi-autonomous cars are coming in 12-18 months, and completely autonomous cars are
set to be available before the end of the decade

Tesla’s founder, Elon Musk said the self-driving function on the Tesla meant that “[t]he probability of having an accident is 50 per cent lower if you have Autopilot on” …. “Even with our first version, it’s almost twice as good as a person.”

Skeptics have tossed in their two cents (see here and here; for rebutting skeptics, see here) but when it comes to questioning new technologies in U.S. culture, skeptics are alien creatures.

While the hype pumping up self-driving cars can lead to accidents and deaths, no such serious consequences accompany promoters of technological innovations who have promised increased teacher efficiency, improved student achievement, and the end of low-performing schools for the  past half-century.  Need I mention that Google has a “Chief Evangelist for Global Education?”

Nothing surprising about hype (even when  injected with steroids)  in a consumer-driven, highly commercial society committed to practicing democracy. Hype is hype either for self-driving cars or for school technologies. Parsing the hyped language and images becomes important because real-life consequences flow from these words and pictures.


Consider these advertisements championing new technologies since the 1950s.





Over-stated claims are  commonplace when it comes to pumping up the benefits of the “next big thing.” Early adopters of new technologies discover the bugs in new hardware and software soon enough.  Glitches, however, seldom dissuade this crowd from peering around the corner for its replacement.

Does hype serve any social and political purpose other than to stimulate consumers to buy the product? I believe it does.

1. Over-the-top statements strengthen the popular belief that change is “good” for individuals and society overall. Not only is change “good” for Americans but in the technology industry and culture of school reform, change morphs into improvement. In Silicon Valley argot, “making the world a better place,” means a new product, a new service, a new app will improve life (a parody of this oft-repeated phrase can be seen here)

Equating change with improvement is a cognitive error. Surely, an improvement implies a change has occurred but because the change has happened, improvement does not necessarily follow. A moment’s thought would quickly squelch equating change with improvement. Stepping on a scale and seeing that you have gained five pounds while on a low-carb diet is clearly a change but not, in your view, an improvement. Think of a divorce in a family. The spouse initiating the divorce sees the split as a change for the better but for the others involved including children, few would see it as an improvement with two homes, living with different parents or weekend visits. Change occurs constantly but improvement is in the mind of the beholder.

Consider whether a new app that has a “smart” button and zipper that alerts you if your fly is down or another app that locates rentable yachts are improvements to one’s life (see here). To those individuals who buy and download these apps they appear as improvements promising a better life but to others, they appear as trivial indulgences that hardly make the “world a better place.”

School reformers who believe that changes lead to improvements in teaching and learning, for example, often refer to gains in student test scores, increases in teacher productivity (i.e., less time to do routine tasks), and other measurable outcomes as evidence of  better schooling. Reformers holding divergent values (e.g., higher civic engagement, student well-being), however, would differ over whether test scores, et. al. are improvements. Quite often, then, the definition of improvement depends upon who does the defining and the values they prize.

2. Hype over new technologies raises questions about the existing institution’s quality.  Consider current health care where millions still lack health insurance, emergency rooms are over-crowded, wait time to see specialists physicians increases, and patients get less and less time when they do see their doctors. Hyping the “next big thing” in medical technology becomes a direct criticism of existing health care. Think of “hospital in a box,” or patient kiosks placed in pharmacies, where ill people go to the kiosk for video conferencing with one or more doctors about what ails them. Such new technologies raises implicit questions about access to adequate health care and to what degree the relationship between doctor and patient is important in improving health.

Or consider the thousands of lives lost on the nation’s roads to accidents and human error in driving. Self-driving cars, once prevalent on the nation’s highways will, promoters claim, dramatically reduce the 32,000 deaths in car accidents while increasing worker productivity since with self-driving cars owners can complete other tasks that heretofore would have not been done. Self-driving cars raises anew questions about the lack of adequate public transportation and a society committed to one-person-per car.

And hype for technological innovations in schools for “personalized” or “adaptive” learning pictures the existing system as factory-like  whole-class, age graded, teacher-dominated instruction that ignores, even neglects individualized lessons, student-centered learning, and reconfigured classrooms.

3. Hype shrinks the time to show results to immediately. Most software products in the educational arena, for example, take time for teachers and students to grasp, understand, and use them in lessons. Education proceeds by short not long steps. Hyping these products leave the distinct impression that unless the desired result hasn’t happened in a few months then someone (note the beginnings of blame) has failed to do it right. And it ain’t the software developer.

4. Software and hardware developers come to believe their own hype. The cliche of “drinking the Kool-Aid–applies here and such self-deception occurs. And when it does, CEOs of start-ups and other companies start making short-cuts to get products into schools and stores. Those short-cuts increase software glitches, highten arguments with consumers of the products, and diminish faith in the innovation.

These outcomes of hype are not justifications for its ubiquity. They  help me understand the role that it (and its cousin, “magical thinking”) perform in U.S. society.







Filed under how teachers teach, school reform policies, technology use

Schools That Integrate Technology: Silicon Valley

As complex as it is for an individual teacher to integrate daily use of high-tech devices into routine classroom practices, technology integration at a school level is even more complex. A classroom teacher with 25-35 students can alter the structures of her classroom and create a culture of learning, achievement and mutual respect. Hard as that is, it is do-able. I and many others have profiled teachers who have created such classrooms.

Imagine, however, schools with 30 to 100 classrooms and getting all of those teachers to work together to create school-wide infrastructure and a learning, achieving, and respectful culture–across scores of classrooms that seamlessly integrates computers to achieve the school-site’s goals. A complex task with many moving parts that is fragile yet strong. It does happen but remains uncommon.

I have observed a few schools in Silicon Valley that have integrated new technologies across the entire school requiring teachers to teach lessons using particular hardware and software. These schools vary from one another but tout that they “personalize learning,” blend instruction, and differentiate their lessons to meet differences among students. Invariably, they say they use project-based instruction.  They have created both an infrastructure and culture that subordinates technology to the larger tasks of preparing children and youth to do well academically and socially, graduate, and enter college (and complete it) or enter a career directly.

Considering what I have observed in Silicon Valley, documented nationally in my studies, and retrieved from the research literature on such schools elsewhere in the U.S., what are the common features of such schools?

Here are eight different yet interacting moving parts that I believe has to go into any reform aimed at creating a high-achieving school using technology to prepare children and youth to enter a career or complete college (or both). Note, please, that what I have garnered from direct observation, interviews, and the literature is not a recipe that can be easily cooked and served. Listing features I have  identified is not an invitation to insert some or all of these into a formula for producing such schools near and far. These schools are rooted in their contexts and context matters.

These features are:

*Recruit and train teachers who have the subject matter knowledge and skills to work with students  before, during, and after the school day.

*Recruit and train school site leaders who have the expertise and skills to lead a school and be a pillow and sandpaper simultaneously with teachers, students, and parents.

*Students have access to non-academic subjects that cultivate the mind, heart, and sensibilities.

*Equip all students with the knowledge and skills not only to enter college,  persist through four years and get a bachelor’s degree but also have the wherewithal to enter a career immediately.

*Organize the school day, week, and month that provides students with sufficient time in and out of class to learn the prescribed material and core cognitive skills to master a subject, acquire the essential skills of planning and assessing their progress in each course they take, receive tutorial help when student skill levels are below and above par, and time for students to receive mentoring from teachers they trust.

*Build a culture of safety, learning, respect, and collaboration for both youth and adults.

*Create a decision-making process that is inclusive, self-critical, and strong enough to make further changes in all of the above.

*Do all of this efficiently within available resources.

Note the absence of new technologies in the features that I have listed. Why is that?

Simply because such schools containing these features have administrators and teacher who figure out when to use software to achieve desired outcomes, create an infrastructure to support staff in using new technologies, determine which new technologies efficiently advance students in reaching these goals, and create the conditions for easy, supported use of the hardware and software. Note, then, that computers and their software are subordinate to the overarching goals for students and adults in the school.

Summit schools, a charter network in Northern California, has been working and re-working a design containing these moving parts for nearly 15 years. Over that period, they have amended, deleted, and added program features as administrators and faculty learned what worked and what didn’t. The time span, the stability in staff, their awareness of context and shifting demographics all came into play as Summit leaders and faculty figured out what to do since 2003.

Over the past two months I have visited two of Summit’s seven charter schools in the Bay area and in those two schools have watched teachers across different academic subjects teach 90-minute lessons during what the schools call “project time.” I have also interviewed administrators.  Each school was part of a different district in Silicon Valley. While one of the schools had a separate building in its district well suited to its mission, scheduling, and space for students, the other school was located on a high school campus in another district where both students and teachers worked in a series of portable classrooms. Also each drew from different populations.*

The network of Summit charter schools has been written about often and positively (see here, here, here, and here). In all instances, these teachers I observed had integrated the software they had loaded onto students’ Chromebooks, the playlists of videos and links to articles for units that teachers created, and students’ self-assessment exercises into daily lessons with varying degrees of student engagement. The charter network claims that through their Personalized Learning Plan (also see here) teachers could give each student individual help while students negotiated their ways through academic content and skills. In the two schools, I observed students during 90-minute classes in different academic subjects working on teacher-chosen projects. Students were using their Chromebooks frequently to access PLP voluntarily and at teachers’ direction.

The cliched statement said over and over again by advocates of new technologies in schools: “It is not about technology, it is about learning,” captured what I saw. Overall aims for Summit students to acquire academic content, cognitive skills, “habits of success,” and the know-how allowing students to assess their own progress involved online work  before, during and after lessons. Clearly, the school did not have to use Chromebooks and extensive software to reach the schools’ overall goals and each student’s personal ones. The technology did enable, however, the process of learning to be more efficient, more timely, and  give real-time feedback to students.

The two Summit schools in very different contexts contained these features I listed above. While differences existed between the two schools in context and staffing, both have implemented these features as best they could. Creating and massaging these many features of the Summit Schools is no easy task. It is not done once; it is a process that is constantly monitored, assessed, and altered by site leaders and staff.  Thus, listing the essential features that mark such enterprises is not a blueprint for action; it is an after-the-fact synthesis of what I saw and not easily replicable for those who have dreams of “going to scale.” It is what emerged from such efforts over a long period of time and requires tender, loving care every day. The program is fragile and easily broken by inattention, changes in leadership and staff, and declining resources. May it continue to thrive.


*Diane Tavenner, a founding teacher at Summit Prep and director of Summit Schools Network and Chief Academic Officer, Adam Carter–also a founding teacher at Summit Prep–picked the two schools. In both schools, I interviewed the principals (called Executive Directors), and they suggested various teachers I should visit. Because of scheduling difficulties, I could not see all of those recommended to me. So in both schools, I reached out to other teachers, introduced myself and asked them if I could observe their classes.  The nine teachers who permitted me to spend a 90-minute block with them taught English, social studies, science, and math. For readers who wish to see my published observations, see posts for March 13, 2016, March 16, March 21, March 23, March 29, April 1, April 6, April 12, April 18.







Filed under Reforming schools, school reform policies, technology use

Using Computers To Transform Teaching and Learning: The Flight of a Butterfly Or a Bullet?*

As regular readers of this blog know, I have embarked on another project examining “best cases” of teachers, schools, and districts integrating computers into daily activities.  After four months of classroom observations, interviews with teachers and principals, and much reading I have begun to think of this project as a possible book. Much remains to be done, however, before it becomes one. In the fall, I will visit more classrooms and schools to do observations and interviews. I will do more reading of national surveys, case studies, and rigorous inquiries into what teachers and students do with devices. But the makings of a book are there in my mind.

So here is part of a proposal that I have sent to a publisher to see if they are interested. Subsequent posts will elaborate on other parts of this book proposal.

Overview and Rationale for Proposed Book

For over 30 years, I have examined the adoption and use of computers in schools (Teachers and Machines, 1986; Oversold and Underused, 2001, Inside the Black Box, 2013). I looked at the policy hype and over-promising accompanying new technologies in each decade. The question I asked was: what happens in schools and classrooms after the school board and superintendent adopt a policy of buying and deploying new technologies to improve schooling? This is the central question for any reform-minded policymaker, entrepreneur, parent, and practitioner because if teaching practices fail to change in the desired direction embedded in the policy then the chances of any changes in student performance are diminished considerably. Thus, in pursuing the issue of changes in classroom lessons in books, articles, and my blog, I moved back and forth between adopted policies for using computers, their classroom implementation, and shifts in teaching practices.

I described and analyzed computers in schools and classrooms across the U.S. including the highly touted Silicon Valley in the San Francisco Bay area. I tracked how these advocates and donors were often disappointed in how little school and classroom practice changed, anemic results in student achievement, and uncertainties in getting the right jobs after graduation, given the claims accompanying these devices and software.

There have been, however, occasional bright spots in individual teachers thoroughly integrating laptops and tablets into their practice and moving from teacher- to student-centered classrooms. And there were scattered instances of schools and districts adopting technologies wholesale and slowly altering cultures and structures to improve how teachers teach and students learn. I documented those occasional exemplars but such instances of classroom, school, and district integration were isolated and infrequent.

What slowly became clear to me over the years of studying the use of computers to improve how teachers teach and students learn and attain the overall purposes of public schooling is that policymakers have avoided asking basic questions accompanying any policy intended to reshape classroom practice. I concluded that those questions and their answers are crucial in understanding the role that computers in schools perform when it comes to teaching and learning.

This conclusion is behind my writing this book.

Reform-driven policymakers, entrepreneurs, researchers, practitioners, and parents have sought substantial changes over the past three decades in classrooms, schools, and districts to transform schooling while improving student outcomes. Yet, too often, they either avoided the inevitable steps that need to occur for such changes to materialize in schools or hastily leap-frogged over important ones. Four simple questions capture the essential steps in going from adopted policy to classroom practice.

  1. Did policies aimed at improving student performance get fully, moderately, or partially implemented?
  2. When implemented fully, did they change the content and practice of teaching?
  3. Did changed classroom practices account for what students learned?
  4. Did what students learn meet the intended policy goals?

These questions apply to innovations aimed at improving student academic performance such as creating small high schools and launching charter schools to states and districts adopting Common Core standards, competency-based learning and project-based teaching. Most importantly for this book, these questions pertain to making new technologies from laptops to hand-held devices not only accessible to every student but also expecting teachers to regularly use computers in lessons.

The questions emphasize the critical first step of actually implementing the adopted policy. Policies are not self-implementing. They require resources, technical assistance, staff development, and administrators and teacher to work together. This is especially so for teachers who are gatekeepers determining what enters the classroom door.

So without full or moderate implementation of a policy aimed at improving student performance, there is not much sense in pursuing answers to the other questions. Evidence of putting the policy into classroom practice is essential to determining the degree to which a policy is effective (or ineffective).

Once evidence of a policy’s implementation in schools and classrooms is available then the question of whether teaching practices have changed arises. This question gets at the nexus between teaching and learning that has been taken for granted in U.S. schools since the introduction of tax-supported public education nearly two centuries ago: Change teaching and then student learning will change. This is (and has been) the taken-for-granted belief driving reformers for the past century. Determining the degree to which teaching practices have changed in the desired direction and which have remained stable is essential.

The third question closes this circle of teaching producing learning by getting at what students have actually learned as a consequence of altered teaching practices. In the past half-century, policymakers have adopted measures of desired student outcomes (e.g., test scores, graduation rates, attendance, engagement in lessons). They assume that these measures capture what students have, indeed, learned. If teaching practices have changed in the desired direction, then changes in student outcomes (i.e., learning) can be attributed to those changes in classroom practices.

The final question returns to the immediate and long-term purposes of the adopted policy and asks for an evaluation of its intended and unintended outcomes. Immediate purposes might have concentrated on student test scores and graduation rates. Long-term purposes, the overall goals for tax-supported public schools, refer to job preparation, civic engagement, and producing independent and whole human beings.

These questions establish clear linkages between reform-driven policies and teaching practice. They steer this proposed book.

What if, however, policymakers, researchers, entrepreneurs, and parents looked not only at failed uses of classroom computers but also exemplary instances that have actually altered teaching practices to achieve policy ends? Examining how such “best cases” happened and their stability (or lack of it) might unlock the crucial next step of assessing changes in teaching practices and student outcomes.


*The sub-title is a quote used by Philip Jackson, Life in Classrooms (1968), pp. 166-167.


Filed under how teachers teach, school reform policies, technology use

How To Do Adaptive Learning Right (Keith Devlin and Randy Weiner)

Keith Devlin is (@profkeithdevlin) Co-founder & Chief Scientist at BrainQuake and a mathematician in the Stanford University Graduate School of Education. Randy Weiner (@randybw15) is Co-founder & CEO at BrainQuake, a former teacher, & Co-founder and former Chair of the Board at Urban Montessori Charter School in Oakland, CA.

This opinion piece appeared in EdSurge, June 30, 2016.

As one variant of the saying goes, if your strength is using a hammer, everything can look like a nail. Examples abound in attempts to use new technologies to enhance (if not “transform”, or even “disrupt”) education. Technologists who have built successful systems in other domains—and who frequently view education as just another market in which to apply their expertise—often doom their project to fail at the start, by adopting a narrow and outdated educational model.

Namely, they see education as the provision of facts, techniques, and procedures to be delivered and explained by instruction and then practiced to mastery. Their role, then, is to bring their technological prowess to bear to make this process more efficient. In most cases they can indeed achieve this. But optimizing a flawed model of education is not in the best interests of our students, and from a learning outcomes perspective may make things worse than they already are.

In the case of adaptive learning, education commentator Audrey Watters gave examples of how things can go badly wrong on her blog. “Serendipity and curiosity are such important elements in learning,” she asks. “Why would we engineer those out of our systems and schools?” More recently, Alfie Kohn provided another summary of the numerous reasons to be skeptical of education technology solutions.

Watters’ bleak future will only come to pass if the algorithms continue to be both naïvely developed and naïvely applied, and moreover, in the case of mathematics learning (the area we both work in) applied to the wrong kind of learning tasks. Almost all the personalized math learning software systems we have seen fall into this category. But there is another way—as our work, and a thorough review by a third-party research organization—has shown.

We both work in the edtech industry and have a background in education. One of us is a university mathematician who spent several years on the US Mathematical Sciences Education Board and is now based in Stanford University’s Graduate School of Education, the other an edtech veteran who is a former teacher and who co-founded Urban Montessori Charter School.

We are both very familiar with the common “production line” model of education, and recognize that it not only appeals to many (perhaps most) technologists, but in fact is a system that they themselves did well in. But collectively, the two of us have many years of experience that indicates just how badly that approach works for the vast majority of students.

Last year, with funding from the Department of Education’s Institute for Educational Sciences, our company, BrainQuake, spent six months designing, testing and developing an adaptive engine to supply players of our launch product, Wuzzit Trouble, with challenges matched to their current ability level. We were delighted when classroom studies conducted by WestEd showed that the adaptive engine worked as intended (i.e., kept students in their zone of proximal development), straight out of the gate.

We developed the game based on a number of key insights accumulated over many years of research by mathematics education professionals that should be applicable to all edtech developers—even those who are not building math tools.

Experience Over Knowledge

First, the most effective way to view K-8 education is not in terms of “content” to be covered, acquired, mastered (and regurgitated in an exam) but as an experience. This is particularly (but not exclusively) true for K-8 mathematics learning. Mathematics is primarily something you do, not something you know.

To be sure, there is quite a lot to know in mathematics—there are facts, rules, and established procedures. Imagine the skills expected of a physician. None of us, we are sure, would want to be treated by someone who had read all the medical textbooks and passed the written tests but had no experience diagnosing and treating patients. And indeed, no medical school teaches future physicians solely by instruction, as any doctor who has gone through the mandatory, long, grueling internship can attest.

In the case of math, the inappropriateness of the classical, instruction-practice-testing dominated model of education has been made particularly acute as a result of the significant advances made in the very technology field we are working in. (Advances we wholeheartedly applaud. Our beef is not with technology—we love algorithms, after all—but with applying it poorly.) In today’s world, all of us carry around in our pockets a device that can execute almost any mathematical procedure, much faster and with greater accuracy than any human. Your smartphone, with its access to the cloud (in particular, Wolfram Alpha), can solve pretty well any university mathematics exam question.

What that device cannot do, however, is take a real world mathematical problem and solve it. To do that, you need the human brain. In order to do that, the human brain has to acquire two things, in particular: a rich and powerful set of general metacognitive problem solving skills, and a more specific ability known as mathematical thinking (a component of which is known as number sense, a term that crops up a lot in the K-8 math education world, since the development of number sense is the first key step toward mathematical thinking).

Human Adaptivity

Another key insight that guided the design of our adaptive engine is that the main adaptivity is provided by the user. After all, the human being is the most adaptive cognitive system on the planet! With good product design, it is possible to leverage that adaptivity.

Most “adaptive” math algorithms will monitor a student’s progress to select the next problem algorithmically. But it is important that these puzzles allow for a wide range of of solutions and a spectrum of “right answers,” leaving the student or teacher in full control of how to move forward and what degree of success to accept. (Of course, such an approach is not possible if the digital learning experiences are of the traditional math problem type, where the problem focuses on one particular formula or method and there is a single answer, with “right” or “wrong” the only possible outcomes.)

Indeed, students still need to grasp the basic concepts of arithmetic, understand what the various rules mean, and know when and how the different procedures can be applied. But what they do not need is to be able to execute the various procedures efficiently in a paper-and-pencil fashion on real world data.

Today’s mathematical learning apps can—and should—focus on the valuable 21st century skills of holistic thinking and creative problem solving. The mastery of specific procedures should be skills that a student acquires automatically, “along the way,” in a meaningful context of working on a complex performance task—an outcome every one of us knows works from our own experience as adults.

Breaking the Symbol Barrier

Mastery of symbolic mathematics is a major goal of math education. But as has been shown by a great deal of research stretching back a quarter of a century, the symbolic representation is the most significant reason why most people have difficulty mastering K-8 grade level math—the all-important “basics.” Almost everyone can achieve a 98 percent success rate at K-8 math if it is presented in a natural-seeming fashion (for example, understanding and perhaps calculating stats at a baseball game), but their performance drops to a low 37 percent if presented with the same math problems expressed in textbook symbolic form.

Well-designed technologies that take advantage of some unique affordances of a computer or tablet can help obliterate this historical impediment to K-8 mathematics proficiency. Students should be able to explore problems on their own until they discover—for themselves—the solution. They don’t require instruction, and they don’t need anyone to evaluate their effort. Students should get instant feedback not in the form of “right” or “wrong,” but information about how their hypotheses varied from their actual experience and how they might revise their strategy accordingly.

An analogy we are particularly fond of is with learning to play a piano (or any other musical instrument). You may benefit greatly from a book, a human teacher, or even YouTube videos, but the bulk of the learning comes from sitting down at the keyboard and attempting to play.

What could be a better example of adaptive learning than that? Tune too easy? Try a harder piece. Too difficult? Back off and practice a bit more with easier ones, or break the harder one up into sections and master each one on its own at a slower pace, and then string them all together. The piano is not adapting. Rather, its design as an instrument makes it ideal for the learner to adapt.

A well-designed math tool should be an instrument on which you can learn mathematics, free from the Symbol Barrier. Now imagine we present a student with an orchestra of instruments.

We think this kind of approach is the future of adaptive learning in math and believe we, the edtech community, should choose to go beyond the “low hanging fruit” approaches to adaptive learning that the first movers adopted.


Filed under how teachers teach, school reform policies, technology use

The Bipolar Literature on Technology in U.S. Schools

Reform-minded researchers, techno-enthusiasts, and skeptics in the U.S. have created an immense, convoluted literature on the use and effectiveness of computers in classroom, schools, and districts. It is a literature that is bipolar.

At one end there is the fiercely manic accumulation of success stories of teachers and schools that use devices imaginatively and, according to some researchers, demonstrate small to moderate gains in test scores, increased student engagement, teacher satisfaction, and other desired outcomes (see here and here). These success stories, often teacher surveys and self-reports, clothed as scientific studies (see here and here) beat the drum directly or hum the tune just loud enough for others to hear that these new technologies, especially if they are student-centered (see here) and “personalize learning” (see here), are just short of magical in their engaging disengaged children and youth in learning.

At the other end is the depressing collection of studies that show disappointing results, even losses, in academic achievement and the lack of substantial change in teaching methods during and after use of the new technologies (see here and here). Included are tales told by upset teachers, irritated parents, and disillusioned school board members who authorized technological expenditures (see here, here, and here).

These two poles of manic and depressive research studies replicate the long-term struggle between factions of Progressives who vowed to reform public schools beginning in the early 20th century. The efficiency-driven, teacher-centered wing of these Progressives whipped the experiential, whole-child, student-centered wing then but these losers in the struggle have returned time and again to preach and teach the ideology they hold so dear. Each pole of this spectrum, then, recapitulates the century-old struggle but this time the slogans and phrases are embedded in the language of new technologies. “Project-based learning” and “personalized learning” have been appropriated by current reformers who, still seeking efficiency and productivity in teaching and learning have adopted the language of their historical opponents. Knowing this historical backdrop, however, does not create a middle to this continuum. And that is necessary.

Reducing modestly the bipolarity of this literature are individual and collective case studies (see here), carefully done ethnographies (see here), and meta-analyses of  research studies done over the past half-century to ascertain the effects (or lack thereof) of computers and software upon students and teachers (see here, here, and here).

Even with these meta-analyses, the overall literature oscillating between manic and depressive has yet to develop a midpoint. Inhabiting that midpoint in this bipolar distribution of computer studies would be rigorous (and longitudinal) studies of classrooms, schools, and districts that combine technology exemplars and failures; carefully done classroom and school analyses that go beyond teacher responses on questionnaires to show the pluses and minuses of “blended learning, “project based teaching,” and “personalized learning” (see here, here, and here).  Yet such studies are occasional, not common, entries into the research swamp of technology-in-schools.

So What?

What’s the big deal about a skewed distribution of research studies and non-scientific articles and books? Here are a few reasons.

  1. By making clear that the literature is bipolar, readers can be more discriminating and less promiscuous in assessing claims researchers make and picking and choosing which research studies meet minimum standards of acceptability (e.g., rigorous qualitative, random controlled trials; size and representativeness of samples; brief or extended time of study; sponsored or independent research studies).
  2. Without much of a middle to the spectrum, readers seeking accurate information about the use of computers in public schools, would end up sampling studies at either end of the bipolar continuum and would get a grossly inaccurate picture of computer use and its effects in U.S. schools.
  3. Being aware that the current pushing and shoving over the aims of the new technologies and how they are implemented in school mirror historic struggles among different wings of educational Progressives a century ago can give the current generation of reform-driven policymakers and practitioners  a broader perspective on the fractious rhetorical and policy choices both educators and non-educators face now.


Filed under school reform policies, technology use