GRADING THE DIGITAL SCHOOL
http://www.nytimes.com/2011/09/04/technology/technology-in-schools-faces-questions-onvalue.html?ref=technology
September 3, 2011
In Classroom of Future, Stagnant
Scores
By MATT RICHTEL
CHANDLER, Ariz. — Amy Furman, a seventh-grade English teacher here, roams
among 31 students sitting at their desks or in clumps on the floor. They’re studying
Shakespeare’s “As You Like It” — but not in any traditional way.
In this technology-centric classroom, students are bent over laptops, some blogging
or building Facebook pages from the perspective of Shakespeare’s characters. One
student compiles a song list from the Internet, picking a tune by the rapper Kanye
West to express the emotions of Shakespeare’s lovelorn Silvius.
The class, and the Kyrene School District as a whole, offer what some see as a utopian
vision of education’s future. Classrooms are decked out with laptops, big interactive
screens and software that drills students on every basic subject. Under a ballot
initiative approved in 2005, the district has invested roughly $33 million in such
technologies.
The digital push here aims to go far beyond gadgets to transform the very nature of
the classroom, turning the teacher into a guide instead of a lecturer, wandering
among students who learn at their own pace on Internet-connected devices.
“This is such a dynamic class,” Ms. Furman says of her 21st-century classroom. “I
really hope it works.”
Hope and enthusiasm are soaring here. But not test scores.
Since 2005, scores in reading and math have stagnated in Kyrene, even as statewide
scores have risen.
To be sure, test scores can go up or down for many reasons. But to many education
experts, something is not adding up — here and across the country. In a nutshell:
schools are spending billions on technology, even as they cut budgets and lay off
teachers, with little proof that this approach is improving basic learning.
This conundrum calls into question one of the most significant contemporary
educational movements. Advocates for giving schools a major technological upgrade
— which include powerful educators, Silicon Valley titans and White House
appointees — say digital devices let students learn at their own pace, teach skills
needed in a modern economy and hold the attention of a generation weaned on
gadgets.
Some backers of this idea say standardized tests, the most widely used measure of
student performance, don’t capture the breadth of skills that computers can help
develop. But they also concede that for now there is no better way to gauge the
educational value of expensive technology investments.
“The data is pretty weak. It’s very difficult when we’re pressed to come up with
convincing data,” said Tom Vander Ark, the former executive director for education
at the Bill and Melinda Gates Foundation and an investor in educational technology
companies. When it comes to showing results, he said, “We better put up or shut up.”
And yet, in virtually the same breath, he said change of a historic magnitude is
inevitably coming to classrooms this decade: “It’s one of the three or four biggest
things happening in the world today.”
Critics counter that, absent clear proof, schools are being motivated by a blind faith in
technology and an overemphasis on digital skills — like using PowerPoint and
multimedia tools — at the expense of math, reading and writing fundamentals. They
say the technology advocates have it backward when they press to upgrade first and
ask questions later.
The spending push comes as schools face tough financial choices. In Kyrene, for
example, even as technology spending has grown, the rest of the district’s budget has
shrunk, leading to bigger classes and fewer periods of music, art and physical
education.
At the same time, the district’s use of technology has earned it widespread praise. It is
upheld as a model of success by the National School Boards Association, which in
2008 organized a visit by 100 educators from 17 states who came to see how the
district was innovating.
And the district has banked its future and reputation on technology. Kyrene, which
serves 18,000 kindergarten to eighth-grade students, mostly from the cities of
Tempe, Phoenix and Chandler, uses its computer-centric classes as a way to attract
children from around the region, shoring up enrollment as its local student
population shrinks. More students mean more state dollars.
The issue of tech investment will reach a critical point in November. The district
plans to go back to local voters for approval of $46.3 million more in taxes over seven
years to allow it to keep investing in technology. That represents around 3.5 percent
of the district’s annual spending, five times what it spends on textbooks.
The district leaders’ position is that technology has inspired students and helped
them grow, but that there is no good way to quantify those achievements — putting
them in a tough spot with voters deciding whether to bankroll this approach again.
“My gut is telling me we’ve had growth,” said David K. Schauer, the superintendent
here. “But we have to have some measure that is valid, and we don’t have that.”
It gives him pause.
“We’ve jumped on bandwagons for different eras without knowing fully what we’re
doing. This might just be the new bandwagon,” he said. “I hope not.”
A Dearth of Proof
The pressure to push technology into the classroom without proof of its value has
deep roots.
In 1997, a science and technology committee assembled by President Clinton issued
an urgent call about the need to equip schools with technology.
If such spending was not increased by billions of dollars, American competitiveness
could suffer, according to the committee, whose members included educators like
Charles M. Vest, then president of the Massachusetts Institute of Technology, and
business executives like John A. Young, the former chief executive of HewlettPackard.
To support its conclusion, the committee’s report cited the successes of individual
schools that embraced computers and saw test scores rise or dropout rates fall. But
while acknowledging that the research on technology’s impact was inadequate, the
committee urged schools to adopt it anyhow.
The report’s final sentence read: “The panel does not, however, recommend that the
deployment of technology within America’s schools be deferred pending the
completion of such research.”
Since then, the ambitions of those who champion educational technology have grown
— from merely equipping schools with computers and instructional software, to
putting technology at the center of the classroom and building the teaching around it.
Kyrene had the same sense of urgency as President Clinton’s committee when, in
November 2005, it asked voters for an initial $46.3 million for laptops, classroom
projectors, networking gear and other technology for teachers and administrators.
Before that, the district had given 300 elementary school teachers five laptops each.
Students and teachers used them with great enthusiasm, said Mark Share, the
district’s 64-year-old director of technology, a white-bearded former teacher from the
Bronx with an iPhone clipped to his belt.
“If we know something works, why wait?” Mr. Share told The Arizona Republic the
month before the vote. The district’s pitch was based not on the idea that test scores
would rise, but that technology represented the future.
The measure, which faced no organized opposition, passed overwhelmingly. It means
that property owners in the dry, sprawling flatlands here, who live in apartment
complexes, cookie-cutter suburban homes and salmon-hued mini-mansions, pay on
average $75 more a year in taxes, depending on the assessed value of their homes,
according to the district.
But the proof sought by President Clinton’s committee remains elusive even today,
though researchers have been seeking answers.
Many studies have found that technology has helped individual classrooms, schools
or districts. For instance, researchers found that writing scores improved for eighthgraders in Maine after they were all issued laptops in 2002. The same researchers,
from the University of Southern Maine, found that math performance picked up among
seventh- and eighth-graders after teachers in the state were trained in using the
laptops to teach.
A question plaguing many education researchers is how to draw broader inferences
from such case studies, which can have serious limitations. For instance, in the Maine
math study, it is hard to separate the effect of the laptops from the effect of the
teacher training.
Educators would like to see major trials years in length that clearly demonstrate
technology’s effect. But such trials are extraordinarily difficult to conduct when
classes and schools can be so different, and technology is changing so quickly.
And often the smaller studies produce conflicting results. Some classroom studies
show that math scores rise among students using instructional software, while others
show that scores actually fall. The high-level analyses that sum up these various
studies, not surprisingly, give researchers pause about whether big investments in
technology make sense.
One broad analysis of laptop programs like the one in Maine, for example, found that
such programs are not a major factor in student performance.
“Rather than being a cure-all or silver bullet, one-to-one laptop programs may simply
amplify what’s already occurring — for better or worse,” wrote Bryan Goodwin,
spokesman for Mid-continent Research for Education and Learning, a nonpartisan group
that did the study, in an essay. Good teachers, he said, can make good use of
computers, while bad teachers won’t, and they and their students could wind up
becoming distracted by the technology.
A review by the Education Department in 2009 of research on online courses — which
more than one million K-12 students are taking — found that few rigorous studies had
been done and that policy makers “lack scientific evidence” of their effectiveness.. A
division of the Education Department that rates classroom curriculums has found that
much educational software is not an improvement over textbooks.
Larry Cuban, an education professor emeritus at Stanford University, said the
research did not justify big investments by districts.
“There is insufficient evidence to spend that kind of money. Period, period, period,”
he said. “There is no body of evidence that shows a trend line.”
Some advocates for technology disagree.
Karen Cator, director of the office of educational technology in the United States
Department of Education, said standardized test scores were an inadequate measure
of the value of technology in schools. Ms. Cator, a former executive at Apple
Computer, said that better measurement tools were needed but, in the meantime,
schools knew what students needed.
“In places where we’ve had a large implementing of technology and scores are flat, I
see that as great,” she said. “Test scores are the same, but look at all the other things
students are doing: learning to use the Internet to research, learning to organize their
work, learning to use professional writing tools, learning to collaborate with others.”
For its part, Kyrene has become a model to many by training teachers to use
technology and getting their ideas on what inspires them. As Mr. Share says in the
signature file at the bottom of every e-mail he sends: “It’s not the stuff that counts —
it’s what you do with it that matters.”
So people here are not sure what to make of the stagnant test scores. Many of the
district’s schools, particularly those in more affluent areas, already had relatively high
scores, making it a challenge to push them significantly higher. A jump in students
qualifying for free or reduced-price lunches was largely a result of the recession, not a
shift in the population the district serves, said Nancy Dundenhoefer, its community
relations manager.
Mr. Share, whose heavy influence on more than $7 million a year in technology
spending has made him a power broker, said he did not think demographic changes
were a good explanation.
“You could argue that test scores would be lower without the technology, but that’s a
copout,” he said, adding that the district should be able to deliver some measure of
what he considers its obvious success with technology. “It’s a conundrum.”
Results aside, it’s easy to see why technology is such an easy sell here, given the
enthusiasm surrounding it in some classrooms.
Engaging With Paper
“I start with pens and pencils,” says Ms. Furman, 41, who is short and bubbly and
devours young-adult novels to stay in touch with students. Her husband teaches
eighth grade in the district, and their son and daughter are both students.
At the beginning of the school year, Ms. Furman tries to inspire her students at
Aprende Middle School to write, a task she says becomes increasingly difficult when
students reach the patently insecure middle-school years.
In one class in 2009 she had them draw a heart on a piece of paper. Inside the heart,
she asked them to write the names of things and people dear to them. One girl started
to cry, then another, as the class shared their stories.
It was something Ms. Furman doubted would have happened if the students had been
using computers. “There is a connection between the physical hand on the paper and
the words on the page,” she said. “It’s intimate.”
But, she said, computers play an important role in helping students get their ideas
down more easily, edit their work so they can see instant improvement, and share it
with the class. She uses a document camera to display a student’s paper at the front of
the room for others to dissect.
Ms. Furman said the creative and editing tools, by inspiring students to make quick
improvements to their writing, pay dividends in the form of higher-quality work. Last
year, 14 of her students were chosen as finalists in a statewide essay contest that
asked them how literature had affected their lives. “I was running down the hall,
weeping, saying, ‘Get these students together. We need to tell them they’ve won!’ ”
Other teachers say the technology is the only way to make this generation learn.
“They’re inundated with 24/7 media, so they expect it,” said Sharon Smith, 44, a
gregarious seventh-grade social studies teacher whose classroom is down the hall
from Ms. Furman’s.
Minutes earlier, Ms. Smith had taught a Civil War lesson in a way unimaginable even
10 years ago. With the lights off, a screen at the front of the room posed a question:
“Jefferson Davis was Commander of the Union Army: True or False?”
The 30 students in the classroom held wireless clickers into which they punched their
answers. Seconds later, a pie chart appeared on the screen: 23 percent answered
“True,” 70 percent “False,” and 6 percent didn’t know.
The students hooted and hollered, reacting to the instant poll. Ms. Smith then drew
the students into a conversation about the answers.
The enthusiasm underscores a key argument for investing in classroom technology:
student engagement.
That idea is central to the National Education Technology Plan released by the White
House last year, which calls for the “revolutionary transformation” of schools. The
plan endorses bringing “state-of-the art technology into learning to enable, motivate
and inspire all students.”
But the research, what little there is of it, does not establish a clear link between
computer-inspired engagement and learning, said Randy Yerrick, associate dean of
educational technology at the University of Buffalo.
For him, the best educational uses of computers are those that have no good digital
equivalent. As examples, he suggests using digital sensors in a science class to help
students observe chemical or physical changes, or using multimedia tools to reach
disabled children.
But he says engagement is a “fluffy term” that can slide past critical analysis. And
Professor Cuban at Stanford argues that keeping children engaged requires an
environment of constant novelty, which cannot be sustained.
“There is very little valid and reliable research that shows the engagement causes or
leads to higher academic achievement,” he said.
Instruct or Distract?
There are times in Kyrene when the technology seems to allow students to disengage
from learning: They are left at computers to perform a task but wind up playing
around, suggesting, as some researchers have found, that computers can distract and
not instruct.
The 23 kindergartners in Christy Asta’s class at Kyrene de las Brisas are broken into
small groups, a common approach in Kyrene. A handful stand at desks, others sit at
computers, typing up reports.
Xavier Diaz, 6, sits quietly, chair pulled close to his Dell laptop, playing “Alien
Addition.” In this math arcade game, Xavier controls a pod at the bottom of the
screen that shoots at spaceships falling from the sky. Inside each ship is a pair of
numbers. Xavier’s goal is to shoot only the spaceship with numbers that are the sum
of the number inside his pod.
But Xavier is just shooting every target in sight. Over and over. Periodically, the game
gives him a message: “Try again.” He tries again.
“Even if he doesn’t get it right, it’s getting him to think quicker,” says the teacher, Ms.
Asta. She leans down next to him: “Six plus one is seven. Click here.” She helps him
shoot the right target. “See, you shot him.”
Perhaps surprisingly given the way young people tend to gravitate toward gadgets,
students here seem divided about whether they prefer learning on computers or
through more traditional methods.
In a different class, Konray Yuan and Marisa Guisto, both 7, take turns touching
letters on the interactive board on the wall. They are playing a spelling game, working
together to spell the word “cool.” Each finds one of the letters in a jumbled grid,
touching them in the proper order.
Marisa says there isn’t a difference between learning this way and learning on paper.
Konray prefers paper, he says, because you get extra credit for good penmanship.
But others, particularly older students, say they enjoy using the technology tools. One
of Ms. Furman’s students, Julia Schroder, loved building a blog to write about
Shakespeare’s “As You Like It.”
In another class, she and several classmates used a video camera to film a skit about
Woodrow Wilson’s 14-point speech during World War I — an approach she preferred
to speaking directly to the class.
“I’d be pretty bummed if I had to do a live thing,” she said. “It’s nerve-racking.”
Teachers vs. Tech
Even as students are getting more access to computers here, they are getting less
access to teachers.
Reflecting budget cuts, class sizes have crept up in Kyrene, as they have in many
places. For example, seventh-grade classes like Ms. Furman’s that had 29 to 31
students grew to more like 31 to 33.
“You can’t continue to be effective if you keep adding one student, then one student,
then one student,” Ms. Furman said. “I’m surprised parents aren’t going into the
classrooms saying ‘Whoa.’ ”
Advocates of high-tech classrooms say computers are not intended to replace
teachers. But they do see a fundamental change in the teacher’s role. Their often-cited
mantra is that teachers should go from being “a sage on the stage to a guide on the
side.”
And they say that, technology issues aside, class sizes can in fact afford to grow
without hurting student performance.
Professor Cuban at Stanford said research showed that student performance did not
improve significantly until classes fell under roughly 15 students, and did not get
much worse unless they rose above 30.
At the same time, he says bigger classes can frustrate teachers, making it hard to
attract and retain talented ones.
In Kyrene, growing class sizes reflect spending cuts; the district’s maintenance and
operation budget fell to $95 million this year from $106 million in 2008. The district
cannot use the money designated for technology to pay for other things. And the
teachers, who make roughly $33,000 to $57,000 a year, have not had a raise since
2008.
Many teachers have second jobs, some in restaurants and retail, said Erin Kirchoff,
president of the Kyrene Education Association, the teacher’s association. Teachers
talk of being exhausted from teaching all day, then selling shoes at the mall.
Ms. Furman works during the summer at the Kyrene district offices. But that job is
being eliminated in 2014, and she is worried about the income loss.
“Without it, we don’t go on vacation,” she said.
Money for other things in the district is short as well. Many teachers say they
regularly bring in their own supplies, like construction paper.
“We have Smart Boards in every classroom but not enough money to buy copy paper,
pencils and hand sanitizer,” said Nicole Cates, a co-president of the Parent Teacher
Organization at Kyrene de la Colina, an elementary school. “You don’t go buy a new
outfit when you don’t have enough dinner to eat.”
But she loves the fact that her two children, a fourth-grader and first-grader, are
learning technology, including PowerPoint and educational games.
To some who favor high-tech classrooms, the resource squeeze presents an
opportunity. Their thinking is that struggling schools will look for more efficient ways
to get the job done, creating an impetus to rethink education entirely.
“Let’s hope the fiscal crisis doesn’t get better too soon. It’ll slow down reform,” said
Tom Watkins, the former superintendent for the Michigan schools, and now a
consultant to businesses in the education sector.
Clearly, the push for technology is to the benefit of one group: technology companies.
The Sellers
It is 4:30 a.m. on a Tuesday. Mr. Share, the director of technology at Kyrene and
often an early riser, awakens to the hard sell. Awaiting him at his home computer are
six pitches from technology companies.
It’s just another day for the man with the checkbook.
“I get one pitch an hour,” he said. He finds most of them useless and sometimes
galling: “They’re mostly car salesmen. I think they believe in the product they’re
selling, but they don’t have a leg to stand on as to why the product is good or bad.”
Mr. Share bases his buying decisions on two main factors: what his teachers tell him
they need, and his experience. For instance, he said he resisted getting the interactive
whiteboards sold as Smart Boards until, one day in 2008, he saw a teacher trying to
mimic the product with a jury-rigged projector setup.
“It was an ‘Aha!’ moment,” he said, leading him to buy Smart Boards, made by a
company called Smart Technologies.
He can make that kind of decision because he has money — and the vendors know it.
Technology companies track which districts get federal funding and which have
passed tax assessments for technology, like Kyrene.
This is big business. Sales of computer software to schools for classroom use were
$1.89 billion in 2010. Spending on hardware is more difficult to measure, researchers
say, but some put the figure at five times that amount.
The vendors relish their relationship with Kyrene.
“I joke I should have an office here, I’m here so often,” said Will Dunham, a salesman
for CCS Presentation Systems, a leading reseller of Smart Boards in Arizona.
Last summer, the district paid $500,000 to CCS to replace ceiling-hung projectors in
400 classrooms. The alternative was to spend $100,000 to replace their aging bulbs,
which Mr. Share said were growing dimmer, causing teachers to sometimes have to
turn down the lights to see a crisp image.
Mr. Dunham said the purchase made sense because new was better. “I could take a
used car down to the mechanic and get it all fixed up and still have a used car.”
But Ms. Kirchoff, the president of the teachers’ association, is furious. “My projector
works just fine,” she said. “Give me Kleenex, Kleenex, Kleenex!”
The Parents
Last November, Kyrene went back to voters to ask them to pay for another seven
years of technology spending in the district. The previous measure from 2005 will not
expire for two years. But the district wanted to get ahead of the issue, and leave
wiggle room just in case the new measure didn’t pass.
It didn’t. It lost by 96 votes out of nearly 50,000 cast. Mr. Share and others here said
they attributed the failure to poor wording on the ballot that made it look like a new
tax increase, rather than the continuation of one.
They say they will not make the same wording mistake this time. And they say the
burden on taxpayers is modest.
“It’s so much bang for the buck,” said Jeremy Calles, Kyrene’s interim chief financial
officer. For a small investment, he said, “we get state-of-the-art technology.”
Regardless, some taxpayers have already decided that they will not vote yes.
“When you look at the big picture, it’s hard to say ‘yes, spend more on technology’
when class sizes increase,” said Kameron Bybee, 34, who has two children in district
schools. “The district has made up its mind to go forward with the technologically
advanced path. Come hell or high water.”
Other parents feel conflicted. Eduarda Schroder, 48, whose daughter Julia was in Ms.
Furman’s English class, worked on the political action committee last November to
push through an extension of the technology tax. Computers, she says, can make
learning more appealing. But she’s also concerned that test scores haven’t gone up.
She says she is starting to ask a basic question. “Do we really need technology to
learn?” she said. “It’s a very valid time to ask the question, right before this goes on
the ballot.”
http://www.nytimes.com/2011/10/09/technology/a-classroom-software-boom-but-mixed-resultsdespite-the-hype.html?_r=1&ref=gradingthedigitalschool&pagewanted=print
October 8, 2011
Inflating the Software Report Card
By TRIP GABRIEL and MATT RICHTEL
The Web site of Carnegie Learning, a company started by scientists at Carnegie Mellon
University that sells classroom software, trumpets this promise: “Revolutionary Math
Curricula. Revolutionary Results.”
The pitch has sounded seductive to thousands of schools across the country for more
than a decade. But a review by the United States Department of Education last year
would suggest a much less alluring come-on: Undistinguished math curricula.
Unproven results.
The federal review of Carnegie Learning’s flagship software, Cognitive Tutor, said the
program had “no discernible effects” on the standardized test scores of high school
students. A separate 2009 federal look at 10 major software products for teaching
algebra as well as elementary and middle school math and reading found that nine of
them, including Cognitive Tutor, “did not have statistically significant effects on test
scores.”
Amid a classroom-based software boom estimated at $2.2 billion a year, debate
continues to rage over the effectiveness of technology on learning and how best to
measure it. But it is hard to tell that from technology companies’ promotional
materials.
Many companies ignore well-regarded independent studies that test their products’
effectiveness. Carnegie’s Web site, for example, makes no mention of the 2010
review, by the Education Department’s What Works Clearinghouse, which analyzed 24
studies of Cognitive Tutor’s effectiveness but found that only four of those met high
research standards. Some firms misrepresent research by cherry-picking results and
promote surveys or limited case studies that lack the scientific rigor required by the
clearinghouse and other authorities.
“The advertising from the companies is tremendous oversell compared to what they
can actually demonstrate,” said Grover J. Whitehurst, a former director of the Institute
of Education Sciences, the federal agency that includes What Works.
School officials, confronted with a morass of complicated and sometimes conflicting
research, often buy products based on personal impressions, marketing hype or faith
in technology for its own sake.
“They want the shiny new one,” said Peter Cohen, chief executive of Pearson School, a
leading publisher of classroom texts and software. “They always want the latest, when
other things have been proven the longest and demonstrated to get results.”
Carnegie, one of the most respected of the educational software firms, is hardly alone
in overpromising or misleading. The Web site of Houghton Mifflin Harcourt says that
“based on scientific research, Destination Reading is a powerful early literacy and
adolescent literacy program,” but it fails to mention that it was one of the products
the Department of Education found in 2009 not to have statistically significant
effects on test scores.
Similarly, Pearson’s Web site cites several studies of its own to support its claim
that Waterford Early Learning improves literacy, without acknowledging the same
2009 study’s conclusion that it had little impact.
And Intel, in a Web document urging schools to buy computers for every student,
acknowledges that “there are no longitudinal, randomized trials linking eLearning to
positive learning outcomes.” Yet it nonetheless argues that research shows that
technology can lead to more engaged and economically successful students, happier
teachers and more involved parents.
“To compare this public relations analysis to a carefully constructed research study is
laughable,” said Alex Molnar, professor of education at the National Education Policy
Center at the University of Colorado. “They are selling their wares.”
Carnegie officials say 600,000 students in 44 states use its products, many taking
teacher-led classes three times a week with Carnegie-provided workbooks and
spending the other two class periods in computer labs using Cognitive Tutor. The full
curriculum can cost nearly three times as much as a typical textbook over six years.
Officials declined to release annual revenue figures, but Carnegie Learning was
acquired in August for $75 million by the parent of the for-profit University of
Phoenix. Carnegie Mellon University, which had retained ownership of the Cognitive
Tutor software and licensed it to Carnegie Learning, earned an additional $21.5
million from the sale.
Steve Ritter, a founder and the chief scientist of Carnegie Learning, said there were
flaws in the What Works Clearinghouse evaluations of Cognitive Tutor and disputed
the Education Department’s judgment of what makes a worthy study.
“What you want to focus on is more of the why,” he said, “and less of a horse race to
find out what works and doesn’t.”
A Carnegie spokeswoman, Mary Murrin, said in a statement that the company used
“the data from all studies with varying outcomes to continuously improve our
programs.”
Karen Billings, a vice president of the Software and Information Industry Association — a
trade group representing many education companies — said the problem was not that
companies overpromise, but that schools often do not properly deploy the products
or train teachers to use them. Ms. Billings’s group helped design the field trials, in 132
schools, for the landmark 2009 government study of 10 software products, which was
ordered by Congress and cost $15 million.
Then came the deflating results. The industry “became very hostile,” recalled Mr.
Whitehurst, now director of education policy at the Brookings Institution. “It seems
to me,” he added, “ ‘hypocrisy’ is the right word for loving something until the results
are not what you expect.”
The Hard Sell
Shelly Allen, the math coordinator for public schools in Augusta, Ga., has seen a lot of
curriculum salespeople pass through. She is wary of their sweet words and hard sell.
In June, when representatives from Carnegie Learning visited, Dr. Allen warned: “I
just want everybody to know I grew up here. I graduated from here. My children go to
school here. When you guys get back where you live, our kids have to still be able to
reach goals we set.”
Augusta is famous for its magnolia-shaded National Golf Club, host to the Masters
Tournament, but its public schools are typical of struggling urban districts. Threequarters of the 32,000 students in the district, Richmond County, are black, and 72
percent are poor enough to qualify for the federal lunch program. The mean SAT
math score last year was 443, below Georgia’s mean of 490 and the nation’s 516.
Six years ago, the district adopted Cognitive Tutor for about 3,000 students at risk of
failing, paying $101,500 annually to use it. As students work through problems, the
computer analyzes their weaknesses and serves up new items until they grasp the
skill and are allowed to move on. To a student, the promotional materials say, it feels
“as if the software is getting to know her and supporting her like a tutor.”
So when the screen says: “You are saving to buy a bicycle. You have $10, and each day
you are able to save $2,” the student must convert the word problem into an algebraic
expression. If he is stumped, he can click on the “Hint” button.
“Define a variable for the time from now,” the software advises. Still stumped? Click
“Next Hint.”
“Use x to represent the time from now.” Aha. The student types “2x+10.”
The software likes this and moves on to highlight a series of questions in green,
beginning with, “How many more days must you save to buy a bike that costs $60?”
Using his 2x+10 formula, the student enters “25.”
After solving several questions of this sort and plotting them on a graph, the student
would click “Skillometer” to see how he had fared. A series of forest-green bars would
show that he did well labeling axes for his graph, but not so well writing the initial
formula.
Moving on, Cognitive Tutor would bump him down to an easier problem: “A skier
noticed that he can complete a run in about 30 minutes (half an hour).” The
expression relating ski runs to time would be 2x, with x representing hours.
“Immediate feedback,” Carnegie Learning explains on its Web site, “enables the
student to self-correct and leads to more effective learning.”
Augusta officials liked the program enough that when concerns arose last winter that
many 11th graders were not on track to pass a new state graduation test, the district
asked to expand the software’s use to all 9,400 of its high school students. The
company agreed to provide access for no additional charge — temporarily.
“As a company, it makes sense to give you the opportunity to prove it works for all
students,” Anita Sprayberry, a regional sales manager, told school leaders. That way,
she said, “We can talk about a bigger sale.”
Going forward, Ms. Sprayberry said, the cost would be about $34,000 for each of the
district’s 11 high schools.
In a recent interview, Dr. Allen said she was familiar with the What Works
Clearinghouse, but not its 2010 finding that Cognitive Tutor did not raise test scores
more than textbooks.
Though the clearinghouse is intended to help school leaders choose proven
curriculum, a 2010 Government Accountability Office survey of district officials found
that 58 percent of them had never heard of What Works, never mind consulted its
reviews.
“Decisions are made on marketing, on politics, on personal preference,” said Robert
A. Slavin, director of the Center for Research and Reform in Education at Johns
Hopkins University. “An intelligent, caring principal who’d never buy a car without
looking at Consumer Reports, when they plunk down serious money to buy a
curriculum, they don’t even look at the evidence.”
Evaluating Curriculums
Founded in 1998 by cognitive and computer scientists along with math teachers,
Carnegie Learning is proud of its academic heritage, and many education researchers
consider it a model of rigor and transparency.
One founder, John R. Anderson, received the 2011 Benjamin Franklin Medal in
Computer and Cognitive Science for work on how humans perceive, learn and reason.
The company’s Web site promises that its curriculums “provide the research-based
foundation for proven results,” citing “success stories” from around the country.
At Dundalk Middle School in Baltimore County, Md., for example, Carnegie Learning
says that Cognitive Tutor led to an increase in the passing rate on a state assessment,
to 86 percent in 2004 from 49 percent in 2002. What it does not say is that the rate
remained at 85 percent last year, even though Dundalk dropped Cognitive Tutor in
2007 because of difficulties arranging lab time.
That is why many academics dismiss case studies: it is too easy for slices of data to be
taken out of context, or for correlation to be confused with causation.
Instead, the gold standard of education research is a field trial in which similar
groups of students are randomly assigned to classes where one uses the curriculum
and the other does not.
The Carnegie Web site lists five such trials and says they all show positive results for
Cognitive Tutor.
Three of these studies, however, were rejected by the What Works Clearinghouse for
flaws in their design; in a fourth, the clearinghouse identified a problem with part of
the study — the part that purported to show benefits. One of the rejected studies had
found that users of Cognitive Tutor in 10 Miami high schools scored better on Florida
state exams than a control group, but the clearinghouse found that the students being
compared were not equivalent.
“The entire ‘effect’ of Cognitive Tutor possibly can be traced to other factors,” said
Mark Dynarski, a former director of the clearinghouse, “and the way in which the
research was carried out does not allow one to know if this is the case.”
Dr. Ritter, Carnegie’s chief scientist, noted that the clearinghouse’s 2010 review was
limited to high schools and that a year earlier it found that Cognitive Tutor had
“potentially positive effects” in middle school.
The middle school finding rested on one study, out of 14 reviewed. That study is
featured prominently on the Carnegie Web site, which omits mention of two others
that the Education Department judged to be well designed but showed no benefits.
Dr. Ritter said he had excluded those studies, in Hawaii and Virginia, because the
students had not used Cognitive Tutor precisely as the company intended. The
researcher who did the Hawaii study, Denis Newman, said it reflected how Cognitive
Tutor was used in the real world.
Dr. Newman is also the author of research guidelines for the Software and
Information Industry Association, where Dr. Ritter sits on the education research
working group. One of those guidelines states, “An expectation in the scientific
community is that research findings are made available regardless of the result.”
Karen Cator, a former Apple executive who directs the Office of Educational
Technology at the Department of Education, said the clearinghouse reports on
software should be “taken with a grain of salt” because they rely on standardized test
scores. Those tests, Ms. Cator said, cannot gauge some skills that technology teaches,
like collaboration, multimedia and research.
Ms. Cator’s office is developing a new framework to measure the educational value of
technology, but she advised schools and districts not to wait to invest in software like
Cognitive Tutor.
“They know what their students need to know and what they need to be able to do,”
she said.
Real-Time Assessments
In Augusta, Dr. Allen, the math coordinator, said her district did not have the means
to study the effectiveness of Cognitive Tutor formally. But she and her staff saw that
low-achieving students who used it were able to join mainstream classes. And
teachers appreciated the way the software transmits assessments in real time to
Carnegie Learning, then kicks back a report indicating the strengths and weaknesses
of each student.
Teachers “just didn’t know, skill by skill, the same type of data they are getting now,”
Dr. Allen said.
On the other hand, when the new state math test was given in March, 27 percent of
the district’s 11th graders did not pass, which Dr. Allen described as “something that
makes us not real excited.”
At the June meeting with Carnegie Learning’s sales team, Dr. Allen said Cognitive
Tutor could be worthwhile if the district, which has recently cut $7 million from its
budget and furloughed employees for nine days, could scrape together the financing.
“Our negotiations are intense because we don’t have any money,” she said to laughter
around the table.
In Georgia, where the state negotiates prices with publishers, an annual license for
Cognitive Tutor software is $32 per student, and the workbook, which must be
replaced annually, is $24 — for a total of $336 over six years, a typical lifespan of a
math textbook that costs about $120.
Ultimately, Dr. Allen’s district did not have the money, so she focused on getting the
most out of her staff. “Giving them the right tools and resources certainly helps,” she
said, “but our teachers are the ones making that difference.”
Gregory W. Capelli, co-chief executive of the Apollo Group, which runs the 400,000student University of Phoenix and bought Carnegie Learning this summer, said his
company first ran its own pilot project with the software and also examined
independent research.
But Mr. Capelli, like others, relied at least in part on personal experience.
“My daughter, who’s in eighth grade, used this product,” he said.
“She would do very well” in some lessons “and not in others,” Mr. Capelli said. “What
I liked about it is that once she got it, it would allow her to go on to the next part of
the tree.”
This article has been revised to reflect the following correction:
Correction: October 13, 2011
An article on Saturday about technology companies that produce software for use in classrooms
misspelled the given name of a researcher whose study showed that the use of Cognitive Tutor, one
such program, produced no benefits. He is Denis Newman, not Dennis.
http://www.nytimes.com/2011/10/23/technology/at-waldorf-school-in-silicon-valley-technology-canwait.html
October 22, 2011
A Silicon Valley School That Doesn’t
Compute
By MATT RICHTEL
LOS ALTOS, Calif. — The chief technology officer of eBay sends his children to a
nine-classroom school here. So do employees of Silicon Valley giants like Google,
Apple, Yahoo and Hewlett-Packard.
But the school’s chief teaching tools are anything but high-tech: pens and paper,
knitting needles and, occasionally, mud. Not a computer to be found. No screens at
all. They are not allowed in the classroom, and the school even frowns on their use at
home.
Schools nationwide have rushed to supply their classrooms with computers, and
many policy makers say it is foolish to do otherwise. But the contrarian point of view
can be found at the epicenter of the tech economy, where some parents and educators
have a message: computers and schools don’t mix.
This is the Waldorf School of the Peninsula, one of around 160 Waldorf schools in the
country that subscribe to a teaching philosophy focused on physical activity and
learning through creative, hands-on tasks. Those who endorse this approach say
computers inhibit creative thinking, movement, human interaction and attention
spans.
The Waldorf method is nearly a century old, but its foothold here among the digerati
puts into sharp relief an intensifying debate about the role of computers in education.
“I fundamentally reject the notion you need technology aids in grammar school,” said
Alan Eagle, 50, whose daughter, Andie, is one of the 196 children at the Waldorf
elementary school; his son William, 13, is at the nearby middle school. “The idea that
an app on an iPad can better teach my kids to read or do arithmetic, that’s ridiculous.”
Mr. Eagle knows a bit about technology. He holds a computer science degree from
Dartmouth and works in executive communications at Google, where he has written
speeches for the chairman, Eric E. Schmidt. He uses an iPad and a smartphone. But
he says his daughter, a fifth grader, “doesn’t know how to use Google,” and his son is
just learning. (Starting in eighth grade, the school endorses the limited use of
gadgets.)
Three-quarters of the students here have parents with a strong high-tech connection.
Mr. Eagle, like other parents, sees no contradiction. Technology, he says, has its time
and place: “If I worked at Miramax and made good, artsy, rated R movies, I wouldn’t
want my kids to see them until they were 17.”
While other schools in the region brag about their wired classrooms, the Waldorf
school embraces a simple, retro look — blackboards with colorful chalk, bookshelves
with encyclopedias, wooden desks filled with workbooks and No. 2 pencils.
On a recent Tuesday, Andie Eagle and her fifth-grade classmates refreshed their
knitting skills, crisscrossing wooden needles around balls of yarn, making fabric
swatches. It’s an activity the school says helps develop problem-solving, patterning,
math skills and coordination. The long-term goal: make socks.
Down the hall, a teacher drilled third-graders on multiplication by asking them to
pretend to turn their bodies into lightning bolts. She asked them a math problem —
four times five — and, in unison, they shouted “20” and zapped their fingers at the
number on the blackboard. A roomful of human calculators.
In second grade, students standing in a circle learned language skills by repeating
verses after the teacher, while simultaneously playing catch with bean bags. It’s an
exercise aimed at synchronizing body and brain. Here, as in other classes, the day can
start with a recitation or verse about God that reflects a nondenominational emphasis
on the divine.
Andie’s teacher, Cathy Waheed, who is a former computer engineer, tries to make
learning both irresistible and highly tactile. Last year she taught fractions by having
the children cut up food — apples, quesadillas, cake — into quarters, halves and
sixteenths.
“For three weeks, we ate our way through fractions,” she said. “When I made enough
fractional pieces of cake to feed everyone, do you think I had their attention?”
Some education experts say that the push to equip classrooms with computers is
unwarranted because studies do not clearly show that this leads to better test scores
or other measurable gains.
Is learning through cake fractions and knitting any better? The Waldorf advocates
make it tough to compare, partly because as private schools they administer no
standardized tests in elementary grades. And they would be the first to admit that
their early-grade students may not score well on such tests because, they say, they
don’t drill them on a standardized math and reading curriculum.
When asked for evidence of the schools’ effectiveness, the Association of Waldorf
Schools of North America points to research by an affiliated group showing that 94
percent of students graduating from Waldorf high schools in the United States
between 1994 and 2004 attended college, with many heading to prestigious
institutions like Oberlin, Berkeley and Vassar.
Of course, that figure may not be surprising, given that these are students from
families that value education highly enough to seek out a selective private school, and
usually have the means to pay for it. And it is difficult to separate the effects of the
low-tech instructional methods from other factors. For example, parents of students
at the Los Altos school say it attracts great teachers who go through extensive training
in the Waldorf approach, creating a strong sense of mission that can be lacking in
other schools.
Absent clear evidence, the debate comes down to subjectivity, parental choice and a
difference of opinion over a single world: engagement. Advocates for equipping
schools with technology say computers can hold students’ attention and, in fact, that
young people who have been weaned on electronic devices will not tune in without
them.
Ann Flynn, director of education technology for the National School Boards Association,
which represents school boards nationwide, said computers were essential. “If
schools have access to the tools and can afford them, but are not using the tools, they
are cheating our children,” Ms. Flynn said.
Paul Thomas, a former teacher and an associate professor of education at Furman
University, who has written 12 books about public educational methods, disagreed,
saying that “a spare approach to technology in the classroom will always benefit
learning.”
“Teaching is a human experience,” he said. “Technology is a distraction when we need
literacy, numeracy and critical thinking.”
And Waldorf parents argue that real engagement comes from great teachers with
interesting lesson plans.
“Engagement is about human contact, the contact with the teacher, the contact with
their peers,” said Pierre Laurent, 50, who works at a high-tech start-up and formerly
worked at Intel and Microsoft. He has three children in Waldorf schools, which so
impressed the family that his wife, Monica, joined one as a teacher in 2006.
And where advocates for stocking classrooms with technology say children need
computer time to compete in the modern world, Waldorf parents counter: what’s the
rush, given how easy it is to pick up those skills?
“It’s supereasy. It’s like learning to use toothpaste,” Mr. Eagle said. “At Google and all
these places, we make technology as brain-dead easy to use as possible. There’s no
reason why kids can’t figure it out when they get older.”
There are also plenty of high-tech parents at a Waldorf school in San Francisco and
just north of it at the Greenwood School in Mill Valley, which doesn’t have Waldorf
accreditation but is inspired by its principles.
California has some 40 Waldorf schools, giving it a disproportionate share — perhaps
because the movement is growing roots here, said Lucy Wurtz, who, along with her
husband, Brad, helped found the Waldorf high school in Los Altos in 2007. Mr.
Wurtz is chief executive of Power Assure, which helps computer data centers reduce
their energy load.
The Waldorf experience does not come cheap: annual tuition at the Silicon Valley
schools is $17,750 for kindergarten through eighth grade and $24,400 for high
school, though Ms. Wurtz said financial assistance was available. She says the typical
Waldorf parent, who has a range of elite private and public schools to choose from,
tends to be liberal and highly educated, with strong views about education; they also
have a knowledge that when they are ready to teach their children about technology
they have ample access and expertise at home.
The students, meanwhile, say they don’t pine for technology, nor have they gone
completely cold turkey. Andie Eagle and her fifth-grade classmates say they
occasionally watch movies. One girl, whose father works as an Apple engineer, says
he sometimes asks her to test games he is debugging. One boy plays with flightsimulator programs on weekends.
The students say they can become frustrated when their parents and relatives get so
wrapped up in phones and other devices. Aurad Kamkar, 11, said he recently went to
visit cousins and found himself sitting around with five of them playing with their
gadgets, not paying attention to him or each other. He started waving his arms at
them: “I said: ‘Hello guys, I’m here.’ ”
Finn Heilig, 10, whose father works at Google, says he liked learning with pen and
paper — rather than on a computer — because he could monitor his progress over the
years.
“You can look back and see how sloppy your handwriting was in first grade. You can’t
do that with computers ’cause all the letters are the same,” Finn said. “Besides, if you
learn to write on paper, you can still write if water spills on the computer or the
power goes out.”