The School Board President
Called…
Can You Justify Technology Next
Tuesday?
A Little Philosophy…
 Hegel and educational technology
 Thesis
 Antithesis
 Synthesis
Jane Healy, Failure to
Connect



Blames technology for bad pedagogy and poor
parenting
Dismisses most positive research as sponsored
by computer corporations or conducted by
wanna-be educational consultants (p. 22) then
quotes a study sponsored by music educators:
“Although one might wish for a more objective
funding source, the results have been
provocative.” (p. 230)
Accuses techno-pushers of “hysteria” then says,
“If you don’t limit computer time, don’t be
surprised when he starts to have attention,
learning, or social problems.” (p. 226)
Clifford Stoll
 Silicon Snake Oil and High Tech Heretic
 Seems to advocate no computers in
classrooms
 Most famous quote: “No computer can
teach what a walk through a pine forest
feels like. Sensation has no substitute.”
(p. 138)
 Thinks field trips cost $100 or $200
Todd Oppenheimer
 “The Computer Delusion,” Atlantic
Monthly, July 1997
 Suggests that schools are buying
computers at the expense of other
programs
 Questions the need to teach computer
skills
Todd Oppenheimer
Bonnie Bracey Replies
 Vanderbilt University
 Indiana’s Buddy System
 Stevens Institute of Technology
 Israeli researchers
Todd Oppenheimer
Quotes Stoll: “Computers in classrooms
are the filmstrips of the 1990s. We loved
them because we didn’t have to think for
an hour, teachers loved them because
they didn’t have to teach, and parents
loved them because it showed their
schools were high-tech. But no learning
happened.”
Larry Cuban
 “Computer Meets Classroom: Classroom
Wins,” Teachers College Record, Winter 1993
 Technology historian
 Begins with the question: “Why is electronic
technology used far less on a daily basis in
classrooms than in other organizations?”
 Only recently has technology been part of
education reform rhetoric
Larry Cuban: 3 Scenarios
 Technophile: Electronic Schools of the
Future Now
 Preservationist: Maintaining While
Improving Schooling
 Cautious Optimist: Slow Growth of
Hybrid Schools and Classrooms
Adopting Technology
35
22
26
16
Years
Telephone
Technology
Television
Radio
PCs
7
Internet
Y
e
a
r
s
40
35
30
25
20
15
10
5
0
Number of Years to Reach 25% of
Households
David Dwyer and ACOT
Impact of Technology Over Time
 Adoption Stage
 Struggles
 May
revert to traditional methods
 Adaptation Stage
 Clear
goals lead to improvements
 Less use of content software and more of
tools
David Dwyer and ACOT
Impact of Technology Over Time
 Appropriation Stage
 Comfortable
enough to forget the
technology
 Changed to “constructivist” methods
 Innovation Stage
 Opened
up instructionally
 Project-based learning
Dwyer’s Vision
 Technology is unlike other educational
innovations because it’s happening
outside the schools, too.
 Recommends a gradual approach
 Must focus on environment and
instruction
President’s Committee of Advisors on
Science and Technology
 Report to the President on the Use of Technology to
Strengthen K-12 Education in the United States,
March 1997
 Due to high costs of hardware and software, all
involved have an interest in evidence about the
efficacy and cost effectiveness of technology
 Research could be used to maximize the ratio of
benefit to cost.
 Small improvements in this ratio can have a real
impact on local, state, and federal education budgets
Jamie McKenzie
 Program evaluation is necessary
 What
kind of investment will pay the
greatest learning dividends?
 What are the desired student outcomes?
Job Skills
 Improvement of test scores
 Curriculum integration

Jamie McKenzie
Program Design and Implementation
 Identify research questions worth asking
 Commission an evaluation design and
explore the significance of findings
 Suggest program changes as data
warrants them
Jamie McKenzie
Formative Evaluation
 The collection of data as programs
proceed
 Collect relevant data
 Quantitative
(numerical)
 Qualitative (descriptive)
 Reflective practice
 Include staff members to avoid fears of
“accountability”
Jamie McKenzie
Summative Evaluation
 Collection of data to judge the overall
success of a program
 Way to reassure governing bodies that
programs are being effective
 Bang
for the buck
 Market your new technologies to tax
payers who feel that the way they
learned was good enough for them…
Jamie McKenzie
Why So Little Evaluation?
 Do not have the expertise or resources
to conduct evaluations
 Want to protect new programs from
scrutiny
 Accountability is a dirty word
 Little understanding of formative
evaluation
 Vendors have a lot at stake
Jamie McKenzie
Why So Little Evaluation?
 Little respect for research
 Technology is seen as capital rather
than program
 Evaluation requires clear goals
 Good evaluation design standards may
create political problems
 Innovation conflicts with need for
stability
Be Your Own PR Person
 Keep track of what’s going on
 Invite teachers to the school board
meeting to show off projects
 Enter contests
 Be proactive and take the time to
document what’s happening
Do Your Own Evaluation
 CEO Forum (The “STaR Chart” People)
 LoTi (Nat’l Business Education Alliance)
 U.S. Department of Education
 Greeneville City School System
 K-5
Survey
 Respondents were supportive
 Had problems with management
Technology-Enriched
Teaching and Learning
Multi-District Project
The Balanced Performance Model
The Indicators
 Communication
Within the School System
Within the Broader Community
 Teaching and Learning
Integration with Curriculum
Instructional Strategies and Techniques
Assessment of Student Learning
The Balanced Performance Model
The Indicators
 Planning and Information Management
 Organizational and Professional
Development
 Access, Reliability, and Equity
 Vision and Leadership
Balanced Performance Model
Levels of Performance
 Entry: A system point of view does not exist.
It is characterized by a lack of clarity about what
to do, inconsistency in implementation, and
variations in results.
 Beginning: A system point of view is being
formulated. It is characterized by the
establishment of basic procedures, limited
implementation, and anecdotal results.
 Proficient: A system point of view is in place.
It is characterized by sound plans and policies,
widespread implementation, and results that are
substantiated by data.
Balanced Performance Model
Levels of Performance
 Exemplary: A system point of view is firmly
in place. It is characterized by comprehensive
plans and policies, integrated and widespread
implementation, and results that are
substantiated by multiple forms of data.
 Optimal: Most favorable or desirable state. A
system point of view prevails. It is
characterized by comprehensive and integrated
plans and policies, full implementation, and
results that are substantiated by local, national
and international measures of excellence.
Levels of Performance and Phases of
Performance Development
Approach
Implementation
Results
Lack of clarity. No plans
or processes developed.
Inconsistent use of
technology.
Wide variation in results.
Little evidence of
improvement.
Comprehensive plans.
Technology is fully
integrated.
Technology is fully
supported and
consistently used.
Strong evidence from
multiple sources of
positive results for all
students.
Benchmarking the Data
The Wheatfield
Approach
Implement.
Results
Communication:
Within the Schools
Communication:
Within the Broader Community
Teaching and Learning:
Integration with Curriculum
Entry
Teaching and Learning:
Instructional Strategies
Beginning
Teaching and Learning:
Assessment of Student Learning
Proficient
Planning and
Information Management
Exemplary
Organizational and
Professional Development
Optimal
Access, Equity
and Reliability
Vision and
Leadership
Technology & Learning
June 2000
 Norris, Smolka and Soloway
 Convergent Analysis web site
 “Fodor’s
Guide to Research Literature”
 Profiles on the use of technology in writing
and science education
How to Use Research
 Teacher education
 Policy decisions (money)
 Curriculum ideas (best practices)
 Motivation and affirmation
Problems with Research
 Identifying relevant articles
 Accessing relevant articles
 Reading articles
 Jargon
 Bias
 Thin
descriptions
Suggestions
 Have a good question
 Use lit reviews
 Avoid obscure things
 Be willing to work a little…
Jay Sivin-Kachala
 Ask the right questions….
 Which is better technology or no
technology? (too broad)
 Under what conditions is technology
valuable?
 Sample Media Questions
 Do
Computers Make Kids Smarter?
 Why Computer Make Bad Teachers
Jay Sivin-Kachala
 What software is being used, how well
is it matched to the school’s curriculum
objects, how well is it matched to the
needs and learning characteristics of
the students, and what role is the
teacher playing before, during, and after
its use?
Sivin-Kachala
 Focuses on need for goals besides
having computers in every classroom
 Are you using the technology in a way
related to those goals and do you have
the right assessment tool?
Sivin-Kachala
Environmental Characteristics
 Key Factors:
 District-level involvement and
 A school-level computer coordinator
 Teachers are more effective after receiving
training in technology integration



Time
Incentive
Support System
Sivin-Kachala
Environmental Characteristics
 Teachers should carefully plan and
actively participate in learning activities
that incorporate tool software
 Teachers should offer students selfdirected learning experiences and
activities that encourage self-expression
 Students benefit from personal
interaction among staff members
Matching Measurement to Goals
Koedinger and Sueker
 Studied effect of an intelligent algebra tutor
on higher order skills development in the
context of “authentic, realistic” problem
solving tasks
 On a performance-based assessment
focusing on qualitative reasoning the
computer group scored significantly higher
 On the standard departmental final which
focused on standard algebra skills there was
no significant difference
Sivin-Kachala
Matching Measurement to Goals
Vanderbilt University
 The Adventures of Jasper Woodbury
 Focused on higher order thinking skills
related to solving math word problems
 Jasper students were superior to non-Jasper
students on a assessment of higher order
skills while the more traditional computation
skills improved at the same rate for both the
computer and non-computer groups.
Sivin-Kachala
Aren’t most of the successes
based on good teaching?
 Don’t separate the elements of a successful program




to give one component all the credit
University of Minnesota study shows how technology
can serve as a support to teacher-led instruction
Software was a catalyst to non-computer activities
In studies where a good teacher taught both the
computer and non-computer groups the positive tech
effects were less pronounced
Many software programs provide experiences a
teacher can’t replicate.
Sivin-Kachala
Sivin-Kachala
Positive software experiences
 Software developers put dollars, time and
research into good high-quality software
 You’re buying research and thinking and
experimentation
 Software can help shift a teacher’s focus

As students complete computer math drill teacher
can be doing something else
Sivin-Kachala
Best Uses
 Combine tools with instruction
 Instruction that combined word processing
and a process writing approach lead to
better results on a holistic writing
assignment
 Technology Usage is more effective is
teachers are trained
 The amount of time spent on the
computer makes a difference
Jay Sivin-Kachala, 1998
 219 research studies
 Students in technology rich environments
experienced positive effects on achievement in all
major subject areas
 Students in technology rich environments showed
increased achievement in preschool through
higher education for both regular and special
needs children
 Students’ attitudes toward learning and their own
self-concept improved consistently when
computers were used for instruction
Jay Sivin-Kachala, 1998
 Inconclusive Findings
 The
level of effectiveness of educational
technology is influenced by
the specific student population,
 the software design,
 the educator’s role, and
 the level of students’ access to the technology

Words of Advice
“It’s not just the technology that determines the
quality of a learning situation; it’s the whole
mix-what the class does before they use
technology, what the teacher does while
students use technology, how the students
are grouped, how prepared students are for
technology-based learning experiences, and
what the class, groups or individual students
do as a follow-up to using the technology.”
Jay Sivin-Kachala, from an interview with
Technology and Learning Online
National Center for Educational Statistics
Report on Teachers’ Technology Use
 Teachers are likely to integrate
computers and the Internet if
 Access
to adequate equipment
 Connections in their rooms
 Key factor:
 Teachers’
preparation and training to use
education technology
National Center for Educational Statistics
Report on Teachers’ Technology Use
 Approximately one-third of teachers
reported feeling well prepared or very
well prepared to use computers and the
Internet for classroom instruction
 Less experienced teachers felt better
prepared to use technology than their
more experienced colleagues
National Center for Educational Statistics
Report on Teachers’ Technology Use
 Barriers to Use
 Not
enough computers
 Lack of release time to learn
 Lack of time in schedule for students to
use computers in class
 Teachers who perceived barriers were less
likely to use technology
Eric Digest, 1998
 Applications of Technology to Basic Skills
 Applications of Technology to Advanced Skills
 Effects of Technology on Student Attitudes
 On-line Technologies
 Use of Technology by Teachers and
Administrators
 Factors That Help Technology Succeed
 Evaluating the Impact of Technology
Basic Skills
Eric Digest, 1998
 Using educational technology for drill and
practice of basic skills can be highly effective
according to a large body of data and a long
history (Kulik 1994)
 Students usually learn more and learn more
rapidly in courses that use computer assisted
instruction across all subject areas and in all
instruction from preschool to higher ed
 CAI in the military can be more cost effective
than additional tutoring, reduced class size,
or increased instruction time
Advanced Skills
Eric Digest, 1998
 “Constructivist” approach is made
easier by technology and may even be
driven by it
 Recent developments are difficult to
gauge in terms of educational effects
Student Attitudes
Eric Digest, 1998
 Bialo & Sivin-Kachala (1996)
 Students feel more successful in school are more
motivated to learn and have increased self
confidence and self esteem when using CAI
 Particularly true when the technology allows the
students to control their own learning


Across a variety of subject areas
Especially noteworthy when students are in at-risk
groups (special education, inner city, rural)
On-Line Technologies
Eric Digest, 1998
 Not much research on the Internet
 Some recent studies illustrate positive
effects
 CAST
(1996) shows higher scores on
measure of information management,
communication, and presentation of ideas
 Corporate-sponsored study
Teachers and Administrators
Eric Digest, 1998
 Use computer tools to streamline record
keeping and administrative tasks
 Free
up time for instruction or professional
development
 Decreasing isolation by using e-mail
and the Internet to communicate
 Increasing professional development
activities
Factors for Success
Eric Digest, 1998
 Glenna & Melmed (1996)
 Evidence
of a detailed technology plan
Consider funding
 Installation
 Integration of equipment
 Ongoing management of technology
 Clear vision of the goals

 Teacher
Training: operating and integrating
Factors for Success
Eric Digest, 1998
 Support from administration
 Funding
 Restructuring
schedules and physical
space to reflect the new learning
environment
 Support from the community
 Support from government
Evaluating Technology
Eric Digest, 1998
 Glenna & Melmed
 Available
tests do not reliably measure the
outcomes being sought
 New tests need to be developed
Evaluating Technology
Eric Digest, 1998
 Assessments of the impact of technology are
really assessments of instructional processes
enabled by technology
 Outcomes are dependent on the quality of
implementation of the entire instructional
process.



Instructional design
Content
Teaching strategies
 Dynamic nature of technology makes
meaningful evaluation difficult.
Wall Street Journal
Hard Lessons
 Computer labs are lousy places for
computers.
 Struggling students get more out of
computers than average or above-average
students.
 Most teachers still don’t know how to use
computers in class.
 School systems must plan computer use
carefully.
Wall Street Journal
Hard Lessons
 Computers are a tool, not a subject.
 Kids flourish when everyone has a computer




but schools aren’t spending enough to
guarantee that.
Schools can’t handle hand-me-downs.
Computers don’t diminish traditional skills.
The Internet and email excite kids by giving
them an audience.
Kids love computers.
Seven Factors for Success
 Technology initiatives should start with
instructional goals.
 Technology should be linked to
curricular goals and frameworks.
 Technology and the assessment system
must be compatible.
 Teachers and technology need to work
together.
Seven Factors for Success
 Teachers require ongoing pedagogical and
technological support.




District has committed at least 30% of budget to
teacher training
Continual training tailored to teacher needs
On-site staff development
Just in time training
 Community and parents involvement
enhance the likelihood of success.
Seven Factors for Success
 Business plays an important role in
technology and school reform.
 Equipment
grants
 Support for specific programs
 Helping with fundamental change
Center for the Study of Writing and
Literacy, 1992
Computers do not function as
independent variables in
classrooms, but rather as part
of a complex network of social
and pedagogical interactions.”
The Vision…
We know from the Apple Classroom of
Tomorrow studies and similar studies from
IBM, that the computers alone make little
difference. It is when the learning environment
changes that student learning changes. When
teachers use the tools to engage children in
active construction of knowledge; projects that
are related to standards for learning; and when
instruction, curriculum and assessment are tied
neatly together, then learning occurs.
Dave Mintz
National Alliance for Restructuring Education
Specific Studies
Kulik Meta Analysis
ACOT
Writing to Read
Accelerated Reader
Middleton and Murray
Kulik Meta Analysis
 Kulik begins with the problems:
 Computer based instruction means many
things
 Evaluation has been used in many ways
 Meta analysis was named by Gene
Glass in 1976
 Refers
to a statistical analysis of a large
collection of results from individual studies
for the purpose of integrating the findings
Kulik Meta Analysis
Overview
 Points to a dozen separate meta-
analyses that have tried to answer the
question of the effectiveness of CAI
 Yielded the conclusion that programs of
CAI have a positive record in the
evaluation literature
Kulik Meta Analysis
Major Points of Previous Studies
 Students usually learn more in classes
in which they receive computer-based
instruction
 Students learn their lessons in less time
with computer-based instructions
Kulik Meta Analysis
Major Points of Previous Studies
 Students like their classes more when
they receive computer help in them
 Students develop more positive
attitudes toward computers when they
receive help from them in school.
 Computers do not have positive effects
in every area in which they are studied.
Kulik Meta Analysis
Conclusions
 Computer tutoring produces positive results
 Students usually learn more faster
 Clearly greater than the gains produced by
instructional technologies that rely on print
materials
 Only Stanford-CCC has been around long
enough to be studied and results are positive
Kulik Meta Analysis
Other Computer Applications
 Findings are unimpressive for
managing, simulations, enrichment, and
programming
 Logo results are variable with individual
tests yielding high results and group
tests yielding indifferent results
ACOT
 Begun in 1985 as a research program on




impact of interactive technologies on teaching
and learning
Provided students and teachers an Apple
computer at home and school
Provision of technology access
Site freedom to develop technologysupported curriculum and pedagogy
Resulting study of what happens
ACOT
Evaluation
 Conducted a series of evaluations at
five original sites from 1987 to 1990
 Had to constantly find new ways to
evaluation outcomes based on up close
observation of sites
 “Formative evaluation” evolutionary
character
ACOT Triangulation
 Assess progress based on a range of measures and




multiple benchmarks
Compared students’ basic skills performance to
nationally reported norms
Comparison of student progress and achievement
over time
Comparison of ACOT classrooms with
demographically similar classrooms
Gathering data on classroom practices and parents’
background
The Importance of ACOT
 The findings about ACOT were less
important than the questions it raised
about evaluation and current
assessment methods
 Had a positive impact on student
attitudes
 Contributed to changing teaching
practices
The Inconclusions of ACOT
 On standardized tests, ACOT students
did not perform any better than
comparison groups or nationally
reported norms who did not have
access to computers or the teaching
and learning reforms implemented in
ACOT schools
David Dwyer and ACOT
 Memphis Schools found significant
gains in mathematics and language arts
basic skills
 30% is the magic number: takes 30%
less time to learn the same things with
help from the computer
 Most remarkable improvement was in
writing fluency
David Dwyer and ACOT
 Points to the importance of instruction
as an element in increasing writing skills
 Admits that the flaw with ACOT is that
participation was voluntary
 Dramatic results
 90%
went to college (15% for school)
 Dropout rate was 0% (30% for school)
 Better attendance
David Dwyer and ACOT
 Emphasizes need for staff development
 Teachers are trained as teams
 Develops
collegiality
 Encourages them to learn from and help
each other
David Dwyer and ACOT
The Down Side
 Lots of work to change the system
 Sometimes teachers perceived changes
that just weren’t there…
 Teaching kids to use the tools took too
much time away from content
Writing to Read
 Name to Know: Jean Casey
 Casey widens the definition of literacy to
include speech (interacting with
computers)
 Computer language must be learned
early just like any other language
 Questions the emphasis on learning to
write
Writing to Read
 Name to Know: Slavin
 Negative findings
 Had
a small but positive effect for
kindergarten students but did not positively
effect the reading achievement of first
graders
 WTR gains were not maintained
Writing to Read
 Casey responds:
 Writing
to Read 2000 has gone through
major changes
 Studies used existing reading tests that
tested isolated skills that don’t test self
esteem and confidence in writing
 Need to use quantitative and ethnographic
data to evaluate program
Writing to Read
 Two positive studies
 Mississippi
first graders made greater
gains in literacy skills than traditional
instruction regardless of socioeconomic,
racial or gender differences
 Alaskan students were empowered to write
and moved beyond the cultural bias of
standardized tests in evaluation
Writing to Read
 Simi Star Project
 Six school districts, 24 classrooms
 Evaluations shows they were writing and
reading two levels higher than peers in
classroom without technology
 Five years later they were still using the
computer as a tool to enhance learning
 Most success was found in schools where
teachers wanted technology integration
Accelerated Reader
 Vollands, Topping and Evans (1996)
found that after using AR for six month,
students showed statistically significant
increases about the control group of
students on measures of silent reading
comprehension, oral reading accuracy,
and comprehension
 Very
small group (27 AR/12 Control)
Accelerated Reader
 Peak and Dewalt (1994) used 50 ninth
graders
has used AR since 4th grade
 Half has never used AR
 Half
 In third grade, non-AR students had
higher reading scores
 In later testing years, AR students
scored higher on reading measures
Accelerated Reader
End of 8th Grade Results
AR Students
Average reading
score of 788
Yearly average
reading gains of
13 points
Non-AR Students
Average reading
score of 766
Yearly average
reading gains of
5.5 points
AR students reported reading more hours per
week and checking out more library books
Accelerated Reader
 Institute for Academic Excellence
 Studied use of AR in 2,511 Texas Schools
 Compared TAAS pass rates of AR schools
to median pass rates of non-AR schools
At all grade levels tested, the AR schools
tended to score above their median on the
reading and writing test
 Proportion of AR schools scoring above nonAR schools was 53 to 58 percent (statistically
significant)

Middleton and Murray, 1999
 Examined relationship between levels of technology
implementation in the classroom and standardized
test scores in reading and mathematics in grades
four and five
 Teachers were surveyed using the Levels of
Technology Implementation (LoTi) instrument to
determine their personal level of technology
implementation in their classroom
 Gathered standardized test achievement data from
students to see if any correlation could be made
between students of low level and high level
technology users
Middleton and Murray, 1999
 Students academic achievement was affected
by the level of technology used by the
classroom teacher
 Higher level of technology usage reported by
5th grade teachers and the findings indicated
that the academic achievement of 5th grade
students was statistically higher than those of
4th grade students
Middleton and Murray, 1999
 Supports finding of Johnson & Johnson
(1996) who determined that successful
implementation of technology depends
on the classroom teacher
 Teachers must see technology as an
important and legitimate tool for learning
Middleton and Murray, 1999
 Emphasizes need for teacher training
 Technology
integration increases with
training
 Teachers’ attitudes towards technology and
implementation are inseparable
 Says it is vital to fund inservice
opportunities for teachers