Chapter 10.3
Measuring the Impact of IT on Students' Learning
Rachel M. Pilkington
School of Education
The University of Birmingham
Birmingham, UK
R.M.Pilkington@bham.ac.uk
Abstract: Has the recent rapid expansion in the use of IT in schools had a positive impact on
learning? Research has presented us with mixed results that are often difficult to interpret.
Providing computers is certainly no guarantee of their effective use: how IT resources are
used in the local context to meet individual students’ needs seems critical to success. In
short, the alignment of particular types of IT to particular educational objectives and
assessment methods, together with planned, structured and guided activity, are likely to
determine whether IT impacts learning. However, this paper argues that the questions of
how, when and why IT impacts learning will require more holistic approaches to datagathering than traditional experimental or survey based approaches have provided. Further
research adopting a range of methods is needed if we are to discover precisely how
particular combinations of IT, instructional strategy and student activity lead to learning
outcomes.
Keywords: experimental design; survey research; case studies; meta analysis; impact on
learning
Introduction
Many governments have continued to fund rapid expansion in the use of IT in schools
and, not unreasonably, want to know if the positive impact of IT on learning is commensurate
with investment (Tolmie, 2001). Politicians want research evidence to address the question
‘was it worth it?’ (Pittard, 2004). However, to pose the question is much easier than to
answer it. This has to do with the potential and limitations of available research approaches
as well as with measuring ‘learning’.
The aim of this chapter is to provide an overview of progress in researching the
impact of Information Technology (IT) on students’ learning. When considering the last two
decades of research in this area a number of recurring issues emerge. These issues have
1
led many to call for a paradigm shift in our approach to educational research. However, the
nature of the shift called for is, itself, controversial as it relates to alternative perspectives on
how educational research should be conducted, how learning should be measured, and how
we should approach teaching and learning if we are to maximise potential. In addition, the
ways in which different authors view the role of IT within learning and teaching processes,
also affects how they evaluate the impact of IT on learning.
In presenting this review I discuss some of these alternative perspectives and, in
doing so, suggest what is known about the impact of IT on learning, gaps in our
understanding and future directions for research.
Impact of IT on Learning – Experimental Research Designs
There is debate as to the best research approach to take when measuring the impact
of IT on learning. Those advocating experimental methods often regard randomised
controlled trials (RCTs) as the ‘gold standard’. The aim in experimental methods is to
compare the performance of students assigned to an intervention group using IT with the
performance of students exposed to more traditional methods. In these studies ‘ learning’ is
often reduced to student performance on a test.
Ainsworth and Grimshaw (2004) point out that evaluations of computer based
Intelligent Tutoring Systems (ITSs) have achieved effect sizes of between 0.4 and 1
compared to classroom teaching (whilst one-to-one tutoring by expert tutors produces on
average an effect size of 2 according to Bloom, 2004). However, such effects are not
consistently gained. Ainsworth & Grimshaw (2004) found when evaluating the REDEEM
Intelligent Tutoring Authoring System the effect size was highly variable from 0.1 to 1.33
(mean 0.51). Effect sizes for other Computer Aided Instruction software (CAI) are often
reported to be even more variable or negative (Andrews, 2004; Eng, 2005). Moreover,
experimental control is easier to achieve for self-contained computer-based learning software
used by individuals than it is for more open social learning in classroom environments where
activity at the computer is just one activity amongst many. REDEEM worked best when
2
teachers used the flexibility of the design to add additional interactivity and when students
took advantage of this extra interactivity by answering questions or writing written notes
whilst learning (Ainsworth & Fleming, 2006).
This study illustrates a number of important issues affecting the value of comparative
experimental and quasi-experimental studies, one of the principal points being that the local
conditions of use are of central importance. Comparative research designs that use different
groups of students and/or tutors can be confounded by individual differences in the
characteristics of student and tutor. Local contextual variables associated with the
implementation of instructional strategies can also impact on success. As Ringstaff and
Kelley (2002) point out, classrooms are not experimental laboratories where scientists can
compare the effectiveness of technology to traditional instructional methods while holding all
other variables constant. Therefore, whilst the RCTs may still be regarded as the ‘gold
standard’ by many, the difficulty of isolating the role of the computer-based element in the
learning context can undermine the value of conclusions drawn (Tolmie, 2001; Pittard, 2004;
Cook, 2006).
Joy and Garcia (2000) argue that inability to control such variables can make it less
likely that researchers will find significant differences between computer-based treatment
groups and no-treatment groups. Similarly, Tolmie (2001) argues that it is unlikely, given the
complexity of the research context, that the addition of any new element into the classroom
environment could have a straightforward impact on learning.
In adopting the quasi-experimental comparative approach, there are often also ethical
issues concerned with the ways students may access resources at particular times. Indeed,
setting up these kinds of study in schools and colleges is notoriously difficult because
educational practitioners are concerned that the research should not interfere with day-to-day
classroom practice. In particular, research should not burden or disadvantage some students
more than others. These problems account in part, for the scarcity of well-controlled
comparative studies that measure the impact of IT on learning.
Joy and Garcia (2000) conclude that the outlook for comparative studies is bleak and
3
we should instead investigate particular combinations of instructional strategies, media and
activities that produce desired learning outcomes. Robust measurement of impact is
important but RCTs should perhaps be supplemented with richer ‘added value’ methods
(Pittard, 2004). Tolmie (2001) also suggests more context sensitive approaches are needed
which consider the interplay of technology with existing practice.
However, alternatives to the quasi-experimental approach are not without their own
difficulties. Rogers & Finlayson (2004) agree interpreting quantitative data from comparative
studies and large-scale surveys is often problematic yet, qualitative studies have also been
criticized for the small number of students they involve and the special conditions which
make drawing general conclusions difficult.
Researchers on the ImpaCT2 project (Harrison, Comber, Fisher, Haw, Lewin and
Lunzar et al., 2002) proposed a socially contextualised model of research that recognises
that IT experience is only part of a larger picture of pupils’ interaction with computer-based
technologies. Consequently they looked at the overlap between out-of-school learning and
school-based learning and attempted to assess the impact of some of these additional
influences through collection of qualitative data. Kennewell (2003), similarly argues that IT
should be studied alongside other variables in natural pedagogic settings using both
quantitative and qualitative research methods. Later in this chapter we explore further what
large-scale surveys and meta-analysis of case-based research can tell us about the impact
of IT on learning.
Measuring the Impact of Learning
An associated problem in drawing general conclusions concerning the impact of IT on
learning relates to how we measure learning. Thus, one of the first casualties of introducing
ITs into the curriculum is this original alignment of aims and objectives with delivery and
assessment strategies (Noss & Pachler, 1999; Ellaway, 2006). In short, the delivery method
has an affect on what is learned and how it can reasonably be assessed. This means that it
is difficult to prepare a common form of assessment that can fairly compare the traditional
4
course with the computer-based course. This problem was of particular concern to those
involved with the ImpaCT2 project (Harrison et al., 2002) discussed further later in this
chapter.
Further, several authors have argued that knowledge gained through IT may be
different in nature from that gained through other methods (Laurillard, 1978; Cheng, 1999).
This is not to say that one or the other is necessarily better but that they are different. Thus
Cheng (1999) notes that the representations used for learning in science and mathematics
can substantially determine what is learnt and how easily this occurs. Clements (2000)
argues that representations used with computational media offer unique opportunities for
problem and project-oriented pedagogical approaches that can catalyse pedagogic
innovation. Hammond (1994) concludes that this kind of innovation makes it difficult to
compare ‘with’ and ‘without’ IT conditions since introducing IT changes the nature of the
learning activity. As McCormick (2004) points out, research in assessment has not kept up,
for example, with the new learning opportunities offered by IT through collaborative
construction of multimedia or web-based products. Such products may employ different
purposes, skills and audiences from those of traditional handwritten essay.
When taking a quasi-experimental approach to research many studies have
addressed this problem by devising their own assessments that more validly reflect the skills
and knowledge to be compared. However, the point remains that the introduction of IT very
often changes the nature of the learning tasks and outcomes for good or ill and we need to
be sure we are sensitive both to evaluating what is actually learned (in both conditions) and
to whether what has been learnt is equally valuable relative to our educational aims.
Impact on Learning – Survey-Based Approaches
In this section the aim is to examine what is known about the impact of IT on learning
from survey-based approaches. A number of large-scale surveys have been commissioned
to evaluate the impact of funding on learning (Harrison et al., 2002; Conlon & Simpson, 2003;
Butt, Fielding, Foster, Gunter, Lance and Lock et al., 2003; Thomas, Butt, Fielding, Foster,
5
Gunter and Lance et al., 2003, 2004; Burns & Ungerleider, 2003; Hennessey & Deaney,
2004; Underwood, Ault, Banyard, Bird, Dillon and Hayes et al. 2005). Such surveys often
seek to discover the impact of IT by comparing a number of case schools. This enables
researchers to study authentic use of IT by teachers and learners without the need for
experimental manipulation and yet still make more general claims than can be provided by a
single local case study.
The ImpaCT2 project (Harrison et al., 2002) involved a large-scale survey of the use
of IT in UK primary and secondary schools to see what effect this investment was having.
Strand 1 of the study looked at baseline tests administered at the beginning and end of each
key stage (standard national attainment tests) alongside performance on GCSEs
(qualifications at 16 years) to try to determine evidence of the value added to the education
of children. Data related to use of IT at home and at school, were further analysed in relation
to gender, ethnicity and socio-economic factors. Overall the project found a small positive
relationship between GSCE performance and IT use with no cases where there was a
significant negative relationship i.e. no case where there was a statistically significant
advantage for lower IT use. However, there was no consistent advantage for higher IT use in
all subjects or at all key stages.
The quantitative data alone raised many questions. However, the authors concluded
that the most likely reasons for lack of consistency were lack of constructive alignment
between assessment and learning and effective teaching i.e. the factor most likely to impact
on learning remained the quality of the teaching (with or without IT). Because the results of
quantitative survey-based research are often confusing in relation to the impact of IT on
learning, there is a need to study a range of other variables that may be implicated through
survey design. Use of and access to IT in schools are perhaps the two related variables that
have been studied most.
The Transforming the School Workforce (TSW) Pathfinder project in the UK (Thomas
et al. 2004) was not designed to look at the impact of IT on learning per se but rather the
ways in which IT was being used in schools. The survey did record through questionnaires
6
and interviews the use of IT in school and at home. What this survey principally revealed was
that despite a push toward integrating IT in to the classroom, use of computers for learning
and teaching remained relatively modest. With notable exceptions, teachers were mainly
using IT to support basic literacy, numeracy and IT skills with many fewer examples of using
IT to support teaching in other subjects, for collaborative work, extended project work and
discussion. The main computer applications used were word-processing, presentation
software and the Internet. These applications were used mostly to support teachers in lesson
preparation rather than by children in the classroom.
The IT Testbed baseline project (Butt et al., 2003) found similar results. Both studies
suggested from a quantitative perspective a disappointing range of IT resources being used
in schools. From quantitative data it was difficult to tell why this was the case although staff
recognised a need for additional training in using IT for pedagogic purposes. However, in
both surveys there were outstanding examples such as the use of specialist multimedia
software (e.g. CAD and data-logging) to improve and extend the curriculum in art and design
and in science classes. There were also examples of use of the Interactive Whiteboard,
Desktop Publishing and PowerPoint software for extended project work and presentations of
children’s work in a range of subject classes.
Similar findings emerge from international studies: Conlon and Simpson (2003)
compared the introduction of IT in Scottish classrooms with introduction of IT in schools in
Silicon Valley and found similarities in access to resources at home and at school and in the
main uses of the technology for word-processing, email and searching the Internet. They
also showed (as in IT Testbeds Baseline study and TSW Pathfinder studies) that teachers
were not inherently resistant to the use of the technology. Around half of teachers regularly
used the computer for report writing and preparing lessons but use of computers by pupils in
schools was much more limited. The computer was seldom used in class unless the subject
studied was technology intensive. Students in secondary schools used computers in class
only once or twice a week and the majority of teachers use technology to reinforce existing
patterns of teaching rather than to innovate.
7
McMullan’s (2002) report in the UK looked at whether schools had access to the
necessary IT infrastructure to integrate IT in schools and concluded that whilst 99% of
schools had Internet access and there had been good progress on meeting targets of
computer: pupil ratios of 1:11 in primary and 1:7 in secondary schools, most schools did not
have broadband, links were slow and bandwidth often below that required to deliver a digital
curriculum. Underwood et al. (2005) surveyed the impact of the roll-out of broadband on UK
schools and found that whilst variations in connectivity persisted, barriers to the use of IT
were shifting away from basic ‘access’ problems toward; providing resources for technical
support, sustainable maintenance of equipment, training teachers and pedagogic strategies
to exploit the technology. McMullan (2002) noted that less than 10% of schools were covered
by a managed service contract. The issue of sustainability, particularly IT technical support
and maintenance of equipment, emerged as barriers affecting teachers’ computer use in the
IT Tested Baseline study (Butt et al. 2003; Pilkington in press) and TSW Pathfinder study
(Thomas et al. 2004; Pilkington in press). Similarly in these studies a high percentage of
teachers felt the need for more training in the instructional use of IT.
Burns & Ungerleider (2003) in discussing the Pan-Canadian Education Indicators
Program (PCEIP) reported that 88% of primary and 97% of secondary schools had Internet
access but 70% of the teachers still reported poor or limited access to computers due either
to low computer pupil ratio or other barriers to incorporating IT into their teaching such as the
need to book a computer laboratory down the hall (see also Watson, 2001). However, no
relationship between the presence of a computer in Ontario classrooms and achievement
was found for third grade students. Similarly Burns & Ungerleider (2003) report that in the
USA children using computers at least once a week did not perform better than children
using computers less than once a week on National Assessment of Education progress tests
for reading.
Although Burns & Ungerleider (2003) did not find a relationship between computer
presence and achievement, they found many examples of innovative programmes where
access to technology was combined with instruction designed to complement its use. For
8
these programmes, learning gains could be demonstrated in reading and spelling and in
science.
The results of large-scale surveys generally present a mixed picture that is hard to
interpret. The reasons for differences in outcome from such studies often relate not only to
differences in access to IT resources but the location or nature of the space for learning, the
ways in which resources are employed (and the teachers’ and pupils’ ownership or control
over them), differences in media, activity, interactivity and feedback, instructional
presentation, the wider cultural setting of the school/college, other less formal learning with
peers and the degree to which teaching staff, parents or other authority figures engage with
IT. These factors can all impact on the way IT is used by students.
A major problem in large-scale, survey-based research is that often detailed
contextual information from rich qualitative data is lacking. The above surveys drew on a
wide range of data including free text boxes in questionnaires and interview data from
teachers and other staff however, large-scale studies often involve little opportunity or
resource for follow-up questioning to decipher interconnections between data, generate more
holistic impressions or uncover the precise reasons for local successes or failures. Pelgrum
and Plomp (2008) in this Handbook discuss the potential and limitations of large scale survey
research in more detail.
Taken together findings from large-scale surveys would seem therefore to suggest,
perhaps unsurprisingly, that providing computers is no guarantee of their effective use but
that not providing enough computers with adequate speed and bandwidth is a barrier to use
(Conlon & Simpson, 2003; Thomas et al. 2004; Underwood et al. 2005; Burns & Ungerleider,
2003). Moreover, large-scale studies have told us that IT in schools can impact positively on
children’s attainment and motivation (Pittard, 2004) but it does not always do so; it can even
have a negative effect (Andrews, 2004). The alignment of particular types of IT to particular
educational objectives and assessment methods, together with tutor planned, structured and
guided activity, is likely to be what makes the difference. We still need much richer contextual
data to support these conclusions (Cox, Webb, Abbott, Blakeley, Beauchamp & Rhodes,
9
2003; Pilkington, in press).
Impact on Learning - Case Studies and Meta-analyses
One response to the appreciation that neither large scale surveys nor quasiexperimental lab-studies can easily provide us with straightforward answers concerning the
impact of IT on learning is to either abandon the attempt to conduct studies aimed at
evaluating the impact of IT or adopt alternative approaches such as rich contextual case
studies that aim to answer more modest questions concerning the impact of factors in
particular contexts.
There are many examples in the literature of this case-based approach. The hope of
such research is to gather sufficient volume of cases to enable the generation of some more
general principles or guidelines for pedagogic design and implementation: the how, when
and why of using computer-based learning (Cook, 2006). There are, therefore, a number of
studies that attempt to compare and contrast more than one ‘case study’ based on a range of
selection criteria. Case studies can therefore represent a single context or multiple contexts
and may use a range of methodologies including quantitative and qualitative data collection
involving outcome, process and attitudinal data on learning and related variables. I will refer
here to research studies that present data collected by the researchers as ‘case studies’
including those that compare more than one case. I will refer to research that reviews or
independently re-evaluates and compares the results of different studies as meta-analyses.
Many of these review case studies but some do not always make clear methodological
distinctions between the types of study reviewed.
Strand 2 of the UK ImpaCT2 project involved researchers looking at six
representative or ‘case’ schools and included more qualitative data such as log books /
diaries and peer-interviewing to gain insights into pupil and teacher perceptions. Concept
mapping was also used as a tool to access students’ conceptualisation of the role computers
played in their lives particularly the purposes of IT and locations of use. Strand 3 extended
the use of qualitative techniques to 15 schools using video diaries and electronic journals.
10
One of the results from this qualitative data was that the majority of pupils had very rich IT
experiences at home mainly using the Internet and playing computer games and that pupils
were frustrated by the IT curriculum at school. The skills focused on in class were often
basic, not challenging and only infrequently involved thinking or reasoning. Other studies
have reached similar conclusions in exploring the relationship between using computers and
motivating pupils in school (Passey, Rogers, Machell, McHugh & Allaway, 2003).
Overall, as reported in the previous section, the UK ImpaCT2 study found a positive
statistically significant association between IT and higher achievement in National Tests for
English at key stage 2 which stands in contrast to some reviews that have looked at the
impact of computers on literacy (Andrews, 2004; Eng, 2005). As discussed earlier surveys of
IT use for learning in UK schools have suggested that one of the most frequently reported
uses of software is for basic literacy. A range of software is available that is aimed at helping
pupils learn to spell. Torgerson & Elbourne (2002) meta-analysis based on pooling data from
six RCT studies of IT and spelling concluded no demonstrably better effect size for
computer-based teaching. Andrews (2004) looked at the impact of IT on literacy based on a
review of 188 international research studies and found small positive effects for spelling
software but overall negative results for CAI software on literacy. In contrast, software that
provides audio accompaniment to text can help some children. Andrews (2004) found
positive effects for speech synthesis. Thus, another reason for mixed results may be failure
to personalise learning to individual needs.
As students progress through the curriculum, basic literacy skills receive less
emphasis and greater emphasis is placed on compositional and critical thinking skills and the
ability to write in different styles for different audiences (Walker, 2003). In the ImpaCT2 study
(Harrison et al., 2002), when pupils used IT in English and achieved higher mean scores at
key stage 3, teachers identified factors leading to success that included high-quality
multimedia outcomes from using the word-processor which motivated commitment to writing
together with the use of e-mail to support collaborative writing. Collaborative writing can
increase reflection on writing products. Similarly, Walker’s (2003) case study using text-
11
based discussion to develop critical argument skills amongst children at this key stage
suggested that computer mediated communication can impact positively on debating skills
(Walker, 2003; Walker & Pilkington, 2005). Further evidence of the potential impact of IT on
higher order critical discussion and writing skills comes from case studies on online
discussion in Higher Education (Pilkington, Bennett & Vaughan, 2000). Positive results are
reported for some groups of students, notably those learning in a second language.
Passey (1999) analysed the learning objectives of the National Curriculum for IT in
England against Bloom’s taxonomy and concluded that there were too many lower-order
learning objectives with little use of IT to support higher order thinking. Moreover, Burns &
Ungerleider (2003) suggest that whilst many studies show computer based learning can
improve motivation, there are few controlled studies and the effect may be higher for boys
than for girls. Mumtaz found the most frequent activity at the school computer was wordprocessing which many pupils considered boring. Moreover, Burns and Ungerleider (2003)
note that when a motivational effect is found for computer use in education this effect may be
associated more with accompanying changes in instruction toward collaborative or social
learning rather than use of the technology per se. Mumtaz (2001) concludes that teachers
need to ask whether the tasks they are setting using word-processors are challenging or
interesting for children and whether they involve higher-level skills.
Clements (2000) based on a review of studies in mathematics education suggests
one of the unique contributions computers can make to learning is through the support of
problem-based learning and extended project work. Collaborative activities also resulted in
enhanced achievement. Voogt & Pelgrum’s (2005) evaluation of case studies in 28 countries
found that those innovating with IT in the curriculum did evidence elements of an emerging
pedagogy of learning with IT that emphasised collaborative and meta-cognitive skills that are
considered important for deeper learning.
Waxman & Huang (1996) in a case study of middle school mathematics found
significant differences in instruction in the classroom depending on the amount of technology
used. Whole-class approaches where pupils generally listened to or watched the teacher
12
tended to be used in classrooms where technology wasn’t often used. When technology was
used moderately there was much less whole-class instruction and more independent work
suggesting using technology may help shift teachers’ activity toward a more pupil centred
approach at least in some instances.
IT is also said to have some unique properties with respect to being able to provide
richer multimedia resources that engage additional perceptual channels to encode and
retrieve information. Najjar (1996) in reviewing a range of studies including work by Mayer &
Anderson (1991) concludes that for understanding particular kinds of processes the dynamic
qualities of video and animation with explanatory narration can improve learning. Cox et al.
(2003) in their review of the literature and a number of case studies conclude that simulation
and modelling software may have similar advantages and, in addition to helping students
envision abstract, complex and/or dynamic relationships, may also help them develop critical
thinking skills through hypothesis testing. However, they suggest the effective exploitation of
the potential of IT depends on the way in which the teacher selects and organises IT
resources and how these are integrated with classroom activities.
Another use of multimedia to help students envision relationships in science is data
logging. Rogers & Finlayson (2004), based on a study of teacher-evaluations of lessons
report positive effects from the use of real time data logging of temperature to offer
simultaneous presentation of graphs. They suggest this adds value to learning in the science
lesson by making results instantly visible. Other examples of teacher-rated effective lessons
were recorded in UK IT Testbeds baseline and TSW Pathfinder studies discussed earlier
including ‘through the bell’ projects conducted over several lessons in different subjects using
Desktop Publishing or video editing to create presentations on cross-curricula themes.
The over-riding difficulty with case studies, when read in isolation as opposed to
within the context of more comprehensive reviews or surveys is the ‘starry nights’ effect
(Ellaway, 2006). Case-based research often seeks out ‘good’ examples of practice focusing
on these interesting ‘stars’ in detail and ignoring the darkness of the night around them.
There are examples of researchers addressing this selectivity problem and comparing and
13
contrasting different cases e.g. case schools at different stages of development in their
progression toward integrating IT in the curriculum (Pilkington, in press). However, there is
potential for distortion in seeking and reporting examples, demonstrating the positive impact
of ICTs on learning particularly in the absence of similar reporting of negative cases.
Wang, Haertel & Walberg (1993) researched factors impacting on education more
generally (regardless of the use of IT) through a meta-analysis based on comparing the
results of 9 previous literature reviews and 179 research studies. The aim was to try to find
out which of the many factors affecting attainment seemed to be most significant. They found
that the following ‘proximal variables’ have most effect: meta-cognitive and cognitive
activities; classroom instruction and management; pupil-teacher social and academic
interactions; the home environment; students’ prior knowledge and level of understanding;
instructional strategies such as reciprocal teaching (see Rosenshine & Meister, 1994 for a
review of this method). One of the principle conclusions was that students benefit from
academic interactions with tutors and positive social interactions with students and that the
actions of students, teachers and parents matter more than policies at the program, schooldistrict, state or national level. Wang, Haertel & Walberg conclude that the limited effect of
the latter more ‘distal variables’ when compared with the day-to-day efforts of the people
most involved in students lives, should help educators and policy makers be mindful of where
they can make the biggest difference. This resonates with Cox et al.’s (2003) conclusion that
the nature of the local classroom interaction and activity with computers are critical in
determining the impact on learning.
Ringstaff and Kelley (2002) based on a U.S. Department of Education funded review
of findings from several studies sum up for many when they say that there is now a
substantial body of research suggesting that technology can have a positive effect on student
achievement under the right conditions. However, there is as yet no ‘magic formula’ that
educators and policy makers can use to determine if the return is worth the investment. Eng
(2005), also based on a meta-analytic review of several studies mainly from the UK, USA
and Australia, noted that the relationship between IT use and attainment was overall positive
14
though weak. Eng concludes that results were likely to be better when IT was used as a
supplement for individual learning and teacher-led programmes were more effective than
commercial software because they linked more closely to educational objectives. Some
consensus thus seems to be building around the notion that applications need to be tailored
to contexts of use and the needs of individual learners for full potential to be realised.
Moreover, based on recent survey approaches presented in the previous section the
barriers to effective use of IT are shifting away from basic provision of resource toward more
complex access issues (such as the location and control of resource within organisations and
the reliability and speed of machines and their connections). There is an identified need to
invest in professional development for staff, to provide teachers with technical support and to
resource the maintenance and upgrading of equipment (Butt et al., 2003; Thomas et al.,
2004; Burns & Ungerleider, 2003; Underwood et al. 2005; Pilkington, in press).
Part of the answer to the dilemma in finding a suitable approach to research therefore
almost certainly lies with recognising the complex nature of the ways different variables
interact in authentic situations and in seeking more holistic approaches to investigation and
data-gathering techniques.
Future Schools: Making Progress and Managing Change
In this section the vision of the IT supported ‘future school’ is revisited looking at
emerging barriers and enablers to effective use of IT in schools and colleges.
Results, from surveys and meta-analyses reviewed so far suggest that from a
quantitative perspective there is a disappointing range of IT resources being used in schools.
Moreover there are some emerging common barriers to IT use. The Fischer Family Trust
(2002) based on an expert consensus building exercise sum up many of these barriers: lack
of access (including remote computer rooms); lack of time to prepare; cost of software; lack
of technical support; resources for maintenance and upgrades; low teacher confidence.
Similarly, Mumtaz (2000) suggests factors affecting teachers’ decisions to use technology
include: access to resources; quality of software/hardware; ease of use; incentive to change;
15
support from the school; commitment to professional development and familiarity with IT.
Many of these issues are difficult for individual teachers or schools to address in
isolation, particularly infrastructure issues. The state of repair of computers can be a ‘hidden’
access problem limiting the usable machines in a class. Another ‘hidden’ access problem
that emerged was the need to book a classroom down the hall. The disruption of moving is a
major disincentive to integrating IT in subject teaching (Watson, 2001; Butt et al. 2003; Burns
& Ungerleider, 2003; Underwood et al., 2005).
Mumtaz (2000) in a review of the literature suggests personal factors outweigh
institutional factors in affecting decisions to use technology, specifically that teachers’
theories about IT use are central and that even with up-to-date resources they may not be
enthusiastic if technology is imposed from outside. Watson (2001) also suggests teachers
are not impressed by the imposition of change that appears to focus on what the technology
can do rather than on the learning. Rogers & Finlayson (2004) argue that teacher rejection of
IT is more likely to be the inevitable result of lack of teacher time to learn how to use the
technology. Conlon and Simpson (2003) also suggest teachers are not inherently resistant to
IT. Several surveys suggest teachers’ belief in the potential of IT to support learning is
positive (Butt et al., 2003; Thomas et al., 2004). However, such surveys have repeatedly
highlighted that teachers feel the need for more professional development, particularly in
pedagogic applications of technology (Cox et al., 2003; Kennewell, 2003).
Flecknoe (2002), in discussing a UK Teacher Training Agency funded programme for
professional development of teachers, notes the difficulty in demonstrating that such training
has a direct impact on pupils’ learning. As explained in Chapter 10.2, early research
evidence from Canada, England, the Netherlands and Spain (Watson & Tinsley, 1995)
suggested that teachers using IT tended to be those who could relate the use of technology
to their own subject. However, those electing for courses may also be early adopters not
reflecting the majority of teachers.
Ideally, students’ activity at the computer holds students’ attention, releasing the
teacher for individual facilitation. However, this requires considerable skill. As Barrows (1992)
16
suggests, there is nothing automatic about becoming a good facilitator. Coutts, Drinkwater &
Simpson (2001) describe the teacher hovering in the background uncertain of what to do as
pupils engage with computer software. Moreover, some technologies e.g. PowerPoint
projected on the Whiteboard, can support teachers in traditional methods (Pilkington, in
press).
Hennessey & Deaney (2004), based on interviews with teachers engaging in IT
projects in 5 case study secondary schools, illustrate how once barriers of access are
overcome, teachers do continue to develop and evolve their practice so that their use of the
technology becomes increasingly integrated with their subject teaching and more innovative.
The Fischer Family Trust (2002) in summing up the common enablers that affect teachers’
use of IT include: creativity; ownership of resources; sharing good practice; strategic
leadership and subject specific knowledge of IT.
Revisiting Learning Theory: Issues for Design
In this section we revisit learning theory looking at what we have found out
concerning the contribution of IT to instructional design and how we should approach
teaching and learning with IT to maximise potential.
As we develop and become adults we need increasingly to be able to take
responsibility for our own learning, to be involved in planning, negotiating and personalising
our learning and ensuring its relevance to our aptitudes, vocation and interests. Approaches
that are thought to scaffold this kind of autonomy emphasise sharing experiences through
collaborative inquiry and authentic problems or tasks. Most importantly, such approaches
give plenty of opportunity for discussion and reflection on experience in social and
constructive contexts (Knowles, 1970; Scardamalia & Bereiter, 1991; Savery & Duffy, 1996;
OECD, 2004).
Many believe a positive aspect of IT is that it encourages a shift in pedagogy toward
more facilitative teaching approaches better suited to social and constructive models of
learning. Use of IT has been associated with a decrease in direction by and exposition from
17
the teacher as students work individually or in pairs and groups around computers. There are
reports of corresponding increases in self-regulation and constructive dialogue (Crook, 1997;
Wegerif & Dawes, 1998; Voogt & Pelgrum, 2005). However, in moving to a facilitator role
teachers still need to lead through the planning, preparation and follow-up of lessons. Where
little planning occurs, class work can be unfocused and outcomes poor. When teachers use
their knowledge of both the subject and the way pupils understand the subject their use of IT
has a more direct affect on attainment (Cox et al., 2003).
However as Rogers & Finlayson (2004) report in relation to science teachers’ use of
IT, perceptions of success are largely expressed in terms of achievement of subject learning
objectives with criteria strongly rooted in existing pedagogy and assessment methods
developed using conventional resources. IT can, therefore, both challenge and change
practice but it is not automatic. Pedagogical practices using IT range from only small
enhancements of existing practice underpinned by traditional methods to more fundamental
changes in approach (Cox et al., 2003; Pilkington, in press).
Similarly, it has emerged from the surveys reviewed here that as teachers begin to
use IT they may do so at first in ways which reinforce traditional practice. Later they may use
IT to make modest enhancements, e.g. exploiting properties of multimedia to improve
resources in ways that impact on the understanding of concepts. Later, as they continue to
integrate their subject knowledge with the use of IT, they may include more subject specific
software to improve and extend the curriculum. As they continue to evolve their practice,
perhaps adopting more social and collaborative ways of working, they may also use IT for
extended collaborative projects (Pelgrum & Anderson, 1999; Cox et al. 2003; Voogt &
Pelgrum, 2005; Passey, 2006; Pilkington, in press).
Voogt & Pelgrum (2005) argue that improving the quality of teaching and learning with
IT should now focus on setting new goals including the design of resources for
comprehension through effective use of multiple modalities and improving students’ critical
engagement, independent and collaborative learning skills (particularly including a focus on
learning how to learn). Perhaps most controversially, curriculum content should be offered in
18
a school-wide, cross-curricula way and embedded in authentic contexts.
Conclusions
Reynolds, Treharne & Tripp (2003) describe claims for the effectiveness of IT as
“optimistic rhetoric” that has led successive British governments to spend billions of pounds
without first establishing through research whether IT improves learning.
The inherent difficulties in conducting comparative studies in educational settings
have made it difficult to obtain robust and conclusive evidence regarding the impact of IT on
learning. This has led many to suggest that whilst robust measurement of impact is important
RCTs should perhaps be supplemented with richer ‘added value’ methods (Pittard, 2004).
Tolmie (2001) also suggests more context sensitive approaches are needed if we are to
consider the interplay of technology with existing practice. Moreover there is a need to
readdress the constructive alignment of assessment methods in evaluating learning from IT.
Based on data at that time Pittard (2004) concluded that despite progress in putting
resources into British classrooms including roll-out of broadband technology, the delivery of
education happens in many of the same ways it did before. Conlon & Simpson (2003) and
Voogt & Pelgrum (2005) looking at the international scene seemed to conclude similarly,
though more positively. There is evidence that activities for students using IT in classrooms
are often not challenging and tend to involve low-level or basic skills development with little
of the reasoning or critical thinking needed to develop deep learning. These activities can
frustrate pupils (Passey, 1999; Mumtaz, 2001; Passey et al., 2003; Passey, 2006). This has
led many to argue that we need to refocus educational goals when using IT to include more
collaborative, cross-curricula, problem-based or project work (Clements, 2000; Voogt &
Pelgrum, 2005). However, perhaps we should not be too quick to say the investment has not
worked. More recently Webb and Vulliamy (2006) revisited 50 primary schools previously
studied in the early ’90s, despite criticisms of initiatives, teachers were reported to believe
teaching methods had been enhanced by IT and that new technologies had enabled
innovation. We have also seen evidence that so far only the initial barriers to teachers using
19
IT in classrooms have been overcome. Whilst basic access problems are largely overcome
for schools in developed countries, this has simply shifted the focus from the barriers of
access to barriers of technical support and maintenance, teacher time and professional
development. There is evidence from more recent studies that practice is continuing to
evolve as teachers extend their familiarity with technologies. For example based on more
recent observation of lessons in 2005, Passey (2006) concludes that although there was still
limited use of IT to directly support critical and collaborative discourse or hypothesis testing,
IT resources were being used directly to support concept formation through video resources.
We know from educational research more widely that however important educational
policy is, the factors that impact most on learning are the local ones, such as classroom
instructional strategies and management, student-teacher social and academic interactions,
the home environment and students’ prior knowledge (Wang, Haertel & Walberg, 1993).
Watson (2001) suggests that technology today holds a major symbolic function in society
associated with the imagery of the new, positive change, renewal and economic revival but
argues for the need to reframe intervention away from the technological model. What is
needed is an intervention of educational philosophy and debate. Teachers may then be
inspired to integrate IT into their existing practice in ways that make a less conservative
impact on subject learning. There are plenty of examples of early adopters leading the way.
However, a lot more rigorous research is needed if we are to discover precisely how
particular combinations of IT, instructional strategies and activities produce desired learning
outcomes.
References
Ainsworth, S., & Grimshaw, S. (2004). Evaluating the REDEEM authoring tool: Can teachers
create effective learning environments? International Journal of Artificial Intelligence
in Education, 14(3-4), 279-312.
Ainsworth, S., & Fleming, P. (2006). Evaluating authoring tools for teachers as instructional
designers. Computers in Human Behaviour, 22, 131-148.
20
Andrews, R. (2004). The impact of ICT on literacy education. London: Routledge Falmer.
Barrows, H. S. (1992). The tutorial process. Springfield, IL: Southern Illinois University
School of Medicine.
Bloom (1984). The 2 sigma problem: The search for methods of group instruction as effective
as one-to-one tutoring. Educational Researcher, 13(6), 4-16.
Burns, T. C., & Ungerleider, C. S. (2003). Information and communication technologies in
elementary and secondary education. International Journal of Educational Policy
Research and Practice, 3(4), 27-54.
Butt, G., Fielding, T., Foster, P., Gunter, H., Lance, A., Lock, R., Potts, L., Pilkington, R.,
Powers, S., Rayner, S., Rutherford, D., Selwood, I. D., & Soares, A. (2003). Baseline
evaluation of the ICT test bed project, Report for DfES.
Cheng, P. C.-H. (1999). Unlocking conceptual learning in mathematics and science with
effective representational systems. Computers and Education, 33(2-3), pp. 109-130.
Cox, M., Webb, M., Abbott, C., Blakeley, B., Beauchamp, T., & Rhodes, V. (2003). ICT and
pedagogy: A review of the literature (No. DfES/0793/2003): Produced by Becta for the
Department of Education for Skills. National Grid for Learning Research and
Evaluation Series Report No. 18.
Clements, D. H. (2000). From exercises and tasks to problems and projects – unique
contributions of computers to innovative mathematics education. Journal of
Mathematical Behaviour, 19(1), 9-47
Conlon, T., & Simpson, M. (2003). Silicon Valley versus Silicon Glen: The impact of
computers upon teaching and learning a comparative study. British Journal of
Educational Technology, 34(2), 137-150.
Cook, D. (2006) Where are we with Web-based learning in medical education? Paper
presented at the SMILE conference. Sestri Levante, Italy, 13th-15th September.
Coutts, N., Drinkwater, R., & Simpson, M. (2001). Using information and communications
technology in learning and teaching: A framework for reflection, planning and
evaluation in school development. Teacher Development, 5, 225-239.
21
Crook, C. (1997). Children as computer users: The case of collaborative learning. Computers
& Education, 30, 237-247.
DfEE. (1997). Connecting the learning society, national grid for learning. Government
consultation paper. Available at URL:
http://www.dfes.gov.uk/consultations/conResults.cfm?consultationId=1104 [last
accessed 10/06/06]
Harrison, C., Comber, C., Fisher, T., Haw, K., Lewin, C., Lunzar, E., McFarlane, A., Mavers,
D., Scrimshaw, P., Somekh, B., & Watling, R. (2002). Impact2: The impact of
Information and Communications Technologies on pupil learning and attainment.
Produced by Becta for the Department for Education and Skills. National Grid for
Learning Research and Evaluation Series report No. 7.
Ellaway, R. (2006). Constructive alignment and integrating e-learning into the curriculum.
Paper presented at the SMILE conference. Sestri Levante, Italy, 13th-15th September.
Eng, T. S. (2005). The impact of ICT on learning: a review of research. International
Education Journal, 6(5), 635-650.
Fischer Family Trust. (2002). Identifying the impact of ICT in secondary subjects. Warwick:
Becta and Fischer Family Trust.
Flecknoe, M. (2002). Measuring the impact of teacher professional development: Can it be
done? European Journal of Teacher Education, 25(2-3), 120-134.
Hammond, M. (1994). Measuring the impact of IT on learning. Journal of Computer Assisted
Learning, 10, 251-260.
Hennessey, S., & Deaney, R. (2004). Sustainability and evolution of ICT-supported
classroom. Final Report for Becta. Coventry: Becta.
Joy, E. H., & Garcia, F. E. (2000). Measuring learning effectiveness: A new look at nosignificant-difference findings. JALN, 4(1), 33-39.
Kennewell, S. (2003). Developing research models for ICT-based pedagogy. Proceedings of
the 3.1 and 3.3 working groups conference on ICT and the teacher of the future.
Melbourne, Australia: International Federation for Information Processing.
22
Knowles, M. (1970) Andragogy: An emerging technology for adult learning. In M. Tight (Ed.),
Education for adults: Adult learning in education. London: Croom Helm.
Laurillard, D. M. (1978). Evaluation of student learning in CAL. Computers and Education, 2,
259-263.
Mayer, R. E., & Anderson, R. B. (1991). Animations need narrations: An experimental test of
a dual-coding hypothesis. Journal of Educational Psychology, 83, 444-452.
Najjar, L. J. (1996). Multimedia information and learning. Journal of Educational Multimedia
and Hypermedia, 5, 129-151.
McCormick, R. (2004). ICT and pupil assessment. The Curriculum Journal, 15(2), 115-137.
McMullan, T. (2002). Wired to learn: What's holding up the school of the future? ASI
Research Ltd., UK: Adam Smith Research Institute.
Mumtaz, S. (2000). Factors affecting teachers' use of information and communications
technology: A review of the literature. Journal of Information Technology for Teacher
Education, 9(3), 319-341.
Mumtaz, S. (2001). Children’s enjoyment and perception of computer use in the home and
the school. Computers and Education, 36, 347-362.
OECD. (2004). Policy brief: Lifelong learning. Organisation for Economic Cooperation and
Development. Available at URL: http://www.oecd.org/dataoecd/17/11/29478789.pdf
[last accessed 10/06/06]
Noss, R., & Pachler, N. (1999). The challenge of new technologies: Doing old things in new
ways or doing old things? In P. Mortimore (Ed.), Understanding pedagogy and its
impact on learning. London: Paul Chapman.
Passey, D. (1999). Strategic evaluation of the impact on learning of educational technology.
Education and Information Technologies, 4(3) 223-250.
Passey, D., Rogers, C., Machell, J., McHugh, G., & Allaway, D. (2003). The motivational
effect of ICT on pupils. Annesley: DfES.
Passey, D. (2006). Technology enhancing learning: Analysing uses of information and
communication technologies by primary and secondary school pupils with learning
23
frameworks. Curriculum Journal, 17(2), 139-166.
Pelgrum, W. J., & Anderson, R. A. (Eds.). (1999). ICT and the emerging paradigm for lifelong
learning: A worldwide educational assessment of infrastructure, goals and practices.
Amsterdam: International Association for the Evaluation of Educational Achievement.
Pilkington, R. M., Bennett, C. L., & Vaughan, S. (2000). An evaluation of computer mediated
communication to support group discussion in continuing education. Educational
Technology and Society, Special Issue on on-line Collaborative Learning
Environments, 3(3), 349-359.
Pilkington, R. M. (in press October 2006). Learning and ICT: The future schools and
embedding ICT in the curriculum. In G. Butt, H. Gunter, & H. Thomas (Eds.),
Modernising schools: People, learning and organisations. (pp. Chapter 9). London:
Continuum.
Pittard, V. (2004). Evidence for e-learning policy. Technology, Pedagogy and Education,
13(2), 181-194.
Preece, J. (2000). Online communities: Designing usability supporting sociability. Chichester:
John Wiley & Sons.
Reynolds, D., Treharne, D., & Tripp, H. (2003). ICT the hopes and the reality. British Journal
of Educational Technology, 34, 151-167.
Ringstaff, C., & Kelley, L. (2002). The learning return on our educational technology
investment: A review of findings from research. San Francisco: WestEd Regional
Technology in Education Consortium of the Southwest.
Rogers, C., & Finlayson, H. (2004). Developing successful pedagogy with Information and
Communications Technology: How are science teachers meeting the challenge.
Technology, Pedagogy and Education, 13(3), 287-305.
Rosenshine, B., & Meister, C. (1994). Reciprocal teaching: A review of the research.
Educational Research, 64, 479-530.
Savery, J. R., & Duffy, T. M. (1996). Problem based learning: An instructional model and its
constructivist framework. In B. Wilson (Ed.), Constructivist learning environments:
24
Case studies in instructional design. Englewood Cliffs, NJ: Educational Technology
Publications.
Scardamalia, M., & Bereiter, C. (1991). Higher levels of agency for children in knowledge
building: A challenge for the design of new knowledge media. The Journal of the
Learning Sciences, 1, 37-68.
Thomas, H., Butt, G., Fielding, A., Foster, J., Gunter, H., Lance, A., Pilkington, R. M., Potts,
L., Powers, S., Rayner, S., Rutherford, D., Selwood, I. D., & Szwed, C. (2004). The
evaluation of the transforming the school workforce pathfinder project. DfES research
report No. 541.
Tinto, V. (2000). Learning better together: The impact of learning communities on student
success in Higher Education. Journal of Institutional Research in Australasia, 9(1).
Tolmie, A. (2001). Examining learning in relation to the contexts of use of ICT. Journal of
Computer Assisted Learning, 17, 235-241.
Torgerson, C. J., & Elbourne, D. (2002). A systematic review and meta-analysis of the
effectiveness of information and communication technology on the teaching of
spelling. Journal of Research in Reading, 25(2), 129-143.
Underwood, J., Ault, A., Banyard, P., Bird, K., Dillon, G., Hayes, M., et al. (2005). The impact
of broadband in schools. Coventry: Becta.
Voogt, J., & Pelgrum, H. (2005). ICT and curriculum change. Human Technology, 1(2), 157175.
Walker, S. A. (2003). The contribution of computer-mediated communication in developing
argument skills and writing-related self-esteem. Unpublished PhD, The University of
Leeds, Leeds, LS2 9JT, U.K.
Walker, A., & Pilkington, R. M. (2005). Using computers to assist in developing key literacy
skills. In M. Monteith (Ed.), Teaching secondary school literacies (pp. 71-96).
Maidenhead, UK & New York: Open University Press, McGraw Hill Education.
Wang, M. C., Haertel, D., & Walberg, H. J. (1993). Toward a knowledge base for school
learning. Review of Educational Research, 63, 249-294.
25
Watson, D., & Tinsley, D. (Eds.). (1995). Integrating information technology into education.
London: Chapman and Hall.
Watson, D. M. (2001). Pedagogy before technology: Re-thinking the relationship between
ICT and teaching. Education and Information Technologies, 6(4), 251-266.
Waxman, H. C., & Huang, S.-Y. L. (1996). Classroom instruction differences by level of
technology use in middle school mathematics. Journal of Educational Computing
Research, 14(2), 157-169.
Webb, R., & Vulliamy, G. (2006). Coming full circle: The impact of New Labour’s education
policies on primary school teachers’ work. Project Report. Available at URL:
http://hdl.handle.net/2428/5572. Last accessed 20/08/07
Wegerif, R., & Dawes, L. (1998) Encouraging exploratory talk around computers. In M.
Monteith, (Ed.), IT for learning enhancement. Exeter: Intellect books.
26