Evaluation and Benchmarking of
ICT in Schools
Professor Paul Bacsich
UK
17 October 2003
Acknowledgement
The Education and Manpower Bureau
would like to offer special thanks to
Professor Paul Bacsich for his powerpoint
presentation on
Evaluation and Benchmarking of ICT in Schools
Evaluation
In 4 easy lessons
Evaluation
•
•
•
Programme-wide with deep drill-down
Based on numbers and qualitative info,
with interpretation
Triangulation and stakeholders
Flashlight (after Ehrmann)


See www.tltgroup.org
And the Flashlight presentation at
www.roundtable.ac.uk/FLIntro.ppt
The Flashlight Oath


"Above all, I will do no harm. I know the value of
people's time, and so I will be ruthless in making
my surveys and interviews focused, brief, and
useful…so that those who have answered my
questions will be even more willing to
collaborate with the next researcher or evaluator
who contacts them."
"My study's process as well as its findings will
stimulate reflection and learning…"
Programme evaluation in UK

Based on Flashlight


and the OU evaluation consensus



Developed in the Annenberg/CPB trials of e-learning
in HE c. 1995
(Bates, Mason, Laurillard et al)
First substantial trial in the JANUS evaluations of
1993-5
Refined through TERG (SHU) into a tool for
evaluation of NLN

Next set of slides are c/o Ash, Carter and Mistry
NLN-e1 Findings




ICT is only appreciated by students when there is a
reason for it
ICT is an increasingly effective tool that is used
with increasing sophistication and appreciation
Increasing ICT skills at schools is pressurising
teacher skills at FE
Involvement in evaluation can facilitate colleges.
organisational development
Evaluation successes




A further £84M funding for the skills sector
DFeS are pushing the toolkit to evaluate
courses
Furthered thought in the identification of
good practice
Helping to raise awareness of the issues in
the pedagogy of ICT teaching
What is UK doing for the second
phase of NLN evaluation




Meta-analysis of current and recent
surveys
Quantitative data gathering from across
the sector
Twelve In-depth longitudinal studies
developed by external evaluators
Focus on the whole organisation rather
than the course
What did Phase one tell us about
conducting internal evaluations?



Tools and processes are often secondary
to good management
The process of reviewing and assessing
work can be threatening and evoke a
deeply political response
Evaluations serve multiple functions not
always those the evaluators envisage
Conducting the evaluation
Data collection





Leave your baggage at the door
Pilot everything
ALWAYS double check and share insights
with your informants
Look from multiple viewpoints
Don’t over-gather the same information
from the same sources
Benchmarking
Not an easy subject
Benchmarking – but who with?

Similar countries/regions – but who is?

Leaders, to provide aspirational role models

Laggards in case they catch up!
Criteria

Population




And population density
Type of language (alphabet, ideographic,
etc)
Level of use of ICT in general life
(Pedagogic) culture
Suggestions








Singapore
Scotland not England
Canadian (province level)
Denmark not Finland
Australia (states)
Thailand (laggard but trying to catch up)
South Korea?
US – are we sure?
ICT findings
Key ratios
Key Performance Indicators


Students per PC
Bandwidth per student PC


Media sophistication of software


Or per student (aspirational?)
Text, graphics, web, audio, video
Production to consumption ratio
Spot emerging trends even if small






Laptops/tablets are less intrusive and fit the
paradigm of “ubiquitous computing”
Binding of home PCs into school
systems: ”extranets”
Hosting: moving servers out of schools back to
server farms
New “worldware” apps?
Email integration with SMS on mobile phones??
Anything coming in from HE (Grid?) or industry
(cooperative working) or real life (games?)