Learning Technology & the
Evaluation of Learning Outcomes
Andrew Oliver
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT
Evaluation Techniques
Educational effectiveness: ability to impart knowledge
and understanding
Techniques:
 Quantitative (pre / post testing)
 Qualitative (questionnaire, interview)
The method used depends on the context
Ideally one method supplements the other
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT
Pre & Post Testing
When?
 directly before and after (control group)
 exam performance (compare with previous year )
But
 time between exercise & assessment
 knowledge gain through other methods, (books,
tutorial, revision)
 varying levels of validity
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT
Questionnaire - Usability
Common usage: usability of the technology
Example
 Did you find it difficult to move from Section A to B?
 Were the download times acceptable?
 Did you enjoy the exercise?
Relates to pedagogical design
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT
Questionnaire - Educational
Rarely educational: student self rating (Likert)
Example
 By how much did you feel the application increased
your knowledge
 How much did you feel the Worked Examples helped
your understanding of the topic
Key word “feel” - accuracy & honesty
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT
Questionnaire
Reality
 mixture of Educational and Usability
 impression of attitude towards technology
 idea of the learning environment
 hence context in which the application works best
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT
Problems, Problems
Problems associated with each method
Pre & Post Testing
 Preparation time (new questions)
 Timing
 Co-operation (staff & student)
 Not anecdotal
 Control group (selection & reliability)
 More time (interpretation)
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT
Problems, Problems
Questionnaire
 Data limitations (banding & differentiation)
 Self rating (accuracy and honesty)
Are students honest?….depends…
 Anecdotal (poor trends)
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT
Yet more…
Logistics
 Timing (student workload, availability) – can you
wait?
 Resources (workstations, rooms, network)
 Liaison (network admin, LRC) – cooperation
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT
Will it ever end?
Does it do what the label says it does?
 measure what its supposed to measure? (learning
outcomes)
 measured accurately?
 any outcomes not measured?
Time needed re design the test items
(Bloom’s taxonomy)
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT
Context of Evaluation
The intended context of use
Stand alone resource:
 Evaluate immediately after exercise
 Pre & post test
 Control group
Integrative approach:
 Questionnaire
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT
Context of Evaluation
Relate the context of the evaluation to the intended
educational setting of the technology
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT
So Which Method’s Best?
(or what can you realistically expect at the UH?)
Ideal
 pick n mix approach – one method augments the other
 but time required (design, analysis)
Realistic
 questionnaire (least time, user friendly, mix of quant/ qual)
 determine attitude towards IT and logistical problems –
curriculum planning
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT
Ways around it
(what you have to do to get what you want )
 pre / post test: disguise the exercise pre & post
testing and questionnaire
 bribery: marks awarded for attendance
 questionnaires: bribery or ambush or both
Finally
 no one method gives the complete answer
 use a variety and your experience / intuition
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT
LTDU web resources / references
 Questionnaire item database
 LTDU website: http://www.herts.ac.uk/ltdu
References:
 Bloom, B. S. (ed.), 1956. Taxonomy of Educational Objectives,
Handbook 1: Cognitive Domain, David McKay Co: New York
 Gronlund, N. E., 1971. Measurement and Evaluation in
Teaching, Macmillan Co: New York
 Laurillard, D., 1993. Rethinking University Education, Routledge:
London
_______________________
LEARNING TECHNOLOGY DEVELOPMENT UNIT