Lim Kin Chew
SIM University
23 Feb 2012
8 th International Conference on E-Learning for a Knowledge-Based
Society
Bangkok, Thailand

“Although recent attention has increased e-learning evaluation, the current research base for evaluating e-learning is inadequate … Due to the initial cost of implementing e-learning programs, it is important to conduct evaluation studies.”
(American Society for Training and
Development, 2001).

“Despite the ample support and enthusiasm for virtual models of learning from both the corporate and educational arenas, the effectiveness of e-Learning is by no means a sure thing. Successful e-Learning programmes require specific conditions and constant monitoring and a lot more research has yet to be done before an acceptable universal model can be developed.”
( Kar-tin Lee, Hong Kong Institute of Education,
Malaysian Online Journal of Instructional Technology,
August 2005, ISSN: 1823-1144, Vol. 2, No.2, pp 61-71 )

• Share experiences on how an actual evaluation of a large-scale e-learning project was carried out
• Add to the research base on e-learning effectiveness

• Why do we need a practical corporate elearning effectiveness model?
• Different approaches to evaluation models
• Case study on an e-learning implementation
• QPICC model
• Lessons learnt
• Conclusion

(Source: Graham Attwell (ed.). (2006). Evaluating E-learning - A Guide to the Evaluation of Elearning. Evaluate Europe Handbook Series Volume 2. ISSN 1610-0875.)

• E-learning effectiveness is difficult to carry out.
• Even if it is carried out, it is usually done in a rather ad hoc manner.
• E-learning evaluation takes up time, resources, manpower, cost and it may end up inconclusive
• E-learning evaluation should be done by experienced e-learning practitioners
• E-learning evaluation depends very much on what the user client wants to evaluate or measure. The consultant guides him along.

• To evaluate the effectiveness of the e-learning efforts in facilitating the training functions of
ABC
• Based on the evaluation outcomes, the client wants appropriate recommendations on elearning capabilities so as to further enhance instructional delivery in the ABC training.

• ABC (name changed because of confidentiality) is a government body in charge of training personnel for home security.
• They have been using e-learning since 2006.
• Most of the training has been supplementary, i.e. used to support the classroom face-to-face sessions.
• Investments have been made to set up three LMSs
• Management now wanted to find out how effective has the e-learning been in training the personnel
• Client has several departments. Only three departments were involved in the evaluation project.

To evaluate the effectiveness of ABC e-learning training platform in the following areas: i.
qualify for the pedagogical underpinnings considered for elearning modules; ii. measure the quality of e-learning course wares; iii. measure the performance output by benchmarking the elearning platform against a set of pre-defined criteria; iv. determine the adequacy of existing e-learning
infrastructure and to validate congruency in these capabilities with their instructional intent; v. identify trainer and training designer’s competency gaps in e-learning facilitation and design based on the evaluation outcome.

Proposed E-learning Quality & Effectiveness Model for ABC

Team
Leader
Team UniSIM Team ABC
• Participative approach
• Expertise transfer
• Joint review of results, observations and conclusions
• PLAN – DO – CHECK – ACT ( PDCA )

S/No. Module
1 Quality Criteria of E-
Learning Courses
2 Performance
Measurements of E-
Learning Courses
Specific Evaluation Tools
Quality Criteria Checklist

Satisfaction survey (for participating e-learners)

Pre-Test Results (if available)

Post-Test Results (if available)

Questionnaire survey for supervisors / facilitators
Frequency of
Use
Once for every
E-Learning course
To do all for one time for one elearner in the sample
Involvement of
ABCs e-Learning
Champions
Some involvement
Considerable involvement as we will be checking their performance results
3 E-Learning
Infrastructure

Inventory report of existing E-
Learning infrastructure

Inventory report of proposed
E-Learning infrastructure
To do a onetime compilation report
4 Staff Competencies Questionnaire survey for each member of the sample
5 Organizational Culture

Questionnaire survey for each member of the sample

Feedback from focus group
One-time survey
One-time survey
Some
Involvement especially from management & support staff
Considerable involvement
Considerable involvement
Evaluation tools used in Project

S/No.
Module
1 Quality Criteria of E-Learning Courses
2 Performance Measurements of E-
Learning Courses
3 E-Learning Infrastructure
4 Staff Competencies
5 Organizational Culture
Specific Evaluation Tools
ECC Quality Criteria Checklist

Satisfaction survey

Pre-Test Results (if available)

Post-Test Results (if available)

Questionnaire survey for supervisors / facilitators

Inventory report of existing E-Learning infrastructure

Inventory report of proposed E-Learning infrastructure
Questionnaire survey for each member of the sample

Questionnaire survey for each member of the sample

Feedback from focus group
Maximum Sample
Size
20
40
20
20
20
Sample size of Trainees

Project Schedule on E-learning Evaluation for ABC

14 competencies defined
2 competencies defined 9 competencies defined
4 competencies defined
3 competencies defined

How effective has the e-learning been in the
ABC training?

Management Level
• Lack of a short-term or long-term plan for e-learning
(vision, goals, targets, timeline, etc)
• Unsure of the position of e-learning in the overall training curriculum
Pedagogical area
• Lack of a consistent explanation or definition of e-learning as used in the context of the ABC environment
• Have a narrow interpretation that e-learning means delivering courseware or content-based learning

• E-learning is used in a “reinforcement” mode
• All the courseware titles (except the scenario-based video courses) that have been evaluated are based on passive learning. Other types like e-Test system are used in Dept A and other Departments have been mentioned but we were not shown.
• Need to deploy appropriate instructional strategies (e.g. discussion, lecture, self-directed learning, mentorship, small group work, project, collaborative learning, case study, forum) for effective training using e-learning. Passive learning is based very much on memory work. Need to be aware of paradigms like constructivism and social constructivism to engage the learners.
• Trainers need to know how to track trainee’s e-learning progress
• Need for a proper assessment of the performance of the trainees in e-learning and then to measure the transfer to his job performance

Organizational Culture towards E-learning
• Need sharing of best practices amongst staff members
• Need to build up e-learning training resources for e-learning training staff
• Need to have an E-learning Outsourcing Development
Methodology
Awareness and competency levels of e-learning stakeholders
• Need to raise awareness on the following areas:
– What is e-learning?
– What e-learning can do and cannot do?
– Why ABC is pushing for e-learning?
– Need to move to a more competency-based model to train staff and assign tasks and responsibilities

Overall
Need to have an ABC centre which coordinates the following:
• Monitor and raise the e-learning competencies of the e-learning personnel.
• Maintain the quality of the courseware or other e-learning activities
• ABC can also position itself as the central body for the following activities:
– e-learning awareness and promotional activities
– Identify foundational e-learning competencies for e-learning personnel in the ABC Training schools
– work with external organizations to get the best deals in any tender i.e. courses, infrastructure requirements etc.

S/No. Criterion
1 Quality of Courseware
Score
2 / 5
2 Performance of Ex-trainees 1.5 / 5
3 E-learning Infrastructure 2 / 5
4 Key Competencies of staff 1.5 / 5
5 Organizational Culture towards E-learning
1.5 / 5
Overall score 8.5 / 25 or 34%

S/No.
Dept.
1.
Dept #1
2.
Dept #2
3.
Dept #3
Total no. of training hours
(approved timetable)
1130
1039
724
Total no. of e-learning hours
(approved timetable)
16
12.5
23
% of E-learning in Training
Curriculum
1.4
1.2
3.2
(Note the low percentages of e-learning component in the training curriculum.)

• E-learning evaluation is not straightforward. No set formula or set template or set model to use.
• Based very much on what the client wants to measure.
• The percentages of e-learning in the overall training curriculum in the three departments are 1.4, 1.2 and
3.2 respectively. Is it worth your while to do the evaluation?
• Most of the feedback are feedback gathered from the trainees, supervisors, focus group discussions. These are all very qualitative.
• Job performance is the hardest area to determine.

• Plan to measure and measure what you have planned. E.g. Quality of courseware, job performance.
• Difficult to measure what you have not planned to measure.
• Senior Management may have other views about e-learning. Good to have their buy-in.

• E-learning is the use of Internet and digital technologies to create experiences that educate our fellow human beings
• E-technologies do not change how human beings learn but remove constraints
• E-learning is not just Web-casting lecture, and not training materials dumped online
Professor Tapio Varis
Unesco Chair in global e-Learning
University of Tampere

LMS
Internet
Intranet
Mobile
CD-ROMs
Self-directed learning
Collaborative learning
Peer learning
Passive learning
Instructor-led
Pedagogy
Technology
Contents
Highly structured
Open Contents
Test-based
Knowledge-based
Learning objects
Rich multimedia

Lim Kin Chew kclim@unisim.edu.sg