A Practical Corporate E-Learning Evaluation Model

advertisement
A Practical Corporate E-Learning
Effectiveness Model
Lim Kin Chew
SIM University
23 Feb 2012
8th International Conference on E-Learning for a Knowledge-Based
Society
Bangkok, Thailand
Quotation - 1
“Although recent attention has increased
e-learning evaluation, the current research
base for evaluating e-learning
is inadequate … Due to the initial cost
of implementing e-learning programs,
it is important to conduct evaluation
studies.”
(American Society for Training and
Development, 2001).
Quotation - 2
“Despite the ample support and enthusiasm for
virtual models of learning from both the
corporate and educational arenas, the
effectiveness of e-Learning is by no means a sure
thing. Successful e-Learning programmes require
specific conditions and constant monitoring and
a lot more research has yet to be done before an
acceptable universal model can be developed.”
(Kar-tin Lee, Hong Kong Institute of Education,
Malaysian Online Journal of Instructional Technology,
August 2005, ISSN: 1823-1144, Vol. 2, No.2, pp 61-71)
Purpose of this Paper
• Share experiences on how an actual
evaluation of a large-scale e-learning project
was carried out
• Add to the research base on e-learning
effectiveness
Agenda
• Why do we need a practical corporate elearning effectiveness model?
• Different approaches to evaluation models
• Case study on an e-learning implementation
• QPICC model
• Lessons learnt
• Conclusion
Different Approaches to E-learning
Evaluation
(Source: Graham Attwell (ed.). (2006). Evaluating E-learning - A Guide to the Evaluation of Elearning. Evaluate Europe Handbook Series Volume 2. ISSN 1610-0875.)
Why do we need a practical e-learning
effectiveness model?
• E-learning effectiveness is difficult to carry out.
• Even if it is carried out, it is usually done in a
rather ad hoc manner.
• E-learning evaluation takes up time, resources,
manpower, cost and it may end up inconclusive
• E-learning evaluation should be done by
experienced e-learning practitioners
• E-learning evaluation depends very much on
what the user client wants to evaluate or
measure. The consultant guides him along.
Purpose of Project
• To evaluate the effectiveness of the e-learning
efforts in facilitating the training functions of
ABC
• Based on the evaluation outcomes, the client
wants appropriate recommendations on elearning capabilities so as to further enhance
instructional delivery in the ABC training.
Case Study on E-learning Evaluation
• ABC (name changed because of confidentiality) is a
government body in charge of training personnel for
home security.
• They have been using e-learning since 2006.
• Most of the training has been supplementary, i.e.
used to support the classroom face-to-face sessions.
• Investments have been made to set up three LMSs
• Management now wanted to find out how effective
has the e-learning been in training the personnel
• Client has several departments. Only three
departments were involved in the evaluation project.
Objectives & Methodologies - 2
To evaluate the effectiveness of ABC e-learning training
platform in the following areas:
i.
ii.
iii.
iv.
v.
qualify for the pedagogical underpinnings considered for elearning modules;
measure the quality of e-learning course wares;
measure the performance output by benchmarking the elearning platform against a set of pre-defined criteria;
determine the adequacy of existing e-learning
infrastructure and to validate congruency in these
capabilities with their instructional intent;
identify trainer and training designer’s competency gaps in
e-learning facilitation and design based on the evaluation
outcome.
Proposed E-learning Quality & Effectiveness Model for ABC
How the Project was organized
Team
Leader
Team UniSIM
•
•
•
•
Team ABC
Participative approach
Expertise transfer
Joint review of results, observations and conclusions
PLAN – DO – CHECK – ACT (PDCA)
S/No. Module
Specific Evaluation Tools
1
Quality Criteria of ELearning Courses
Quality Criteria Checklist
2
Performance
Measurements of ELearning Courses

E-Learning
Infrastructure

4
Staff Competencies
5
Organizational Culture
Questionnaire survey for each
member of the sample
 Questionnaire survey for each
member of the sample
 Feedback from focus group
3




Frequency of
Use
Once for every
E-Learning
course
Satisfaction survey (for
To do all for one
participating e-learners)
time for one ePre-Test Results (if available) learner in the
Post-Test Results (if available) sample
Questionnaire survey for
supervisors / facilitators
Inventory report of existing ELearning infrastructure
Inventory report of proposed
E-Learning infrastructure
To do a onetime
compilation
report
One-time
survey
One-time
survey
Evaluation tools used in Project
Involvement of
ABCs e-Learning
Champions
Some
involvement
Considerable
involvement as
we will be
checking their
performance
results
Some
Involvement
especially from
management &
support staff
Considerable
involvement
Considerable
involvement
S/No.
Module
Specific Evaluation Tools
1
Quality Criteria of E-Learning Courses
ECC Quality Criteria Checklist
2
Performance Measurements of ELearning Courses




3
E-Learning Infrastructure
4
Staff Competencies
5
Organizational Culture
Satisfaction survey
Pre-Test Results (if
available)
Post-Test Results (if
available)
Questionnaire survey for
supervisors / facilitators

Maximum Sample
Size
20
40
Inventory report of
existing E-Learning
infrastructure
 Inventory report of
proposed E-Learning
infrastructure
Questionnaire survey for each
member of the sample
20

20

Questionnaire survey for
each member of the
sample
Feedback from focus
group
Sample size of Trainees
20
Project Schedule on E-learning Evaluation for ABC
E-learning Development Cycle
14 competencies defined
2 competencies defined
4 competencies defined
9 competencies defined
3 competencies defined
Effectiveness of E-learning
How effective has the e-learning been in the
ABC training?
Overall Key Findings of the Evaluation - 1
Management Level
• Lack of a short-term or long-term plan for e-learning
(vision, goals, targets, timeline, etc)
• Unsure of the position of e-learning in the overall training
curriculum
Pedagogical area
• Lack of a consistent explanation or definition of e-learning
as used in the context of the ABC environment
• Have a narrow interpretation that e-learning means
delivering courseware or content-based learning
Overall Key Findings of the Evaluation - 2
• E-learning is used in a “reinforcement” mode
• All the courseware titles (except the scenario-based video courses)
that have been evaluated are based on passive learning. Other
types like e-Test system are used in Dept A and other Departments
have been mentioned but we were not shown.
• Need to deploy appropriate instructional strategies (e.g. discussion,
lecture, self-directed learning, mentorship, small group work,
project, collaborative learning, case study, forum) for effective
training using e-learning. Passive learning is based very much on
memory work. Need to be aware of paradigms like constructivism
and social constructivism to engage the learners.
• Trainers need to know how to track trainee’s e-learning progress
• Need for a proper assessment of the performance of the trainees
in e-learning and then to measure the transfer to his job
performance
Overall Key Findings of the Evaluation - 3
Organizational Culture towards E-learning
• Need sharing of best practices amongst staff members
• Need to build up e-learning training resources for e-learning
training staff
• Need to have an E-learning Outsourcing Development
Methodology
Awareness and competency levels of e-learning stakeholders
• Need to raise awareness on the following areas:
–
–
–
–
What is e-learning?
What e-learning can do and cannot do?
Why ABC is pushing for e-learning?
Need to move to a more competency-based model to train staff and assign
tasks and responsibilities
Overall Findings of the Evaluation - 4
Overall
Need to have an ABC centre which coordinates the following:
• Monitor and raise the e-learning competencies of the e-learning
personnel.
• Maintain the quality of the courseware or other e-learning
activities
• ABC can also position itself as the central body for the following
activities:
– e-learning awareness and promotional activities
– Identify foundational e-learning competencies for e-learning
personnel in the ABC Training schools
– work with external organizations to get the best deals in any
tender i.e. courses, infrastructure requirements etc.
Scoring Scheme
Totally ineffective
1
Ineffective
2
Satisfactory
3
Effective
4
Very effective
5
Overall Effectiveness Model
S/No. Criterion
Score
1
Quality of Courseware
2/5
2
Performance of Ex-trainees
1.5 / 5
3
E-learning Infrastructure
2/5
4
Key Competencies of staff
1.5 / 5
5
Organizational Culture
towards E-learning
Overall score
8.5 / 25 or 34%
1.5 / 5
% of E-learning in Training Curriculum
S/No.
Dept.
Total no.
of
training hours
(approved
timetable)
Total no. of
e-learning
hours
(approved
timetable)
% of E-learning
in Training
Curriculum
1.
Dept #1
1130
16
1.4
2.
Dept #2
1039
12.5
1.2
3.
Dept #3
724
23
3.2
(Note the low percentages of e-learning component in the training curriculum.)
Lessons Learnt - 1
• E-learning evaluation is not straightforward. No set
formula or set template or set model to use.
• Based very much on what the client wants to measure.
• The percentages of e-learning in the overall training
curriculum in the three departments are 1.4, 1.2 and
3.2 respectively. Is it worth your while to do the
evaluation?
• Most of the feedback are feedback gathered from the
trainees, supervisors, focus group discussions. These
are all very qualitative.
• Job performance is the hardest area to determine.
Lessons Learnt - 2
• Plan to measure and measure what you have
planned. E.g. Quality of courseware, job
performance.
• Difficult to measure what you have not
planned to measure.
• Senior Management may have other views
about e-learning. Good to have their buy-in.
What is E-learning?
• E-learning is the use of Internet and digital
technologies to create experiences that
educate our fellow human beings
• E-technologies do not change how human
beings learn but remove constraints
• E-learning is not just Web-casting lecture, and
not training materials dumped online
Professor Tapio Varis
Unesco Chair in global e-Learning
University of Tampere
E-Learning
Pedagogy
LMS
Internet
Intranet
Mobile
CD-ROMs
Technology
Contents
Self-directed learning
Collaborative learning
Peer learning
Passive learning
Instructor-led
Highly structured
Open Contents
Test-based
Knowledge-based
Learning objects
Rich multimedia
End of Presentation
Lim Kin Chew
[email protected]
Download
Related flashcards

Canadian Hockey League

15 cards

Civil defense

33 cards

Create Flashcards