A Process to Justify eLearning
How to Use a
Systems Approach for
eLearning eValuation
Based Upon
Enabling Objectives
International Society of Performance Improvement
Boston - 2003
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
A Cooperative Team Approach
BAYLOR
UNIVERSITY
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Meet the eValuation Team
• Mr. Ara Ohanian, CEO, Vuepoint Learning System
aohanian@vuepoint.com
• Dr. Jim (Mo) Moshinskie, CPT, Baylor University
james_moshinskie@baylor.edu
• Ms. Ann Kennedy, eLearning Director, David
Weekly Homes, Akennedy@dwhomes.com
• Mr. Jeff Davis, eLearning Director, Black & Decker
University, jeff.davis@bdk.com
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Meet David Weekly Homes
• 8 Southern states
• 1,200 team
members
• 18 courses
developed by Dr.
Moshinskie and
powered
by Vuepoint in
past 3 years.
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Why Vuepoint: An Integrated System
• Content Creator
• Learner
• Administrator
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Vuepoint Learning
System
eLearning eValuation
Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
• Short modules – 15-20 screens (1 minute per screen)
• Teach theeLearning
“needeValuation
to know”
– –link
“nice
to know”
Model
ISPI –to
Boston
- 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
• Make it as interactive as possible – be creative - fun
• Use variety
of VLS
Templates
to– vary
eLearning
eValuation
Model – ISPI
Bostonpace
- 2003 and appeal
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Screen shot - template
• 18 types of interactive templates
• Wizard toeLearning
guideeValuation
courseModel
development
– ISPI – Bostonfast.
- 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Award Winning
Winner of the 2001
Award of Excellence
from the International
Society of
Performance
Improvement (ISPI)
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Award Winning
2003 Professional
Builder Work-Force
Development
Achievement
Award
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
But Is It Working?
•
•
•
•
•
•
•
Do learners like it?
Is their knowledge increased?
Is it a significant change, or random?
Can mangers see a difference?
Are house sales up?
Are customers happier?
What are our lessons
learned?
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
What Is Our eValuation Process?
• Three phases broken down into stages.
• Each stage breaks down into procedures (steps)
and principles (tips and guidelines)
Each Stage Includes:
Procedures
Principles
Exact steps to follow
Tips and Guidelines
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Handout: Our Working Document
• Our process
is divided
into:
Stages
Procedures
Principles
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
PHASE 1: BEFORE
1. Select two SME’s.
2. Meet with Dr. Moshinskie
to develop the Content.
3. Form Content into a Process.
4. Determine key Procedures
and Principles.
5. Write the Enabling Objectives.
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
What Are Enabling Objectives?
• A short statement of the
procedure or principle
learner expected to perform.
• Start with a simple
measurable verb.
• Form into an organized list.
• Write the questions to them.
• Tie blended workshop
to them (checklist).
• Develop coaching and
evaluation checklists.
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Type of content
Critical?
Link to
question
Numbered
• Short
• Start with
verb
• Measurable
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
PHASE 2: DURING
• Classify Enabling Objectives (critical or not?)
• Write Pre-Test Questions
- Tie to specific critical Enabling Objectives
- VLS links Enabling Objectives  Screens
- Results in a customized Roadmap
• Design Each Module using ROPES
• Prepare Post Assessment
- VLS allows pooling, randomization
- VLS allows automatic Level 1 evaluation
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Using the ROPES Model
R
Review
O
Overview
Review previous topics and give needed
prerequisite information
Explain enabling objectives and WIIFMs
P
Present
Teach enabling objectives interactively
E
Exercise
S
Provide realistic scenarios based
on enabling objectives
Summarize Restate enabling objectives
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
SMEs Used the VLS
Comment feature
for quick editing.
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Screen shot - roadmap
• Base questions on exact enabling objectives
• Customized Roadmap based upon Pre-Test
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Motivate Learners – Give Awards
Service
Portfolio
B TWW
Sales
Change
Safety
Houses
Warranty
Email
Diversity
Communications
Builder
OSHA
Selling
Safety
Closing
DWH101
Posting
101
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
PHASE 3: FOLLOWING
1. Analyze Level 1 data
(VLS)
2. Analyze Level 2 (VLS)
3. Conduct Blended
Workshops
4. Allow self-assessment
5. Query Managers
6. Correlate Results
7. Analyze Job Performance
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Did learners like the course?
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Role of Blended Workshops
• Direct the learners in
applying the enabling
objectives by teams.
• Allow coaches to
evaluate learners
using checklist
(1-5 scale).
• Calculate scores.
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Randomly Select Learners
• Select 18-20 learners
• Query their line managers.
• Survey their customers.
(J.D. Powers)
• Analyze their job performance.
- Quality walk at home sites
- Mystery shop sales calls
- Videotape performance
• Tie results to the
enabling objectives.
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Self Assessment
• Ask eLearners to evaluate
their confidence levels
on performing critical
enabling objectives
• Rank 1-5 (Strongly disagree
to strongly agree)
• Include open ended:
“Which performance areas do you need
specific help on now?”
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Take the Demo
• www.vuepoint.com/vlsdemo/default.aasp
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Statistical Toolbox - eXcel
1.
2.
3.
4.
Means – What are the averages?
Standard Deviations – How consistent?
Paired T-Test – Is the difference significant?
Pearson Coefficient – Is there a relationship?
See pages 7 and 8
of handout.
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Statistical
Category
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Calculating Early Results
Level 1
Average: 4.6 (out of 5)
Level 2 –
Average PreTest = 73
Average Post Test = 92
Level 3 – Line Managers
My employees know
more – 4.5 (out of 5)
Qualitative Comments
- Too many interruptions
- Regional content varies
Confidence Level
- 89%
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Calculating Early Results
Blended Workshop Evaluations
Course 1 – 2.73 out of 5
Course 2 – 2.71 out of 5
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Pearson Correlation possibilities?
Is there a relationship between:
•
•
•
•
•
post-test scores + sales performance?
post-test scores + customer surveys?
blended workshop scores + sales performance?
walk throughs + customer satisfaction?
J.D. Powers surveys + blended workshop scores?
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Return on Investment angles?
•
•
•
•
•
•
•
Fewer warranty calls
Fewer defects
Fewer re-do’s
Fewer repairs
Increased customer satisfaction
Increased house sales – relate to recession trends
Performance improvement analysis
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Barriers to Success?
• Group Exercise
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Avoiding Interruptions
CAUTION:
Brain Under
Construction
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
ROI – Maybe?
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Next: Black & Decker
• 15 courses finished using Vuepoint Learning System;
8 more under development.
• Modules based upon ROPES.
• Combine with hands-on blended training in corporate
university workshops in Baltimore.
• Collect data based upon our eLearning eValuation
model.
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Summary
• Our eLearning eValuation uses a systems approach
for corporations to evaluate eLearning.
• It was developed by a team approach:
Client + Consultant + Vendor
• It offers a process consisting of 3 phases:
BEFORE, DURING, and AFTER
• Each phase contains a series of stages.
• The key is developing solid enabling objectives.
• Using statistical tools helps collect defendable data.
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
Want to Learn More?
Dr. Jim Moshinskie’s Website on eLearning
http://business.baylor.edu/james_moshinskie
On-site Workshops available
- Converting Content into eLearning: ROPES.
- How to eValuate eLearning with Confidence.
- FLASH-MX Basics Boot Camp.
Vuepoint Learning System
www.vuepoint.com
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker
eLearning Contract Work
• Dr. Jim Moshinskie
• Office: 254-710-4097
• Cell: 254-744-7934
We accept corporate eLearning
development projects.
eLearning eValuation Model – ISPI – Boston - 2003
Baylor University – David Weekly Homes – Vuepoint Learning System – Black & Decker