The Criterion Choices
Dr. Steve
Training & Development
INP6325
Why Evaluate Training?
What can be gained from evaluating training?
1.
Determine effectiveness of program.
2.
Demonstrate benefits of training to top
management and stakeholders
3.
Demonstrate job-relatedness of training (legal
implications)
4.
Research value in aiding future training
development
5.
Ability to make personnel decisions (promotion,
retention, etc.)
Why not evaluate training?

In a survey of 611 orgs, 92% claim to evaluate training
programs, however the vast majority of these were
simply trainee reaction, rather than learning or transfer
What’s preventing them?
1.
2.
3.
4.
Evaluation not often emphasized by management
Training directors do not know how
HR may not understand importance
View that evaluation is expensive and risky
Determining Criteria
To determine criteria, must know purpose of
evaluation:
 To predict job success, must evaluate training for
relationship between training performance and job
performance
 Must determine whether one training program is
better than another or no formal training at all.
Pessimistic view of criteria selection
Guion (1961)
1.
2.
3.
4.
5.
6.
7.
I/O psychologist has a hunch about a problem
Read a vague, ambiguous job description
Form a fuzzy concept of the ultimate criterion
Develop a set of measures that can be combined to
approximate ultimate criterion
Judge relevance of measure: deficiency & contamination
Data required for the measure are not available in
company personnel files (and never will be)
Select best available criterion
Evaluating training effectiveness

To assess effectiveness of training, must:
1.
2.
3.
Develop criteria that assesses trainees’ learning of the
KSAs necessary for the job
Assess performance at the end of training (progress)
Assess performance after period of time on job
(transfer)
Criterion Selection
Ultimate Criterion – better time management
B
A
D
C
A – KSAs not in Needs Assessment or Criterion
B – Criterion Deficiency
- KSAs in Needs Assessment, but not Criterion
C – Criterion Contamination (error + bias)
- KSAs in Criterion, but not Needs Assessment
D – Criterion Relevance
- KSAs in both Needs Assessment & Criterion
Actual Criterion - files reports on time
- Avoids overtime
- Meets deadlines
Criterion Deficiency

Criterion Deficiency – training program intended
to teach certain KSAs required for the job, but
criteria used to evaluate training are missing KSAs
- Sort Mail
- Weigh pkg
- K of pricing
Example: Postal Clerk
- mail sorting skill not part of the
training
Criterion Contamination

Criterion Contamination – extraneous variables
included in the criteria that were not part of
training program

Opportunity bias – some individuals might have a greater
opportunity for successful job performance which had
nothing to do with training
salesmanship
- KSAs
Geographic
location
Example: Salesperson
- performance affected by location
(opportunity) as much as by training
Criterion Reliability

Criterion Reliability – consistency of criterion
measure.


Example: inter-rater reliability of supervisory ratings
Negatively impacted by:




Competence of judges
Simplicity of behaviors
Overtness of behaviors
Operational definition of behaviors
Two Views of Criterion Development

Composite Criteria
Mastery of
needs assessment
A+B+C=X
Grade:
A if X > 90%
Problem: not very diagnostic
May not know where one was
successful or failed

Multiple Criteria
Mastery of
needs assessment
A
Skill at writing
task statements
B
Knowledge of
task analysis
techniques
C
Skill at
presenting
Results
Problem: may meet some
criteria, but not others. What
constitutes success?
Proximal vs. Distal Criteria


Proximal – short term criteria
Distal – Long term criteria

Example: Political training


Proximal: performance during campaign
Distal: performance in office
USA TODAY/CNN/Gallup Poll results
Below are the results of a USA TODAY/CNN/Gallup Poll:
1. Question: Do you approve or disapprove of the way George W. Bush is handling his
job as president?
Poll of National
Adults
Approve
Disapprove
No Opinion
9/22/2001
90
8
2
9/22/2002
66
30
4
9/22/2004
48
49
2
9/22/2006
42
55
3
Levels of Criteria
Proximal
1.
Reaction – opinion of trainees (survey)


2.
Learning – mastery of training material (test)



3.
Distal
Immediate knowledge
Knowledge retention
Behavior/Skill demonstration
Behavioral – trainee job performance (ratings)

4.
Affective reactions
Utility judgments
Transfer
Results – org profit by training ($$$$)
Reaction Criteria

Guidelines for Reaction Criteria Development
a.
b.
c.
d.
e.


Questions based on information from needs assessment
Q’naire includes quantifiable data (do not use ONLY open-ended
questions)
Q’naire should be anonymous
Should include SOME open-ended questions
Pilot test Q’naire for length and comprehension
Benefit: provides info from all trainees, not just those
with extreme opinions
Disadvantage: may have nothing to do w/ eventual
performance, but if training is perceived as poor it is less
likely to be taken seriously or skills retained
Reaction Criteria Example
Strongly
Disagree
Disagree
Neither
Agree nor
Disagree
Agree
Strongly
Agree
1. The objectives of this program were
clear
1
2
3
4
5
2. The instructor was helpful and
contributed to the learning experience
1
2
3
4
5
3. There was an appropriate balance
between lecture, participant involvement,
and exercises in the program
1
2
3
4
5
4. The topics covered in this program were
relevant to the things I do on my job
1
2
3
4
5
5. I can see myself performing more
effectively after attending this program
1
2
3
4
5
6. The logistics for this program (e.g.,
arrangements, food, equipment) were
satisfactory
1
2
3
4
5
Learning Criteria



Test at end of training addressing material covered
during training
Example: Can receptionist trainee recall the steps
to transfer a phone call on the company’s phone
system?
Pre-test/Post-test comparison
Behavior Criteria




Criteria should come directly from Task and KSA analyses.
Use experimental methods to demonstrate improvements
due to training.
Assess whether performance goals are met.
Example: Bus Driver performance:






Stops and restarts without rolling back
Tests brakes at tops of hills
Uses mirrors to check traffic
Signals following traffic
Stops before crossing sidewalk when coming out of driveway
Stops clear of pedestrian crosswalks
Results Criteria
Measure of training program in terms of meeting
organizational goals


Money saved = lower turnover, lower absenteeism,
improved morale, improved productivity, etc.
Utility – Whether training saves more money than
it costs

Ex: if no formal training in place, senior workers lose
time showing junior workers what to do.
Utility Analysis
U = (T x N x dt x Sdy) – (N x C)




U = $ value of training program (Value) – (Cost)
T = # years duration of training effect on performance
N = # trainees
dt = effect size or true difference in pre-post perf



= (Xc – Xe) /ryy
Sdy = std deviation of performance in $ of untrained
group
C = cost per trainee

Example: U = (2yrs x 100N x .5 x $5,000) – (100N x $200)
= $500,000 - $20,000 = $480,000 over 2 yrs
Utility Analysis
Reasons for NOT using utility analysis
 Data may not be readily available or unreliable
 Seeking non-monetary benefits of training
 Other variables confound results
Choosing Criteria
Conclusions:
 Reaction information is important to know for
whether trainees will accept training program, but it
does not translate into effectiveness in terms of
learning, transfer, or monetary savings
 Learning criteria is a good predictor of results
 Learning criteria is a modest predictor of transfer
 Behavior criteria is a modest predictor of results
Other Criteria Concerns

Can’t judge effectiveness of training strictly on
outcome (summative), must also look at process
(formative)

Process measures tell us source of outcome changes


Outcome alone is not diagnostic
Importance variables affecting process include: differences
between trainers, settings, student samples, motivation of
groups, etc.
Subjective vs. Objective Measures

Subjective – ratings, opinions



Problem of rater biases (halo, central tendency, leniency)
Easy to use
Was rater well-trained to make ratings?


Subjective ratings may be improved by training the rater
Objective – countable, observable measures such
as production, absences, defects missed, etc.

Problem of opportunity bias

Objective measures should be used when possible, but must be
aware of potential contamination and account for it