Evaluation Procedures
Dr. Steve
Training & Development
INP6325
Perspectives on Training
1.
2.
3.
Trainee – What if trainee does well in training,
but gets fired or laid off for being ineffective on
the job?
Trainer – What if trainee develops what s/he
thinks is effective training, but supervisors won’t
let workers use newly learned techniques/skills?
Organization – What if organization spent big
money on training program, but employees are
still not effective?
Applied Questions

Training specialists interested in:
1.
2.
3.
4.
Whether criteria (performance, $ savings) indicate
improvement following training
Whether improvement was a result of training
Whether training will be equally effective with different
group of trainees
Whether training will be beneficial to other
organizations.
Research Questions
1.
2.
3.
Which of two or more training programs is the
best and why?
Which type of training works best with which type
of worker?
What type of organization can benefit most by this
type of training?
Types of Training Researcher

Negativists – Evaluating training is either
impossible or unnecessary


Often decisions made based on anecdotal evidence
Positivists – Only rigorous scientific evaluation is
worthwhile

If there’s no experimental data, it’s a waste of time


(Randall)
Often don’t have resources for full-blown experiment
Frustrates (Activists) – All training should be
evaluated, but must recognize that evaluation
quality varies depending on constraints.
Training Evaluations
Training Evaluation only as good as the:
 Question that is asked
 Appropriateness of the criteria
 Evaluation design
Training Evaluations: Ethical Issues
Ethical issues of empirical study

Use of control group means some may not get training



Eliminates promotion opportunities
Control group may feel slighted
Trainees act differently if know in experiment

Hawthorne Effect
Formative vs. Summative Evaluation

Formative Evaluation – evaluate whether
training program is operating as planned BEFORE
it is implemented


Focus on process criteria – how training should be
implemented
Summative Evaluation – Does training produce
the expected outcome?


Trained vs untrained group comparisons’
Comparative summative evaluation – which of 2 or
more training approaches produces greatest benefit
Threats to Validity of Training

Internal Validity – did training make a difference
in this situation?



Training Validity – Does trainee’s training performance
meet criteria for training program?
Transfer Validity – Does trainee’s job performance
meet criteria?
External Validity – Can training results be
generalized to other trainees or settings?


Intraorganizational Validity – Will training be
effective on new group of trainees?
Interorganizational Validity – Can training program
be used successfully at other organizations?
Threats to Internal Validity
1.
History – Events that happen between pre- &
post- tests have nothing to do with training

2.
Contamination of results (ex: layoffs, accidents,
holidays, etc.)
Maturation – Biological or psychological effects
of time on training

Effects of fatigue or disinterest
Threats to Internal Validity
3.
Testing – pre-test sensitizes trainee to upcoming
post-test

Trainee prepares for it, or learn from pretest

4.
Ex: GRE score  GRE course  GRE score improves
Instrumentation – questionable reliability of
test measure

If fluctuation in scores is normal, can’t attribute change
to training
Threats to Internal Validity
5.
Statistical Regression – Extreme (hi or lo) pretest scores will often regress to the mean on posttest
• Experimental
• Control
Pretest
6.
Posttest
Differential Selection of Participants – if
allow participants to volunteer, experimental group
may be more highly motivated

Random selection, random assign, or matched groups
Threats to Internal Validity
7.
Experimental Mortality – differential loss of
participants between groups


8.
Trainees who did poorly on pre-test may get
discouraged, fired, quit, etc.
Training may coincide with other
deadlines
project
Interactions – differential effects that one threat
might have on another

Ex: testing effects different in each group due to
selection
Threats to Internal Validity
9.
10.
Diffusion or Limitation of Treatments –
members of one group share information with the
other
Compensatory Equalization of Treatments –
to avoid perceived differential treatment, some
alternative might be given to control group

No longer true control, but 2nd experimental group
Threats to Internal Validity
11.
12.
John Henry Effect – competition between
experimental and control group
increases motivation of control
group to work harder
Demoralization of Control Groups – passiveaggressive response to not being selected for
training
Threats to External Validity
1.
Reactive Effect of Pre-testing
(sensitization)- effects of pre-test lead to
increase sensitivity of instruction

2.
Increased attention to material that was seen in pretest
Interaction of Selection & Experimental
Treatment (representative sample) –
characteristics of the group chosen for evaluation
may be different than in future groups
Threats to External Validity
3.
Reactive Effects of Experimental Settings –
Experimental group knows they’re being observed
causing them to react differently from future
groups

4.
Hawthorne, Guinea pig, or Pygmalion Effects
Multiple-Treatment Interference (carryover)–
problem with within subjects experiments

participant is exposed to more than one treatment previous treatments will affect later ones
Training Evaluation Designs

Pre-Experimental Designs

One Shot Case Study:

No control group
X T2

One Group Pre-Test/Post-Test:

Before and after comparison, no control group
T1 X T 2

Static Group Comparison:
X T2
-- T2
Training Evaluation Designs

True Experimental Designs

Pre-test/Post-test Control Group:
Controls for most internal validity threats (except diff treatment)
R T1 X T2
R T1 -- T2


Solomon 4-Group Design:

R
R
R
R

Controls for both internal and external threats
T1 X T2
T1 --- T2
-- X T2
-- --- T2
Post-Test Only Control:

Like static group, but with randomization
R X T2
R
T2
Training Evaluation Designs

Quasi-Experimental Designs

Time Series
Longitudinal method rules out maturation, testing, stat
regression
T 1 T 2 T 3 T 4 X T5 T 6 T 7 T 8


Non-Equivalent Control Group
Same as pre-test/post-test control, but not random selection
T1 X T2
T1 --- T2

Other Evaluation Methods


ANCOVA – use pre-test as covariate in comparing post-test
means
 Example covary measure of ability prior to training
 Useful when small r between pre & post test
R T 1 X T2
R T1 --- T2
Correlations – correlate training performance to on-thejob performance


Doesn’t necessarily mean training had effect
Content Validity – SMEs opinion that KSAs covered in
training were those identified in needs assessment