Valid Research Methods for
Evaluating Recognition Programs
David J. Cherrington, SPHR, DBA
Recognition Professionals International Conference
27 April 2010
Henderson, Nevada
Why Evaluate?
Significant questions managers ought to ask:
 How do our employees feel about our recognition
programs? Which ones, if any, are having the
biggest impact?
 Are supervisors adequately trained to provide
meaningful recognition?
 Does the involvement of top management make
any difference?
 Is our annual recognition banquet worth the time
and money we spend on it?
 How do the people who are not recognized feel
about our recognition program?
Three Strategic Research Questions:
• 1. Have we chosen the best recognition programs? (Selection
and Design issues)
• Informal programs
• Formal programs
• 2. How well have we implemented them? (Training,
Communication, and Presentation issues)
• How much do employees know about them?
• How often are they being used?
• How do they make employees feel?
• 3. Do our recognition programs impact employee attitudes,
productivity, or profitability? (Outcomes or Results issues)
Valid Research Requires Two Things:
• 1. Reliable Data – also called dependent variables or metrics
– Research Question: “How do you assess reliability?”
• 2. A Valid Design – also called the research method
– When should you collect the data?
• After the program is implemented
• Before and after
• Several times before and after
– From whom?
• Experimental group
• Control group
– How much?
• Total sample
• Random sample
• Stratified random sample
– Research Question: “How do you assess validity?”
RELIABLE DATA
Reliability
• Definition: Reliability means repeatability or
consistency of measurement. Reliable data are
not random numbers, subjective estimates, or
capricious measures.
• Operational Definitions:




Test-retest reliability
Alternate forms reliability
Split-halves reliability
Conspect reliability (inter-rater agreement)
Dependent Variables or Metrics
Employee Attitudes
• Job satisfaction
• Appreciation
• Company satisfaction
• Organizational commitment
• Intent to leave
• Perceived organizational
support
Organizational Metrics
 Absence Rate
 Turnover Rate
 HR Expense Factor
 Profit per employee
 Revenue per employee
 Cost of recognition
programs as a percent of
payroll
Absence Rate:
The absence rate allows organizations to track employee attendance over time or
to compare their number of lost days due to absences with the rates of others in
their industry and nationwide.
Absence rate =
Workdays lost due to absences
# of employees x # of days
Workdays Lost Due to Absence = full days of employee absence whether paid
or unpaid, excused or unexcused and includes unscheduled days for sickness,
personal business, emergency, family illness or death, disciplinary suspension,
and unexcused absences. For long-term absences, count only the first four
days. Do not count scheduled time off for holidays, vacations, and other leave.
Illustration
What is the Absence Rate for a company that has 60 full-time employees and
during the past year they had 410 total days of absence. All employees have two
weeks of paid vacation during the year.
410 days of absence
60 Employees x 250 days*
=
0.027 or
2.70%
*250 days = 50 weeks per year times 5 days per week
Absence Rates by Size of Company and for selected Industries
Size of company
25th % tile
Median
75th %tile
1-100 employees
0.84
1.34
2.18
101-500
0.61
1.27
2.12
501-1000
1.30
1.74
3.64
Over 1000
1.48
2.32
3.05
Construction
0.14
0.57
0.90
Manufacturing
0.67
1.48
2.22
Wholesale Retail trade
0.96
1.49
1.92
Transportation
1.05
1.60
2.07
Financial
0.79
1.27
2.21
Other services
0.56
1.50
2.51
Industry
Data provided by the Utah Employers Council, 2009
HR Functions that can be costed
Wayne Cascio: Costing Human Resources: The Financial Impact of Behavior in
Organizations, 3rd ed., Boston: Kent Pub. Co. 1991.
•
•
•
•
•
•
•
•
•
•
Turnover
Absenteeism and sick leave
Smoking
Employee Assistance programs
Programs to improve Attitudes
Labor Contract Costs: wages, benefits, vacations, overtime,
shift differentials, holidays, pensions
Employee Selection Tests
Assessment Centers for Selection
Dollar Value of Job performance
Utility of Training Programs.
Criteria for Evaluating Training
[From: Ralph F. Catalanello and Donald L. Kirkpatrick, “Evaluating Training Programs:
The State of the Art,” Training and Development Journal, 22, 1968, 2-9. ]
1. Reactions of participants, e.g. job satisfaction,
appreciation, company satisfaction,
organizational commitment, engagement.
2. Learning new knowledge and skills, e.g. test
scores using multiple choice, fill in the blank,
matching, and essay questions.
3. Behavior changes, e.g. quantity and quality of
performance, attendance, job performance.
4. Measurable Results, e.g. organizational metrics.
Sources of Data





Questionnaires and surveys
Interviews
Observations
Performance appraisals
Unobtrusive Measures
 Archive data
 Number of hits on the awards and catalogue websites
 Number of supervisors who complete online training
 Email responses from family members to say thank-you.
EMPLOYEE ATTITUDE MEASURES
Semantic Differential Scales
Me at Work – How I feel Most of the Time
Extremely
Quite
Slightly
Neutral
Slightly
Quite
Extremely
Appreciated
Unappreciated
Efficient
Inefficient
Penalized
Rewarded
Satisfied
Dissatisfied
Unproductive
Productive
Encouraged
Discouraged
Ineffective
Effective
Valuable
Worthless
The average of these semantic differential scales measures a “General Affective Tone.” See, William E. Scott, Jr. “The
Development of Semantic Differential Scales as Measures of ‘Morale’,” Personnel Psychology, 20 (1967): 179-198.
Appreciation
Likert Scales
Strongly
Disagree
The company expresses sincere gratitude to its
employees.
My supervisor expresses appreciation to me
for what I do.
I feel personally recognized and appreciated
for my contribution to this company.
When I do a good job, I am praised and
recognized.
Scales created by David J. Cherrington, Brigham Young University, Provo, UT, 84602
Disagree
Neutral
Agree
Strongly
Agree
Job Satisfaction
Strongly
Disagree
Disagree
Neutral
Agree
Strongly
Agree
I find real enjoyment in my work.
I consider my job rather unpleasant.*
I am often bored with my job.*
I am fairly well satisfied with my present job.
I definitely dislike my work.*
Most days I am enthusiastic about my work.
What should be done to improve job satisfaction? ____________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
* The scoring of this item is reversed.
From: Brayfield, A. H., & Rothe, H. F. (1951). “An index of job satisfaction.” Journal of Applied Psychology, 35, 307-311.
Engagement
Strongly
Disagree
Disagree
Neutral
Agree
Strongly
Agree
Performing my job is so absorbing that I forget
about everything else.
Time passes quickly when I am performing my
job.
I really put my heart into my job.
My own feelings are affected by how well I
perform my job.
I exert a lot of energy performing my job.
I take work home to do.
From May, D., Gilson, R., & Harter, L. (2004). “The psychological conditions of meaningfulness, safety and availability and
the engagement of the human spirit at work.” Journal of Occupational and Organizational Psychology. 77, 11-37.
Organizational Commitment:
Affective Commitment
Strongly
Disagree
Disagree
Neutral
Agree
Strongly
Agree
I would be very happy to spend the rest of my
career with this organization.
I enjoy discussing my organization with people
outside it.
I feel emotionally attached to this organization.
This organization has a great deal of personal
meaning for me.
I feel a strong sense of belonging to my
organization.
From: Allen, N. J., & Meyer, J. P. (1990). “The measurement and antecedents of affective, continuance and normative
commitment to the organization.” Journal of Occupational Psychology, 63, 1-18.
Recognition Budget per Employee
• The recognition budgets for four companies that
have been recognized for outstanding recognition
programs are:
– A telecommunications company with 36,000
employees budgets 1.0 % of payroll
– An international bank with 60,000 employees
budgets 0.7% of payroll
– A healthcare company with 3,400 employees budgets
0.2 % of payroll
– A bank with 23,000 employees budgets $8 mil (which
equals $348 per employee).
VALID RESEARCH DESIGNS
Research Designs~
1. Post-test–only design (case study)
X O
2. Pretest–post-test comparison
O X O
3. Pretest–post-test control group design
R O X O
R O – O
4. Post-test–only control group design
R – X O
R – – O
5. Solomon four-group design
R O X O
R O – O
R
– X O
R
– – O
6. Time-series design
O1 O2 O3 X
O4
O5
7. Separate sample pretest–post-test
control group design
Group 1 R O X –
R –
X O
Group 2 R O –
–
R
–
– O
Key
X = Experimental treatment or training program
O = Observation or measurement
R = Random assignment of participants to group/training condition
O6
Validity
• Definition: Do the results measure what you
think they are measuring? Can you interpret
the results accurately?
• Two types of validity:
– Internal validity: is the experimental design sound
so that the data can be properly interpreted?
– External validity: can the results of this study be
generalized (or applied) to other situations?
1. Case Study Design
• XO
You implement the program and then measure it.
 X = independent variable, e.g.
• New recognition program
• Training program
• New incentive program
 O = dependent variable
• “smile sheets” after a training program
• Participation in the peer-to-peer nominations
• Measures of job satisfaction or engagement
• A very weak but highly popular research design
2. Pretest-Posttest Comparison
• OXO
• Illustration: Changes in turnover or job satisfaction before
and after a new recognition program is initiated. ???
• Problems with this design:
 Mortality – some people drop out, usually the dissatisfied.
 Maturity – people constantly change during any period of time.
 History – unique external events can influence the measures.
 Instrument decay – measurement errors and unreliable data.
 Sensitizing effect of the pretest – measuring is not a neutral
event; people often behave differently after they are measured.
• This is a very weak research design.
3. Pretest-Posttest Control Group Design
• ROXO
• R O -- O
 R = random assignment to groups
 X = new recognition program, such as a new kit for
supervisors to use.
 O = number of thank you notes sent to subordinates
• This is an excellent research design when it can
be implemented.
4. Posttest Only Control Group Design
• R XO
• R -- O
• This design assumes that the two groups are
equivalent due to random assignment.
• This is an excellent research design that is
especially appropriate when pretesting is not
possible.
• Illustration: testing the effects of a new
employee orientation training on participation in
the peer-to-peer recognition program.
5. Solomon 4-group Design
• ROXO
RO–O
R -XO
R - -O
• This is the ideal research design because it
controls for all of the challenges to the internal
validity of a research design.
• Illustration: testing the impact of a training
program on “Developing a Culture of
Recognition” on employee attitudes, such as
empowerment, satisfaction, or appreciation.
6. Time Series Design
• OOOOOOXOOOOOO
• This design fails to account for the effects of history.
• Otherwise, this research design is ideal for examining
systematic changes in an ongoing program. Pretest
measures serve as a contrast for posttest measures.
• Illustrations:
–
–
–
–
–
–
awarding points to nominators,
increasing the points for supervisors,
increasing the percentage of employees recognized,
adding new merchandise to the catalogue,
Creating a “wall of fame”
Including a recognition column in the company’s newsletter
7. Quasi-Experimental Design
• ROXO
O–O
ROXO
O–O
This design uses groups that are already formed
and members cannot be randomly assigned.
Illustration: A new recognition program cannot be
implemented to randomly selected members of a
group. Therefore, randomly select some groups
to start the program first, while other groups
have to wait for six months before starting.
8. Path Analysis
• This design utilizes a regression analysis of
multiple variables to test the relationships
between them.
• The unit of analysis may be groups, such as
testing whether the average satisfaction scores of
employees in various divisions or departments
predict levels of customer satisfaction in those
divisions or departments.
• Path analysis tests the relationships in a
hypothetical model, also known as structural
equation modeling.
Hypothesized Recognition Model
• Do recognition programs influence employee
attitudes?
• Do improved employee attitudes lead to
improved employee performance?
• Does better employee performance impact the
company’s bottom line?
Do employee recognition programs
contribute to the bottom line?
Employee
Recognition
Improved
Employee
Attitudes
Higher
Employee
performance
Possible Metrics:
Absence Rate
Turnover Rate
HR Expense Factor
Profit per employee
Revenue per employee
Increased
Profits
“Applause” Program:
ROI Linkage Analysis
Applause
Participation
Employee
Satisfaction
Higher Applause
participation is
linked to higher
employee
satisfaction
Growth in
Applause =
3-4% growth
in employee
satisfaction
Customer
Loyalty
6-9% difference in
employee
satisfaction =
3-4% difference in
customer
satisfaction
Financial
Return
1% change in
customer
satisfaction =
3.4% change
in revenue