The Power of Peer Evaluations:
Why Don’t They Get More Respect?
 Allen I. Kraut
 Professor Emeritus of Management
 Baruch College, CUNY
Presented at METRO
New York, NY
May 8, 2013
1
2
3
Example of Peer Assessment
In Industry
Reginald Jones, GE CEO in 1978, asked key
executives:
“Suppose you and I were killed in a plane
crash this week. Who should be the next
chairman of GE?”
Answers: “Jack Welch”… “Jack Welch “…
“Jack Welch”
Recounted in J. Sonnenfeld’s The Hero’s Farewell (1991) from report in R. Vancil’s
Passing the Baton (1987) .
4
Non-use of Peer Appraisal Systems
Is “ A Curious Paradox”*
“Long been known … Peer
assessments… are among the most
accurate assessments….predicting
future performance….
Yet only a handful of organizations are
using them.”
* From Latham &Associates (1993)
5
Three Recent Talent Management Books,
With Not One Index Entry on Peer Evaluation
6
Research* Shows Peer Evaluations
Useful in Predicting Success
 Have High Reliabilities (in .80s)
 Even after short acquaintanceship
 Consistent across groups
 Good Validities (.30s & 40s)




For Training, Promotion, & Job Performance
Unaffected by Friendship Patterns
Same whether Research or Administrative Use
Better than Staff/Observer ratings
*See Kane & Lawler, (1978), Downey, R. G., & Duffy, P. J. (1978).
7
“Modest” Correlations Have Powerful Practical
Effects
Example for r=0.30 *
Predictor
Outcome
Low
High
35%
65%
High
Low
•
65%
65%
35%
From R. Rosenthal & R. L. Rosnow, “Essentials of Behavioral Research” (1984),
on Binomial Effect – Size Display, pp.209-211
8
Peer Evaluations Have Higher Predictive
Validity Than Most Other Measures
For Overall Job Performance
Top Measures




(Partial list)*
Validity (r)
General Mental Ability
Peer Ratings
Interview (Unstructured)
Assessment Center
.51
.49
.38
.37
* From Meta-Analytic study by F. L. Schmidt & J. E. Hunter (1998), Psych. Bulletin,
on 19 predictors
9
Peer Evaluations: Three Types
 Peer Nominations:
Name Top People
(and Sometimes the Bottom Ones)
 Peer Ratings:
All Judged on a Common Scale
 Peer Rankings:
Listed from Highest to Lowest
10
Study #1
A Powerful and Simple Way
to Predict Executive Success:
Results From a 25-Year Study
of Peer Evaluations
11
Evaluation in Month-Long Training Program,
At Integrated Fortune 100 Corporation
 Middle Managers: (“Minors, Triple AAA”)
 N = 184, 2nd & 3rd Level Managers
 Class size = 16
 Executives:
(“Major League”)
 N = 99, higher level (titles like Director,
V-P)
 Class size = 12
 From All Divisions and Functions
 Attended Program 1967 through 1968
 Peer Ratings Done in Third Week
12
Form of Evaluation
Rating (Forced)
Mean= 2.93
LO SD= 0.77
HI
1
2
3
4
5
13
Participants’ Reactions to Peer Evaluation
Two Major Complaints:
1. Great deal of work to rate all course-mates
on many attributes (11, plus 2 “Overall”)
2. Resent giving low ratings to fellow
attendees
14
Two Overall Predictors
General Impression
“Considering all of the (specific) factors, how would
you rate this person?”
Capacity for Advancement
“Indicate your judgment of the potential of this
person for the position of general manager of a new
(company) division or its corporate staff equivalent?”
Implied administrative set, but actually
locked up for research
15
Development of Nomination Rating
 What if we asked only for top nominations?
 Smaller number of ratings
 No “low” evaluations
 To develop a “nomination” rating:
- Count as “high” any rating of “1” or a “2”
- For each person, divide number of
nominations by maximum possible
16
Two Forms of Evaluation
Rating
HI
1
2
3
4
LO
5
Mean= 2.93
SD= 0.77
Nomination
HI
1
1
2
3
0
4
Mean= 0.30
SD=0.26
LO
5
17
Peer Ratings and Peer Nominations
Are Highly Correlated
Rating Vs.
Nomination
For Total Sample, N = 283
 General Impression
0.92
 Capacity for Advancement
0.93
18
Study # 1
Used Two Criteria of Success
After 25+ Years
 Highest Career Level Achieved?
 Became a Corporate Officer?
(Less than One per 5,000 employees)
19
Peer Evaluations Predict
Highest Career Levels Achieved
Correlation with Highest Level
Middle Managers
Attribute
General
Impression
Capacity for
Advancement
Rating Nomination
Executives
Rating Nomination
.43*
.38*
.29*
.26*
.42*
.38*
.26*
.21
p < .05; Note: evaluations were not related to year of leaving company.
20
A Criterion of Really Long-Term
Executive Success
 Only 5.7% Became Corporate Officers
(16 of 283)
 These are the “Major Leaguers”
Who Made it to The “Hall of Fame”
21
“High” Nominees More Likely
To Become Corporate Officers
Attribute
General Impression
Correlation With
Rating
Nomination
.22*
.23*
Capacity for Advancement .25*
.24*
* p < .05, N = 283
Note: Maximum possible r estimated at .49
22
Top Nominations on “Capacity for Advancement”
Much More Likely to Become Corporate Officers
Became Corporate Officer
Peer Nomination
Top 30%
14.1%
Middle 40%
2.7%
1.1%
Bottom 30%
0
2
4
6
8
10
12
14
16
PERCENT
Total (n=283) = 5.7%
23
Another Look at Future Corporate Officers
Of Those Who Became
Officers (n=16)
Their Peer Nominations Were
In Top 30%
75%
In Middle 40%
19%
In Bottom 30%
6%
24
An Army Study of Long-Term Success,
Shows Peer Nominations Valid Predictors
R. J. Gerard, 2002
 370 USMA Grads, Tracked 1976-2000
 Peer Nominations of Leadership
(top ¼; bottom ¼)
 Better than Tactical Officer Ratings,
Cadet Rank, or GPA
 In Predicting Career Success
 Highest Rank Achieved (r = .33**)
 Top Service School Attended (r = .30**)
 Largest unit Commanded (r = .33**)
** = <.01
25
Study # 2,
Same Industry Population,
Shorter Time, Differing Criteria
 After Two Years
 Promotion (in Levels)
 Performance Appraisal
 After Eight years
 Promotion (in Levels)
 Performance Appraisal
 Pay Increase (%)
26
Peer Ratingsa Predict Short-Term Success,
But Vary By Criteria, and Population
a
almost identical to peer nominations
Criterion
Correlation with Criterion
Middle Managers
Attribute
Levels Moved:
M = 1.75, SD = 1.55
Promotions (2 Yrs.)
General Impression
Capacity for Advancement
Performance Appraisal (2 Yrs.)
General Impression
Capacity for Advancement
Pay % Increase (After 8 Yrs.)
General Impression
Capacity for Advancement
Executives
M = 1.00, SD = 1.75b
.38*
.41*
.10
.12
.12*
.14*
.43*
.37*
.41*
.41*
.23*
.24*
*p < .05; Note: evaluations were not related to year of leaving company.
B
Wide due to demotions
27
Peer Ratingsa Prediction of Managers’
Mid-Term Promotions Illusory?
a
almost identical to peer nominations
Correlation with Promotions
Attribute
General Impression
Capacity for Advancement
After 2 yrs.
After 8 Yrs.
After 8 Yrs.
Controlling
For First 2 yrs.
.38*
.41*
.12
.41*
.42*
.17
Conclusion?
•Peer Prediction most Valid in Fixed Short-Term
•Career Movement like a Tournament in Short-Term,
But a Marathon over Long-Term
*p < .05
28
Overall Conclusions
 Again, Peer Evaluations Predict Career Success
 Peer Nominations as Effective as Peer Ratings
 Are Simpler
 Avoid Devaluing Colleagues
 Proof of Effectiveness Varies with Situation
 Basis of Selection into Group
 Homogeneity of the Group
 Criteria Used, Restriction of Range
 Peer Evaluations Yield Major Benefits in Near-Term
 Help Put Light on Promising Long-Term Candidates
 A Simple, Easy Method Can Be Quite Powerful!
29
Some Limitations of This Research




Setting in a Large Stable Hi-Tech Firm
Long Careers in Same Company
All Were Male
Some Data NA, Pay Grades Shift to “Bands”
 Note:
Data are “Old” is Not a Limitation
30
Today’s Organizational Conditions
Favor Peer Evaluations For Selection
 Greater Emphasis on Teams
 Group’s Opinion More Important Than in Past
 Peer Acceptance is Critical
 Less Time to Develop Track Records
 Peers Can Make Relevant Judgments
 Real Issue may be “Political,” not Scientific
31
Recommendation:
Use Peer Nominations More Often
 Include in Training Programs
 Ask Only For Top Nominations
 Do Not Confuse with “Feedback For Development”
 Use in Formal Assessment Centers
 Use in 360 Evaluations (not Just Development)
 In Current Organizational Hierarchies (Gingerly)
 Include Data in Executive Succession Planning
 Word Evaluations for Specific Criteria Desired
 (“to Head Division X, Start New Product Line, etc.)
 Respect Evaluation’s Short “Shelf Life” (2 to 4
Years)
32
An Example:
Peer Ratings Are Used For Promotion
Of State Troopers in Minnesota
 Ratings on Seven Scales (on BARS)
 Also, "I Would Feel Fine Reporting to
Him/Her?”
 Used together with Supervisor Ratings, and
Tests: Analytic Skills, Work Styles; Experience
Reactions?
 Peer Input seen As Valuable
 Candidates Comfortable with This
Source: Ron Page
33
34
If Peer Evaluations Are So Good,
Why Aren’t They Used More??
Maybe They Are Being Used,
and I Don’t Know?
Some Other Possibilities...?
35
Don’t Show Up in Top Journals,
Because They Lack “Theory”
 “In general, journals publish validity
studies only when a case can be
made for a contribution to scientific
understanding.”
Paul Sackett
P. 773,
In J. C. Scott and D. H. Reynolds, 2010
36
Squeezed Out by Other Topics
In Today’s Zeitgeist?
Some Examples:





Work-family balance
Emotional Labor
Workplace Incivility
Employee Engagement
Organizational Citizenship Behavior
37
Profitability to Con$ultant$ ??
 Proven, but Simple, Powerful Method
Vs.
 More Complex, Larger Efforts




Competency Modeling
360 Feedback Programs
Talent Management Programs
Et Cetera
38
‘Political’ Concerns
 Management Giving Up Their Power?
 Friendship Biases, “Popularity”
Contests?
 Get Divisiveness, Peer Competitiveness?
 Reaction to Changing Rules of Game?
 Individualistic American Culture Hostile
to Peer Inputs?
39
Your Thoughts??
…and Thank You!
40