The Power of Peer Evaluations: Why Don’t They Get More Respect? Allen I. Kraut Professor Emeritus of Management Baruch College, CUNY Presented at METRO New York, NY May 8, 2013 1 2 3 Example of Peer Assessment In Industry Reginald Jones, GE CEO in 1978, asked key executives: “Suppose you and I were killed in a plane crash this week. Who should be the next chairman of GE?” Answers: “Jack Welch”… “Jack Welch “… “Jack Welch” Recounted in J. Sonnenfeld’s The Hero’s Farewell (1991) from report in R. Vancil’s Passing the Baton (1987) . 4 Non-use of Peer Appraisal Systems Is “ A Curious Paradox”* “Long been known … Peer assessments… are among the most accurate assessments….predicting future performance…. Yet only a handful of organizations are using them.” * From Latham &Associates (1993) 5 Three Recent Talent Management Books, With Not One Index Entry on Peer Evaluation 6 Research* Shows Peer Evaluations Useful in Predicting Success Have High Reliabilities (in .80s) Even after short acquaintanceship Consistent across groups Good Validities (.30s & 40s) For Training, Promotion, & Job Performance Unaffected by Friendship Patterns Same whether Research or Administrative Use Better than Staff/Observer ratings *See Kane & Lawler, (1978), Downey, R. G., & Duffy, P. J. (1978). 7 “Modest” Correlations Have Powerful Practical Effects Example for r=0.30 * Predictor Outcome Low High 35% 65% High Low • 65% 65% 35% From R. Rosenthal & R. L. Rosnow, “Essentials of Behavioral Research” (1984), on Binomial Effect – Size Display, pp.209-211 8 Peer Evaluations Have Higher Predictive Validity Than Most Other Measures For Overall Job Performance Top Measures (Partial list)* Validity (r) General Mental Ability Peer Ratings Interview (Unstructured) Assessment Center .51 .49 .38 .37 * From Meta-Analytic study by F. L. Schmidt & J. E. Hunter (1998), Psych. Bulletin, on 19 predictors 9 Peer Evaluations: Three Types Peer Nominations: Name Top People (and Sometimes the Bottom Ones) Peer Ratings: All Judged on a Common Scale Peer Rankings: Listed from Highest to Lowest 10 Study #1 A Powerful and Simple Way to Predict Executive Success: Results From a 25-Year Study of Peer Evaluations 11 Evaluation in Month-Long Training Program, At Integrated Fortune 100 Corporation Middle Managers: (“Minors, Triple AAA”) N = 184, 2nd & 3rd Level Managers Class size = 16 Executives: (“Major League”) N = 99, higher level (titles like Director, V-P) Class size = 12 From All Divisions and Functions Attended Program 1967 through 1968 Peer Ratings Done in Third Week 12 Form of Evaluation Rating (Forced) Mean= 2.93 LO SD= 0.77 HI 1 2 3 4 5 13 Participants’ Reactions to Peer Evaluation Two Major Complaints: 1. Great deal of work to rate all course-mates on many attributes (11, plus 2 “Overall”) 2. Resent giving low ratings to fellow attendees 14 Two Overall Predictors General Impression “Considering all of the (specific) factors, how would you rate this person?” Capacity for Advancement “Indicate your judgment of the potential of this person for the position of general manager of a new (company) division or its corporate staff equivalent?” Implied administrative set, but actually locked up for research 15 Development of Nomination Rating What if we asked only for top nominations? Smaller number of ratings No “low” evaluations To develop a “nomination” rating: - Count as “high” any rating of “1” or a “2” - For each person, divide number of nominations by maximum possible 16 Two Forms of Evaluation Rating HI 1 2 3 4 LO 5 Mean= 2.93 SD= 0.77 Nomination HI 1 1 2 3 0 4 Mean= 0.30 SD=0.26 LO 5 17 Peer Ratings and Peer Nominations Are Highly Correlated Rating Vs. Nomination For Total Sample, N = 283 General Impression 0.92 Capacity for Advancement 0.93 18 Study # 1 Used Two Criteria of Success After 25+ Years Highest Career Level Achieved? Became a Corporate Officer? (Less than One per 5,000 employees) 19 Peer Evaluations Predict Highest Career Levels Achieved Correlation with Highest Level Middle Managers Attribute General Impression Capacity for Advancement Rating Nomination Executives Rating Nomination .43* .38* .29* .26* .42* .38* .26* .21 p < .05; Note: evaluations were not related to year of leaving company. 20 A Criterion of Really Long-Term Executive Success Only 5.7% Became Corporate Officers (16 of 283) These are the “Major Leaguers” Who Made it to The “Hall of Fame” 21 “High” Nominees More Likely To Become Corporate Officers Attribute General Impression Correlation With Rating Nomination .22* .23* Capacity for Advancement .25* .24* * p < .05, N = 283 Note: Maximum possible r estimated at .49 22 Top Nominations on “Capacity for Advancement” Much More Likely to Become Corporate Officers Became Corporate Officer Peer Nomination Top 30% 14.1% Middle 40% 2.7% 1.1% Bottom 30% 0 2 4 6 8 10 12 14 16 PERCENT Total (n=283) = 5.7% 23 Another Look at Future Corporate Officers Of Those Who Became Officers (n=16) Their Peer Nominations Were In Top 30% 75% In Middle 40% 19% In Bottom 30% 6% 24 An Army Study of Long-Term Success, Shows Peer Nominations Valid Predictors R. J. Gerard, 2002 370 USMA Grads, Tracked 1976-2000 Peer Nominations of Leadership (top ¼; bottom ¼) Better than Tactical Officer Ratings, Cadet Rank, or GPA In Predicting Career Success Highest Rank Achieved (r = .33**) Top Service School Attended (r = .30**) Largest unit Commanded (r = .33**) ** = <.01 25 Study # 2, Same Industry Population, Shorter Time, Differing Criteria After Two Years Promotion (in Levels) Performance Appraisal After Eight years Promotion (in Levels) Performance Appraisal Pay Increase (%) 26 Peer Ratingsa Predict Short-Term Success, But Vary By Criteria, and Population a almost identical to peer nominations Criterion Correlation with Criterion Middle Managers Attribute Levels Moved: M = 1.75, SD = 1.55 Promotions (2 Yrs.) General Impression Capacity for Advancement Performance Appraisal (2 Yrs.) General Impression Capacity for Advancement Pay % Increase (After 8 Yrs.) General Impression Capacity for Advancement Executives M = 1.00, SD = 1.75b .38* .41* .10 .12 .12* .14* .43* .37* .41* .41* .23* .24* *p < .05; Note: evaluations were not related to year of leaving company. B Wide due to demotions 27 Peer Ratingsa Prediction of Managers’ Mid-Term Promotions Illusory? a almost identical to peer nominations Correlation with Promotions Attribute General Impression Capacity for Advancement After 2 yrs. After 8 Yrs. After 8 Yrs. Controlling For First 2 yrs. .38* .41* .12 .41* .42* .17 Conclusion? •Peer Prediction most Valid in Fixed Short-Term •Career Movement like a Tournament in Short-Term, But a Marathon over Long-Term *p < .05 28 Overall Conclusions Again, Peer Evaluations Predict Career Success Peer Nominations as Effective as Peer Ratings Are Simpler Avoid Devaluing Colleagues Proof of Effectiveness Varies with Situation Basis of Selection into Group Homogeneity of the Group Criteria Used, Restriction of Range Peer Evaluations Yield Major Benefits in Near-Term Help Put Light on Promising Long-Term Candidates A Simple, Easy Method Can Be Quite Powerful! 29 Some Limitations of This Research Setting in a Large Stable Hi-Tech Firm Long Careers in Same Company All Were Male Some Data NA, Pay Grades Shift to “Bands” Note: Data are “Old” is Not a Limitation 30 Today’s Organizational Conditions Favor Peer Evaluations For Selection Greater Emphasis on Teams Group’s Opinion More Important Than in Past Peer Acceptance is Critical Less Time to Develop Track Records Peers Can Make Relevant Judgments Real Issue may be “Political,” not Scientific 31 Recommendation: Use Peer Nominations More Often Include in Training Programs Ask Only For Top Nominations Do Not Confuse with “Feedback For Development” Use in Formal Assessment Centers Use in 360 Evaluations (not Just Development) In Current Organizational Hierarchies (Gingerly) Include Data in Executive Succession Planning Word Evaluations for Specific Criteria Desired (“to Head Division X, Start New Product Line, etc.) Respect Evaluation’s Short “Shelf Life” (2 to 4 Years) 32 An Example: Peer Ratings Are Used For Promotion Of State Troopers in Minnesota Ratings on Seven Scales (on BARS) Also, "I Would Feel Fine Reporting to Him/Her?” Used together with Supervisor Ratings, and Tests: Analytic Skills, Work Styles; Experience Reactions? Peer Input seen As Valuable Candidates Comfortable with This Source: Ron Page 33 34 If Peer Evaluations Are So Good, Why Aren’t They Used More?? Maybe They Are Being Used, and I Don’t Know? Some Other Possibilities...? 35 Don’t Show Up in Top Journals, Because They Lack “Theory” “In general, journals publish validity studies only when a case can be made for a contribution to scientific understanding.” Paul Sackett P. 773, In J. C. Scott and D. H. Reynolds, 2010 36 Squeezed Out by Other Topics In Today’s Zeitgeist? Some Examples: Work-family balance Emotional Labor Workplace Incivility Employee Engagement Organizational Citizenship Behavior 37 Profitability to Con$ultant$ ?? Proven, but Simple, Powerful Method Vs. More Complex, Larger Efforts Competency Modeling 360 Feedback Programs Talent Management Programs Et Cetera 38 ‘Political’ Concerns Management Giving Up Their Power? Friendship Biases, “Popularity” Contests? Get Divisiveness, Peer Competitiveness? Reaction to Changing Rules of Game? Individualistic American Culture Hostile to Peer Inputs? 39 Your Thoughts?? …and Thank You! 40