The Flaw in the Fatal Flaw with 360 Surveys
By Bruce Bennett
Condemn all cars because the engine in the lemon you bought is broken and you miss the value
of every other car on the road. The ‘fatal flaw’ in 360 feedback that Marcus Buckingham
describes in his Harvard Business Review Blog is equivalent to a broken engine in a lemon.
Marcus asserts the big problem with 360 feedback is, in his words “bad data.” He correctly
observes that the flaw occurs because raters score a leader subjectively. Raters, he suggests,
mentally compare the leader to themselves and score accordingly, making the data worthless.
This is a real flaw for poorly written 360 survey questions.
Write the questions well, and that flaw does not exist.
A specific, observable behavior is among the first requirements for a good survey question. If the
behavioral description is clear, anyone and everyone observing a leader can recognize when the
behavior is performed. In statistical parlance it’s called inter-rater reliability.
Such observations are not only the basis of good 360 questions, they are the basis of all scientific
research. Whether it’s a biologist counting bacteria in a petri dish or naturalist studying tree
frogs in the Amazon, the quality of their data depends on how well they have defined what they
are looking for.
In a 360 survey question, the behavioral description is combined with a scale; both have to be
good for the question to work. Chose the wrong scale and you destroy the question. Agreement
scales are very risky because you are trying to measure what the person is thinking and you
can’t see “thinking.” A frequency scale, combined with a clear behavioral description, creates the
condition for valid observations. How many bacteria in the petri dish? How many times did this
leader ____?
Take the question, “Do you agree this person has a clear vision for the future?” Marcus’s
discovered flaw is present. No answer can be independently confirmed because no one rater
can see the thinking that produced the answer. Change the question: “How often does this
person discuss her vision for the future with you?” and you will get data that can be confirmed
by anyone witnessing the discussions.
How many people have to agree before their observations are worthwhile? Researchers pull a
representative sample when it’s too difficult to poll the whole population. Random samples are
critical with large populations. Respondent groups in 360 feedback are small populations and
often 100% of possible respondents are invited. It’s not a sample so it can’t be a skewed sample.
If the question is, “how often does the leader do behavior X when working with his direct
reports,” and 100% of his direct reports respond, you can’t get a better sample.
If all research had that luxury of polling 100% of the research population, the Chicago Tribune
would not have declared Dewey the President on election night in 1948 and no one would have
to wait up for the election results today.
More important than sample size is the question, “what is the assessment measuring?”
An excellent 360 does not try to determine IF a leader possesses a particular set of skills, instead
it measures if the leader DEMONSTRATES those skills.
An excellent 360 is not just about self-awareness; it’s about illuminating the environment in
which the leader functions. A difference or gap in respondent group scores does not mean one
respondent group is right and the other wrong. The gap may not mean the leader is lacking a
particular skill. It does means there is an important difference in the perception of reality and
that difference can inhibit the leader’s ability to lead.
Bridging the gap may not be a skill issue; it may be a motivation issue, a system issue or a lack of
information. 360 feedback is a powerful diagnostic tool that highlights issues then tells the
leader what group to talk with and what to talk about to uncover root of the issue. The gap data
helps effective leaders understand the real issue, then make relevant changes to resolve it.
Not all gaps are bad. Often others rate a leader higher than the leader rates herself. In many
high performing organizations, the opposite of ‘benevolent distortion’ occurs. Leaders, driven
by the sense that they can never be good enough, rate themselves lower than their actual
performance.
That type of gap identifies an underutilized asset in the leader’s toolbox. Understanding how
valuable others consider a specific behavior informs the leader how to build on her strengths
and use those underutilized, highly valued behaviors more often.
360 feedback is hard to do well. Observing human behavior is not as easy or exact as counting
bacteria in a petri dish, but clear behavioral descriptions and the right scale produce useable,
valid, and valuable data.
Even Marcus admits he’s “…seen some extraordinary coaches use 360 results as the jumping off
point for insightful and practical feedback sessions.” Extraordinary coaches don’t use flawed
data and know what to do with good data. 360 surveys are a powerful leadership development
tool. That’s why 360’s continue to be used by 77 % of organizations surveyed by T&D magazine.
If you happen to buy a lemon, with a broken engine, don’t let it prevent you from experiencing
the thrill and performance of a well-designed automobile.
Bruce Bennett is the founder and CEO of STAR 360feedback, a full-service 360 survey partner to
consultants and organizations intent on building exceptional leaders. He is also the founder of Real
Victory, a non-profit organization helping underserved populations develop the skills to achieve
difficult goals.